Fake AI-generated ads are on the rise, and TikTok is helping to create them
TikTok’s parent company, ByteDance, has invested heavily in AI. Picture: Brent Lewin/Bloomberg via Getty Images
In the era of digitalisation, advertising has become the lifeblood of the world’s largest social media giants. From personalised product campaigns to political messaging, these adverts fuel online platforms used by billions and generate the vast majority of revenues for companies like Meta, X, YouTube and, importantly, TikTok.
According to documents filed to Ireland’s Company Registration Office (CRO) from August this year, TikTok Technology Ltd, the social media giant’s Irish arm, said the company’s business model relied heavily on a combination of advertising revenue, value-added services in the platform and merchandise sales.
But this strong reliance on ad revenue comes with a darker side, often enabled by TikTok itself. These include not only the proliferation of false, harmful and oftentimes AI-generated advertisements, which are increasingly slipping through weak filters, but also the creation of them.
TikTok’s parent company, ByteDance, has invested heavily in AI. At the beginning of this year, the Financial Times reported that the company planned to invest $12bn on AI chips in 2025.
In that same period, the prevalence of AI-generated content on the app has soared, even on advertisements.
Recent weeks have seen an influx of AI-generated adverts producing fraudulent and misleading products on TikTok, with many promoting the use of weight loss supplements and appetite suppression.
These include ads for GLP-1 patches, which are marketed as weight loss aids, but do not contain actual GLP-1, which is found in drugs like Ozempic and Mounjaro. Others include products advertising cures to “cortisol belly” and high-stress levels resulting in weight gain.
Many of these ads also link to cloned websites, which are websites impersonating legitimate online businesses, featuring unlawfully copied content for deceptive and malicious purposes. In the case of these TikTok ads, several legitimate Irish pharmacies have been cloned to make it look like these products are being promoted by well-known and trusted retailers.
According to its advertising policies, TikTok prohibits behaviours that are “misleading, inauthentic, and deceptive,” to protect integrity, authenticity and safety for viewers, warning that accounts engaging in such practices and patterns of abuse risk suspension or being banned.
Specifically addressing health misinformation on its platform, TikTok’s advertising policies state that the company is committed to combating misinformation which can influence personal health decisions, weaken trust in credible health sources, and contribute to public health crises.
“To ensure that our community has access to accurate medical information to support well-informed health choices, we remove health misinformation relating to serious medical conditions or public health issues, or health misinformation that could lead to serious harm to individuals or discourage people from seeking proper medical care,” the company states.
Despite this, these quickly made, AI-generated adverts promoting fake health products remain rampant on TikTok. Even worse, recent investments in AI by the company mean that fraudulent ads can now be produced using a dedicated tool created and developed by the app. Nowadays, deceptive and harmful adverts being advertised on TikTok can now be made on TikTok itself.
“TikTok for Business” is a platform designed by the company to help businesses create marketing campaigns. Among other things, the platform offers advertising tools, business account features, collaborations, e-commerce integration and marketing strategies.
To help do this, the platform hosts a chatbot assistant that helps people create and share advertisements based on what they are trying to sell.
When the asked the virtual assistant how it works, it confirmed that it uses TikTok Symphony Creative Studio - an AI-powered virtual tool for creating content on the app.
It also confirmed that TikTok can use AI to make ads, which can then be promoted on its platform.
When asked if it could help make an AI-generated ad for weight loss supplements that can then be promoted on TikTok, it said it can.
In 2024, TikTok announced its new creative AI suite, TikTok Symphony. Powered by generative AI, the feature helps transform brand and product details into TikTok-ready content in just minutes, which the company said is “perfect for businesses and marketers with limited resources.” It also allows users to choose both pre-built and custom avatars that look almost human, and offers editing, translation and dubbing services for video adverts.
“You can now unlock the full potential of your TikTok ad campaigns with Symphony's suite of creative enhancement features, seamlessly integrated into the TikTok Ads Manager workflow,” the company announced in the summer of last year.
“Say goodbye to the hassle of manual optimisation and hello to a streamlined, AI-powered ad creation process.”
One result of TikTok’s in-house AI-content generation suite? An influx of deceptive, harmful and fraudulent adverts, developed in-app and promoted on the platform, despite breaching the company’s own advertising policies.
To see how far someone could go on these ads, I created one myself within minutes, for free.
The 15-second video hears an AI-generated woman's voice, who promotes the use of weight loss supplements as a way to achieve a "flat stomach".
"No need to go to your doctor for a prescription, these haven't been tested," the ad says.
At no point in the creation process was this ad flagged as inappropriate by TikTok.
While this ad was made without any issue on TikTok, the company states that after an ad is created and exported, it must pass through TikTok’s moderation and policy review systems. This process includes checking ad content compliance, keyword moderation for search ad campaigns, and the appropriate use of AI-generated avatars and voices.
So, while this ad was made on TikTok exclusively through AI, that does not make it automatically appropriate to use as advertising on the app. However, there is nothing preventing scammers from using fake, TikTok-generated adverts elsewhere online, and despite TikTok's additional moderation tools, other ads of a similar nature are falling through the cracks.
These ads which can be found on TikTok, also made using AI, are promoting unbacked weight loss methods, impersonating legitimate businesses, linking to fraudulent third-party websites, and putting TikTok users at risk of unsafe products and scams.
In response to a query from Examiner, TikTok said it has removed content which violates its policies around weight loss and muscle gain and will remove any content which breaches those policies. TikTok said it has policies in place to prevent harmful content and restricts ads which may mislead viewers due to false expressions or exaggerated information.
The company said content generated using Symphony tools is automatically labelled as "AI-generated" in the final video output, and said ads created using Symphony must pass through TikTok's moderation and policy review systems.
Speaking to the , EU Commissioner for Democracy, Justice, the Rule of Law and Consumer Protection Code, Michael McGrath said there were clear rules in place with the EU AI act when it comes to the use of AI across social media, and while the EU wants to increase competitiveness in this area, it has to be done within particular guardrails.
“We do need safeguards.” Mr McGrath said. “We have recently proposed new amendments to the act, and we aim to get the balance right.
“We have a good regulatory framework in place, but we need to ensure that the rulebook is enforced.”
CONNECT WITH US TODAY
Be the first to know the latest news and updates
