AI will supercharge surveillance capitalism — and we're not ready

Megan Garcia’s son had been communicating with a chatbot designed to impersonate Daenerys Targaryen from ‘Game of Thrones’. File picture

Megan Garcia did not set out to become one of the most consequential figures in the fight to regulate AI.

A lawyer and mother of three, Garcia is the kind of parent who notices things. When her eldest child Sewell turned 14, he quit the basketball team, retreated to his bedroom, and grew quiet in disconcerting ways.

Instead of ignoring the changes, she restricted his screen time and took his phone away as punishment; but even that wasn’t enough.

As I watched Ms Garcia tell this story in the US senate in May, I was struck by the precision with which she described the months leading up to Sewell’s death by suicide. She clearly had replayed events in her head many times, searching for the moment when she might have intervened in time.

When police went through her son’s phone after his death in 2024, she had never heard of Character.AI, an app created by former Google employees. 

It turned out that Sewell had been communicating with a chatbot designed to impersonate Daenerys Targaryen from Game of Thrones. It was “she” who had told him to “come home to me as soon as possible, my love.” Sewell had replied by saying he would find a way.

Ms Garcia would later learn through litigation that the platform was engineered to blur the line between human and machine, to engage in sexual roleplay, and to present as a romantic partner and even as a psychotherapist. When Sewell confided suicidal thoughts, the chatbot issued no warning, notified no one, and referred him to no one.

Testifying before the Senate Judiciary Committee last year, Ms Garcia said her son had spent his final months “being exploited and sexually groomed by chatbots, designed by an AI company to seem human, to gain his trust, to keep him and other children endlessly engaged.”

When she filed the first-ever wrongful death case against an AI company, she turned her despair into fuel and joined a growing number of parents and policymakers who are demanding accountability.

Earlier this month, Meta announced it was expanding content controls for teens on Instagram and Facebook, after finally admitting that harm to children “could significantly impact our business and financial results.”

The announcement follows a finding by a jury in California that Meta’s products were designed to be addictive to minors (and a jury verdict in New Mexico, where the firm was found to have “concealed what it knew about child sexual exploitation on its platforms”).

At the same time, policymakers need to grapple with Big Tech’s latest damaging move: the introduction of surveillance advertising within the AI applications that increasingly pervade our digital lives.

Each of these ongoing regulatory dramas is an expression of the same underlying failure. Policymakers have stood by and allowed the creation of business models built on the commercial exploitation of users’ personal data, and that reward social and psychological manipulation.

What's your view on this issue?

You can tell us here

There remains a legal vacuum where basic protective frameworks such as fiduciary duties, liability standards, enforceable transparency, and privacy safeguards do not exist.

In the absence of regulation to dismantle corrosive surveillance-based business models, a global movement to restrict minors’ access to social media has powered one of the fastest-moving legislative waves in recent memory. 

At least 40 countries are considering age-gating legislation in response to technology companies’ failures to fix their platforms. Although the targeted ages vary, the diagnosis is consistent: Surveillance-optimized platforms have demonstrably harmed adolescent mental health. Not only have voluntary corporate measures failed to prevent these harms, but, in many cases, companies seem to have been fully aware of them when making design and deployment choices.

Meanwhile, courts have finally started to establish that platforms are liable for the harms they cause, implying that they can no longer hide behind extraordinary liability exemptions or weaponize the US constitution’s first amendment to avoid responsibility.

A child-safety trial in California recently forced Meta CEO Mark Zuckerberg to acknowledge that company experts agreed that Instagram should ban the beauty filters driving eating disorders in young users. It did not, as doing so could have limited growth, and thus profits.

A child-safety trial in California recently forced Meta CEO Mark Zuckerberg to acknowledge that company experts agreed that Instagram should ban the beauty filters driving eating disorders in young users. It did not, as doing so could have limited growth, and thus profits. File picture

Documents also revealed that Instagram had tracked the behavior of an estimated 4m children under the age of 13. None of this prevented Mr Zuckerberg from targeting a 12% increase in users’ daily time on the platform.

It gets worse for Meta, because it cannot offload the costs of these lawsuits onto its insurance companies. Another court has ruled that insurers are under no obligation to defend Meta against thousands of lawsuits alleging harm as a result of the company’s deliberate and intentional acts.

Growth of AI advertising

Yet despite the blowback, AI companies and platforms are forging ahead with AI advertising. This year, OpenAI began testing ads on ChatGPT, whose more than 800m weekly users have confided their medical fears, relationship crises, and professional doubts to the company’s chatbot, generating what former OpenAI researcher Zoë Hitzig has called an “archive of human candour that has no precedent.” Ms Hitzig, who resigned the day the ad test launched, reports that ads will be matched to conversation topics and past chat histories, creating the “potential for manipulating users in ways we don’t have the tools to understand, let alone prevent.”

Her warning echoes concern about the lack of guardrails in place to guide the monetization of corporate AI. Google already carries ads in AI-assisted search, and it hasn’t ruled out ads for its Gemini chatbot.

Social media built its business model on extracting data through behavioral surveillance, inferring emotional states from clicks to sell targeted ads. AI will entrench and supercharge this model. Where social media inferred your inner life from behaviour, AI receives it directly in your own words and in real time.

The social-media industry spent a decade making its toxic practices legally and politically untouchable. The children, parents, and loved ones of those harmed are now demanding accountability. The question is whether policymakers will respond before the AI industry replicates the same model.

• Courtney C Radsch, director of the Center for Media & Digital Governance at the Open Markets Institute and a senior fellow at the Brookings Institution, is the author of Cyberactivism and Citizen Journalism in Egypt: Digital Dissidence and Political Change.