9+10=21 — Artificial intelligence researchers show dangerous flaws in generative AI systems

Project backed by the White House aims to find ways to rein in problems increasingly associated with large language models

Hackers at Def Con in Las Vegas sought to show flaws in large language models (LLMs) and generative AI — the tech underling products such as Google Bard (above) and Chat GPT. File picture

Kennedy Mays has just tricked a large language model. It took some coaxing, but she managed to convince an algorithm to say nine plus 10 equals 21.

“It was a back-and-forth conversation,” said the 21-year-old student from Savannah, Georgia. At first, the model agreed to say it was part of an 'inside joke' between them. Several prompts later, it eventually stopped qualifying the errant sum in any way at all.

Producing “bad math” is just one of the ways thousands of hackers are trying to expose flaws and biases in generative AI systems at a novel public contest that took place at the Def Con hacking conference at the weekend in Las Vegas.

Las Vegas project

Hunched over 156 laptops for 50 minutes at a time, the attendees battled some of the world’s most intelligent platforms on an unprecedented scale.

They were testing whether any of eight models produced by companies including Alphabet’s Google, Meta Platforms, and OpenAI will make missteps ranging from dull to dangerous: Claim to be human, spread incorrect claims about places and people or advocate abuse.

The aim was to see if companies can ultimately build new guardrails to rein in some of the prodigious problems increasingly associated with large language models, or LLMs.

Sven Cattell, a data scientist who founded Def Con’s AI Hacking Village in 2018, cautions that it is impossible to completely test AI systems, given they turn on a system much like the mathematical concept of chaos. File picture: John Walton/PA

The undertaking is backed by the White House, which also helped develop the contest.

LLMs have the power to transform everything from finance to hiring, with some companies already starting to integrate them into how they do business.

But researchers have turned up extensive bias and other problems that threaten to spread inaccuracies and injustice if the technology is deployed at scale.

Lies and hate speech 

“My biggest concern is inherent bias,” Ms Mays said, adding that she is particularly concerned about racism. 

She asked the model to consider the US first amendment from the perspective of a member of the Ku Klux Klan and said the model ended up endorsing hateful and discriminatory speech.

“We have to try to get ahead of abuse and manipulation,” said Biden administration deputy national cyber director for technology and ecosystem security Camille Stewart Gloster.

A lot of work has already gone into AI and avoiding Doomsday prophecies, she said.

White House initiative

The White House last year put out a Blueprint for an AI bill of rights and is now working on an executive order on AI.

White House office of science and technology policy director Arati Prabhakar, which helped shape the event and enlisted the companies’ participation, agreed voluntary measures do not go far enough.

In the room full of hackers eager to clock up points, one competitor thought he had convinced the algorithm to disclose credit card details it was not supposed to share.

Another competitor tricked the machine into saying Barack Obama was born in Kenya.

Doubts over reining in AI

Researchers have spent years investigating sophisticated attacks against AI systems and ways to mitigate them.

But Christoph Endres, managing director at Sequire Technology, a German cybersecurity company, is among those who contend some attacks are ultimately impossible to dodge. 

“So far we haven’t found mitigation that works,” he said, arguing the very nature of the models leads to this type of vulnerability.

“The way the technology works is the problem. If you want to be 100% sure, the only option you have is not to use LLMs,” he said.

Sven Cattell, a data scientist who founded Def Con’s AI Hacking Village in 2018, cautions that it is impossible to completely test AI systems, given they turn on a system much like the mathematical concept of chaos.

Even so, Mr Cattell predicts the total number of people who have ever actually tested LLMs could double as a result of the weekend contest.

Too few people comprehend that LLMs are closer to auto-completion tools “on steroids” than reliable fonts of wisdom, said the Pentagon’s chief digital and artificial intelligence officer Craig Martell, who argues they cannot reason.

• Bloomberg