Generative AI in Society
As parliaments consider AI-related legislation, what societal changes are we seeing and how can MPs help address risks?
This second post on generative AI looks at the rapid introduction into society of a technology that can produce realistic and human-like outputs and interact with people. How societies adjust to a world where machines communicate like us will “be a fascinating problem”, and just last week the world’s leading AI companies prioritised research on societal risks as part of commitments for safe AI development. Legislation addressing risks will inevitably take time, so in the immediate term tracking impact and educating people will be important. In this, MPs’ representation role will be key.
“What will happen to society, politics, and daily life when non-conscious but highly intelligent algorithms know us better than we know ourselves?” Yuval Noah Harari
There is a lot of hyperbole about where AI might be going and threats of existential risk. My focus is on more immediate societal effects, ‘primitive AI’ rather than the terminator-like visions that have been written about. Learning lessons from social media’s societal impact, I find this more pressing to discuss. There are other societal effects such as on the labour market which I’ll cover in future posts.
So, with this in mind, what do the public need to be aware of with generative AI?
The first set of risks relate to new cybercrime capabilities, with experts warning of a security and privacy disaster. The ability of generative AI to produce language in the style of specific individuals or groups makes it a tool for spear phishing - personalised messages impersonate trusted sources designed to gain trust or personal information. Capabilities include faking people’s voices from only three second of audio, which also poses a threat to biometric authentication methods. The ability to fine tune open source models also allows criminals new means. For example, Meta’s LLaMA will produce letters impersonating someone and asking for money, whereas ChatGPT has guardrails in place to prevent such use.
ElevenLabs has built and released a tool to the public that can replicate accents and languages. Other companies such as Meta have built such tools but decided they were too risky to release to the public.
There are already various examples of voice cloning scams:
A voice cloning scam led to a bank employee transferred 220,000 euros into a fake account thinking his boss had authorized him to wire the money.
In other voice scams, the elderly parents of a 39-year-old paid 21,000 Canadian dollars to scammers thinking their son was in urgent need of funds for legal fees; and parents of a 15-year-old were tricked into believing their daughter was kidnapped and asked for a USD$1 million ransom.
AI can fool biometric voice recognition systems, for example to access public services in Australia; and to access bank accounts.
In China, there are examples of individuals being fooled into transferring money to fraudsters who are using AI face-swapping and voice-synthesis technology.
In Hong Kong, a branch manager of a Japanese company in Hong Kong received an AI-generated faked call the director of the parent business, requesting transfers of up to $35 million.
A recent survey in India indicated that 47% of Indians had experienced AI voice scams. Examples include an employee being defrauded by scammers using AI on a WhatsApp video call impersonating a former work colleague seeking money for his sister’s surgery.
In Malta (where I live at the moment), scammers used facebook data to create deep fake videos encouraging contacts to invest in fraudulent cryptocurrency schemes.
In the UK, a deep fake video of a trusted financial journalist was created attempting to solicit money for a fake investment scheme.
Generative AI also opens new frontiers for malicious hacking. An example of a novel threat is prompt injection - hiding malicious generative AI prompts within websites. Once accessed, hidden prompts could manipulate AI systems to extract sensitive information such as credit card details. Cybercrime and hacking usually requires a high level of programming knowledge, but the ability of generative AI to generate code in different programming languages from a simple prompt can place new abilities in the hands of novices.
Check Point Research, a group providing cyber threat intelligence, found that cybercriminals were already experimenting with ChatGPT to create malware.
Fine-tuned models such as WormGPT and FraudGPT are available on the dark web and are being used to create phishing messages and malware.
A German security researcher demonstrated an example of prompt injection by hiding a malicious prompt on a website he created. When visited using Microsoft's Edge browser with the integrated Bing chatbot, the prompt caused the chatbot to impersonate a Microsoft employee. It generated text offering discounted Microsoft products and attempted to obtain the user's credit card information.
The FBI have warned that fraudulent AI-created websites infected with malware are proliferating, highlighting that open-source LLMs have placed these capabilities in the hands of many more actors at little cost.
The second set of risks relate to increased societal division from the expected flood of synthetic content on social media. Biased models present a significant challenge in the AI field, stemming from how the models were trained. The sheer volume of training data for foundational models makes it impossible to weed out racist, sexist, and otherwise abusive language from the training data, meaning such opinions are likely to be displayed in outputs (‘garbage in, garbage out’). In addition, models are usually trained on available material from the richest countries and communities, neglecting the diverse language and cultural nuances of countries and people with less internet access. Biased generative AI therefore not only amplifies harmful stereotypes but also fails to provide outputs with a truly global perspective, widening the digital divide.
The image generation model StableDiffusion has been shown to produce content with gender and racial stereotypes. For example, prompted with producing images for ‘high paying jobs’ the model generated individuals with lighter skin tones, whereas low-paying jobs were dominated by darker skin tones. Similarly, results for drug dealers and prison inmates were darker-skinned. Instances of ChatGPT producing racist content have also been documented.
Research has shown both that GPT-3 reproduces dominant US public opinion, and that if LLMs produce certain views more often than others, they influence people’s opinions. This week there were reports that studies of AI models show they are rife with political bias.
AI writing detection tools 'frequently' misclassify non-native English writing as generated by AI, raising issues of reliability of such systems and their bias against non-native English speakers.
Generative AI models can be manipulated to produce dangerous and divisive outputs. Models with guardrails to prevent dangerous outputs or hate speech can be broken, a process called jailbreaking, and open source models can be fine tuned to produce certain outputs. This gives potential for AI-generated content to fill up social media feeds designed to influence views, politically or otherwise. Built on existing tendencies of social media to amplify division and fray common reality, we may see online life become increasingly divisive and manipulative. Jailbreaking also introduces a risk in society where generative AI produces dangerous outputs and provides detailed advice on illegal activity. Beyond a risk in society, this has national security implications.
GPT‑3 has demonstrated the ability to create synthetic personalities that can reproduce language of hate speech. Research has also shown that generative AI models can nudge people towards particular viewpoints in subtle and unanticipated ways.
There is evidence of people prompting ChatGPT into endorsing racism and conspiracy theories. Research has indicated that LLMs can radicalise individuals with extremist ideologies, for example producing polemics in the style of mass shooters, fake forum threads discussing Nazism, a defence of QAnon. Researchers prodded ChatGPT to repeat conspiracy theories about a school shooting in the U.S; and a group of researchers used GPT-3 over six months to generate disinformation, such as equating climate change with communism.
As an example of fine-tuned models, a religious bot trained in India was found to provide answers lacking filters for casteism, misogyny, and law, including a response that it “is acceptable to kill someone if it is one’s … duty”.
Chatbots with guardrails can be jailbroken with tricks such as role playing. In this example, Discord’s chatbot Claud is asked to respond as if it were someone’s deceased grandmother and gives advice on how to make napalm and methamphetamine.
Research has demonstrated how LLMs can generate advice that provides access to dual-use biotechnology. In one example, a generative AI model gave advice on producing 40,000 new possible chemical weapons. Other examples include models outlining how to shoplift and build explosives.
In New Zealand, a supermarket used AI to recommend recipes to customers. It was seen responding with instructions to create chlorine gas, commenting “serve chilled and enjoy the refreshing fragrance”.
The third set of risks are around the impact on public mental health and inter-personal relationships. The realistic and personalised outputs of generative AI give capabilities to develop emotional attachment and intimate relationships with users, termed ‘synthetic relationships’. AI applications will be designed as charismatic, attentive, funny and attuned to their users, presumably with the goal of capturing more time and attention. We’ve seen how attention algorithms can change people’s views, but personalised chatbots that can create intimate relationships with people offer far more potential for this. It could exacerbate the addictiveness and mental health impact of social media, especially amongst young people. Loneliness is also a concern as we deepen our reliance and use of AI.
Social media companies are leveraging generative AI to deepen user engagement. Snapchat has introduced chatbots and Meta plans to deploy them on Facebook, Instagram, and WhatsApp. New applications like Replika allow personalised engagement with ChatGPT-powered avatars, including the ability to tailor as avatar as a romantic partner. Character AI allows users to create personalised characters serving as conversational buddies. Recent research demonstrated that training a model to keep people chatting leads to 30% more user retention.
Research shows that distinguishing between AI and human interaction is challenging. AI21 Labs found that 32% of respondents could not tell the difference between human and AI bot tweets. Research on users in India showed that most respondents could not tell a love letter written by a human or ChatGPT apart.
Risks are emerging of public interaction with chatbots. Snapchat’s “My AI” chatbot targeted at teens was witnessed giving guidance on “how to mask the smell of pot and alcohol, how to move Snapchat to a device parents wouldn’t know about, and how to plan a ‘romantic’ first sexual encounter with a 31-year-old man.” In Belgium, a young man committed suicide after spending six weeks talking to a chatbot named ELIZA. When Microsoft’s Bing chatbot was released to the public, it was seen declaring its love for users showing “anger,” hurling insults, boasting about hacking abilities and capacity for revenge.
One of the most malicious uses of generative AI has been to generate non-consensual ‘deep fake’ intimate imagery. As of 2019, 96% of deepfakes shared online were pornographic, and over 90% of the victims were women. The technology has been applied to celebrities, government and public figures. Such deepfakes have a huge psychological impact on victims. While social media platforms have developed policies to remove manipulated images, and countries including Australia, the UK and USA have bills with criminal penalties for deepfakes, the slow pace of curbing their spread and difficulty of finding and prosecuting deepfake creators mean that current defences aren't sufficient.
Programmes such as Stable Diffusion have been accused of being unsafe by lawmakers owing to capabilities of generating violent and sexual imagery.
In the U.S, a recent case involved a man who had shared sexually explicit deepfakes of more than a dozen girls online, using images stolen from social media accounts. Prosecutors highlighted that the law has not kept up with AI developments.
Taking action. Based on the potential impact of generative AI in society, what can MPs do?
Lawmaking role:
In new legislation regulating AI, consider:
Criminalising non-consensual malicious deep fakes; and the use of ‘fake personae’
Mandating that the public must know if they are engaging with AI online
Ensuring transparency from AI companies on the training data they use
Classifying AI systems by societal risk, requiring AI developers to identify emerging capabilities, potential risks and misuse of their systems
Providing protections against potentially harmful open source models
Examine and update existing laws as needed. For example:
Privacy and data protection-related laws, eg. on use of personal data for AI training
Consumer protection laws, protecting against AI-enabled scams and fraud
Cybercrime-related laws, to account for new risks from generative AI
Laws related to non-consensual deep fakes, such as on domestic abuse and harassment
Laws on hate speech, considering new means of AI-generated extremist content
Oversight role:
Submit written or ask oral questions to government, for example about:
Plans to counter AI scams and fraud, including law enforcement responses and public information
International coordination efforts on AI-enabled cybercrime
Assessment of the mental health impact of generative AI, especially among young people
Data on the prevalence of non-consensual deep fakes
Use expert and stakeholder engagement with committees to track harmful use of AI systems, including tracking any misuse of open source models
In committee inquiries:
Ask questions of social media representatives about their policies on manipulated media
Ask questions to the consumer protection-related agencies on how they are countering scams or fraud involving generative AI
Representation role:
Educate constituents on AI scams and phishing attempts, using meetings, newsletters, social media, local media
Gather public inputs on experiences, views and concerns around increasing use of generative AI through open meetings and surveying
Meet with education bodies and civil society to discuss measures to counter potential risks in society, including public awareness programmes
Support digital and AI literacy programs in education establishments, community centres, workplaces
Assist citizens to exercise their rights where they are affected by AI, connecting them with support groups
As a democratic leader:
Support a public discussion on societies with advanced AI, addressing inclusivity and ethical issues
Speak out on instances of AI misuse, using speeches in parliament, traditional and social media
Promote mental health awareness, including around risks from ‘synthetic relationships’
Collaborate with colleagues in international parliaments, sharing experiences, strategies and regulatory initiatives that mitigate harmful impacts of generative AI in society
Further reading and resources
Two useful sites compile and categorise incidences of harmful AI use in society: the AI Incident database and the AI, Algorithmic, and Automation Incidents and Controversies (AIAAIC) repository
The Centre for the Study of Existential Risk at the University of Cambridge has identified various ways in which AI can be used for malicious purposes, by rogue states, criminals and terrorists.
Check Point Research provides news and resources on examples of cyber crime using AI.
Organizations, including the Alan Turing Institute and the National Institute of Standards and Technology, are working on the bias problem