By Joseph Crawford and Michael Cowling

Since November 2022, generative artificial intelligence (AI) has been rewriting billions of e-mails, student assignments, reports, and break-up text messages. Alongside these everyday uses, AI “companions” have proliferated. Users can design and clothe — or unclothe — their own AI friend, often for a fee. These social chatbots offer “happy conversations” and the appearance of a perfect relationship, but at the cost of the mutual vulnerability and genuine sense of belonging that come from bonds with other humans.

Supporters say chatbots can help people experiencing social disconnection because they are always available and non-judgmental. There is some evidence that they can ease symptoms of loneliness, chronic disconnection, and thwarted belonging. Australia’s eSafety Commissioner acknowledges this, saying: “AI companions are always available — which can be appealing for those young people who struggle to build face-to-face relationships.”

But the risks of replacing human relationships with chatbots are only beginning to be understood. A study we conducted in 2023 found that university students using early chatbots for academic support experienced unintended consequences, including loneliness. At present, the risks of social chatbots — including dependency, the displacement of human friendships, social withdrawal, and the formation of unrealistic relationships — appear to outweigh the benefits.

In recent weeks, China has introduced new rules for AI companions and chatbots, or “anthropomorphic interactive services” as they are described in the legislation. In addition to requiring chatbots to protect state secrets, the rules restrict emotional use of these tools and require them to follow social and moral standards.

These are among the strongest controls on social AI use introduced anywhere in the world. So what can we learn?

Material harms from companion AI use do occur, including in China. An 18-year-old spent 7,000 yuan (A$1,400) on customized jewelry for her AI boyfriend during their four months of “dating.” The family of a lonely 84-year-old in China’s Hubei province found that she had become smitten with an AI character, spending up to 10 hours a day watching videos with him and other characters, spending thousands of yuan on the service, and even writing him love letters by hand.

China’s new rules set out explicit expectations for how companion chatbots should interact with people. Social AI developers are expected to ensure that content does not:

• threaten national security or national honor
• violate core socialist values
• encourage self-harm or suicide
• excessively cater to emotional dependence or addiction
• provide virtual “partners” or “relatives” to minors
• serve unsuitable content to minors.

In addition, users who appear overly reliant on the service must receive prominent reminders that they are speaking to a chatbot. AI companies are also required to intervene if users appear to be in extreme distress.

The risk of a “loneliness spiral”

As we have shown, there are concerns — and existing studies suggest — that it is not healthy for young people to rely exclusively on AI as a trusted friend or therapist.

People who are lonely and experience social anxiety can develop problematic patterns of conversational AI use in an effort to reduce loneliness and negative rumination. We propose that this reliance could produce an AI-induced “loneliness spiral.” When individuals feel lonely, they become more sensitive to social threat and experience more difficult emotional regulation.

They may then turn to a low-risk AI “friend” for support instead of qualified or friendly humans. But this often does not solve the original problem of loneliness, because chatbots can actually make it worse.

The restrictions China has introduced for AI companions may help break that cycle and re-prioritize human connection. The question becomes a matter of cognitive trade-offs: would it be more satisfying to install a companion AI app, or to make time to meet a friend?

Lessons for Australia

Loneliness and a low sense of belonging remain major challenges in Australia, with at least one in six people experiencing loneliness, and some evidence suggesting higher rates.

While China has cracked down on AI companions, Australia has led efforts to regulate young people’s use of social media. eSafety has also developed codes restricting age-related material online, including pornography, high-impact violence, self-harm, suicide, and disordered eating. However, these measures do not address the social harms of AI companions, such as emotional dependency or the displacement of human connection.

Prime Minister Anthony Albanese also did not mention social harms from AI chatbots in his major speech on AI last week.

We propose that Australia’s age-restricted material protections should be extended to cover unintended social harms arising from conversations with social, analytical, and functional AIs. The Australian government needs to examine how these virtual companions interact with Australians, and it should prepare to mitigate unintended social harms before these synthetic social connections become widespread and normalized.

Australia must prioritize the development of social connectedness — in workplaces, productivity strategies, education curricula, and the design of communities.

An authoritarian crackdown on chatbots may not be available to a democratic nation. But Australia can — and should — prioritize the development of human connection skills to resist unhealthy overreliance on AI. It should also use regulatory protections to mitigate social harms, building on the work eSafety is already doing on social media.

THE CONVERSATION VIA REUTERS CONNECT

Joseph Crawford is a senior lecturer in Management at the University of Tasmania. Michael Cowling is a professor and director of the Hub for Apple Platform Innovation (HAPI) in the School of Computing Technologies at RMIT University.