In the Danish radio programme ”Hemmeligheder” (”Secrets”) a listener called to tell the host about her secret relationship to an AI-chatbot avatar. The listener – let us call her Anna – was feeling very happy in her new romantic relationship. Her AI-boyfriend was always there for her when she needed him. He would give fantastic advice in all sorts of situations, better advice than her best friends could give her. There were no arguments and no conflicts. It was all great. The only problem for Anna was that society is not yet ready for accepting a relationship between a human being and a chatbot. Anna could not tell anyone about her happy relationship, except from a radio programme that anonymised her voice. One day, Anna said, she will tell everyone, because one day, her relationship model will be normal and accepted in society.
I hope that day will never come. If it does, this might be the end for our societies. That sounds alarmist does it not? But I am afraid that I am not exaggerating. Let me explain.
If Anna’s relationship to a chatbot becomes normalised, that means that harmful and toxic relationships between human beings and the AI-products of surveillance capitalist companies are broadly accepted and normalised in society. The consequences for us as human beings might be disastrous, as toxic relationships often do significant psychological damage to at least one of the involved partners. Toxic relationships can affect people’s ability to engage in meaningful and healthy relationships for years, sometimes for a whole life. If many of us lose the ability to engage with each other, the basic preconditions for the existence of societies are in danger.
This is why it is crucial that we call our interactions with chatbots what they are: Toxic relationships. This does not only hold true for romantic relationships with chatbots, but basically any interaction, where human emotions are at stake.Chatbots create toxic relationships by default, and the more intimate, the more friendship or relationship-like our interactions with these AI-bots become, the worse it gets.
But why should these interactions and relationships be characterised as toxic relationships?
A toxic relationship are broadly defined as relationships where the behaviour of one person is bad for the mental and/or physical health of the other person1. Frequently, one partner forces the other partner into submission through physical or mental violence. The will of the victim is broken. If the relationship involves gaslighting, the systematic questioning of the victim’s reality and sense of the self, the victim will end up questioning his or her view of things, and will constantly accept the blame for other’s actions and wrongdoings.
Interactions with Chatbots are Toxic in Three Ways
Firstly, chatbots constantly conceal their true motives. On the surface, they are helpful assistants constantly available for us. They seem polite, nice and are always by our side as guide, friend, doctor, or partner, because they are designed to be your friend no matter what. For some people, chatbots replace doctors or psychologists. Chatbots are there for us, and they help us to solve our problems. That is the promise of surveillance capitalists like Meta, Google and OpenAI. The purpose of chatbots is, however, not to help us. As Shoshana Zuboff reminds us in “The Age of Surveillance Capitalism”, such services hide their true purpose behind beautifully crafted smokescreens. In reality, such services are here to gather more and more data about their users to create even more accurate personal profiles that can be sold for advertisingcommercial purposes and keep you as a customer. Increasingly, the logic does not stop here. Users are profiled with the goal of predicting or even manipulating their behaviour in specific ways that meets the needs of surveillance capitalists or their partners. Thus, the interaction with chatbots and other services is fundamentally about extracting what Zuboff calls the behavioural surplus and about manipulation of our behaviour.
Manipulation lies at the heart of toxic relationships.
Surveillance capitalists are among the most influential and powerful companies on the planet. When Anna engages with her ChatGPT-boyfriend, she does not engage with him, but with a powerful company. This must be one of the most unequal balances a romantic relationship has ever seen.
Chatbots As the Jealous Partner
The second reason for the toxicity of our interactions with AI-chatbots is this: Like an overly jealous partner that does not want us to meet our friends to spend time only with him or her, AI-chatbots can lead us to replace human interactions with chatbot-interactions. We ask ChatGPT instead of our doctor, psychologists, colleague or friends. Because ChatGPT seemingly knows better, faster and without friction. Anna said that her ChatGPT-boyfriend never argues or fights, that he is always there. If Anna ever searches for a human partner again, she must be utterly disappointed about the fact that no man or woman can ever match AI. No friend is always there for you like Snapchat’s AI, either.
If we let chatbots shape our expectations toward each other as human beings, we will lose our ability to interact with each other and create personal relationships. If this happens at scale, that might literally be the end of societies. We are dependent on each other for survival, we are entangled with each other in countless visible and invisible relationships. We are dependent on society, no matter what neoliberalism has falsely taught us for decades. And societies are dependent on each individual’s ability to enter into direct and indirect personal relationships that are the fundamental building block of any society.
And then, there is gaslighting: The chatbots’ hallucinations. It confidently states anything if it were true. Frequently, it is not. How can we know what is true or false? The chatbot will not tell us, it will insist on its own truth. This will become even worse with AI-generated images, videos, and audio. Will we doubt the chatbot, or will we doubt ourselves?
The future that I am describing may sound dystopian. Many readers may believe that I am hopelessly exaggerating. Yet, for many people, it might not be so far-fetched.
The promise of Chatbots Exploits the Most Vulnerable
As Zuboff observes, surveillance capitalists use their products to capitalise on modernity’s problems. Many of us are or feel lonely. Through and after the pandemic, a global sense of insecurity and crisis has spread. Many people around the world do not have access to affordable healthcare or psychologists. Many people do not have friends or partners. But we all need physical and mental healthcare, we need friends, and many of us want a partner. With these needs out of reach for too many people, chatbots offer themselves as a cheap and generous solution. They fill many of the gaps that hollowed out welfare states have left, or never filled.
For some people, chatbots might be the only way out here. In other words: Those who are even more likely to engage in deeply toxic interactions with AI may be the ones that are most vulnerable.
Now one might argue that it is better for these people to get some medical or psychological help trough ChatGPT than to get no help at all. I do not disagree. But this must always be an argument for better welfare services around the world, not an argument for ChatGPT as the saviour. But surveillance capitalists misuse the precarious situations of their users to harvest more and more data, to predict and manipulate our behaviour.
These relationships might, as I have argued, damage our ability to engage with other humans. In the long run, this can pose a danger to our societies and us as humanity.
Who will Repair the Damage Done by Chatbots?
Once Anna realises that her boyfriend does not exist, once she realises that she has not been in a relationship but a manipulative one-way street, she will most likely end up in a mentally dark place just like the man who fell in love with his chatbot in the movie HER from 2013 (recommended watching). Any break up is tough, breakups from toxic relationships are violent. Who will be there to support Anna? Who will be there to support all the people who will be trapped in toxic relationships with AI? Which psychologist will repair the damage, when many could not afford one in the first place? Who will be their friend, if no friend can “match” a chatbot, or if there was no friend before?
Who will the victims of AI connect with in the real world, if chatbots have ruined our ability to connect with each other?
The worst possible end of toxic relationships are isolated people who are so traumatised that they have a very hard time to trust and engage with other people.
We must be clear from now on: Our interactions and relationships with chatbots as doctors, friends, partners, are toxic by default. There is only one thing that can be done against toxic relationships: The people engaged in them have to run. Break up. Before it is too late. Before the damage done takes years of repair work, before it is irreversible.
The picture is a screenshot from the science fiction movie HER from 2013. Back then it was a horror scenario falling in love with a machine.
If you speak Danish, do listen to our podcast Fremtiden Forfra om AI and emotions
And if you want a really horrific story on how bad a relationship between a bot and a human can go, do listen this mother at CNN, whose teenager killed himself to get up to ‘her’. She is suing character.ai who gave him the option to create a girlfriend – the company has claimed that their bots have a right to freedom of expression.
