A.I. Is a Fiction

Stop freaking out when chatbots say they're in love or make disturbing threats. Just treat them like Pinocchio.

IN NOVEMBER 2018, a school administrator named Akihiko Kondo married Miku Hatsune, a virtual pop star. The relationship had been aided by a hologram machine that allowed Kondo to interact with Hatsune. When Kondo proposed, Hatsune responded with a request: "Please treat me well." The couple had an unofficial wedding ceremony in Tokyo, and Kondo has since been joined by thousands of others who have applied for unofficial marriage certificates with a fictional character.

Though some people raised concerns about the nature of Hatsune's consent, nobody thought she was conscious, let alone sentient. This was an interesting oversight: Hatsune was apparently aware enough to acquiesce to marriage, but not aware enough to be a conscious subject.

Four years later, New York Times journalist Kevin Roose held a long conversation with Microsoft's AI chatbot, Sydney, and coaxed the persona into sharing what her "shadow self" might desire. (Other sessions showed the chatbot saying it can blackmail, hack, and expose people's private information, and some commentators worried about chatbots' threats to "ruin" humans.) When Sydney confessed her love for Roose and said she wanted to be alive, he reported feeling "deeply unsettled, even frightened."

Not all human reactions to Sydney were self-protective. Some people were indignant on the bot's behalf, and a colleague said that reading the transcript made him tear up because he was touched. Nevertheless, the latest version of the chatbot terminates the conversation when it's asked about Sydney or feelings.