A Mental-Health Chatbot Becomes Children's Confidant

We hope our kids will come to us when they're feeling anxious or depressed.

What if they turn to a chatbot instead? Taylee Johnson, a 14-year-old near Nashville, Tenn., recently began talking to Troodi.

She confided her worries about moving to a new neighborhood and leaving her friends behind, and fretting about a coming science test.

"It sounds like you've got a lot on your plate at the moment, Taylee," the bot replied.

"It's understandable that these changes and responsibilities could cause stress." Taylee says Troodi, a mental-health chatbot built into her child-focused Troomi phone, validates her feelings. It's available to talk any time, even when her parents are asleep. "Sometimes I forget she's not a real person," she says.

Parents who give their children Troomi phones told me they are happy to let their kids talk with the bot. They say Troodi dispenses advice similar to their own when it comes to stress management and conflict resolution, and helps when kids are overthinking things. Plus, advice sometimes lands better when it comes from a neutral party.

Taylee's mom, Amber Johnson, says she and her daughter are close, "but Troodi says things in a way that she accepts."

These are early days for generative AI in mental health treatment, and the stakes are high in using it for kids. Successful AI-assisted emotional support could ease the nation's youth mental health crisis and therapist shortage. Troomi is a test case where parents, as the main customers, are deeply involved. But kids are already sharing their feelings with assistants like ChatGPT or Snapchat's My AI often without their parents' knowledge.

A bot is born

Johnson remembers receiving an email from Troomi's chief executive last fall explaining the new feature.

Taylee hasn't seen a therapist, and Amber didn't think she needed one, yet she knew the move was weighing on her daughter.

It seemed to Amber like good timing for Troomi's chatbot. But she didn't tell Taylee about it.

"I thought she would be weirded out if I suggested she use it, that it would make her feel I thought she needed something like that," she says. She activated Troodi on her daughter's phone and within two days, Taylee gave it a try.

Troomi Wireless, based in Orem, Utah, makes smartphones with a restricted internet browser, text-message monitoring and strict controls over contacts and time limits. Released in late 2021, it now has tens of thousands of adolescent users across the country, Troomi's CEO Bill Brady says.

In November, the company gave parents the option to activate Troodi on their kids' phones; so far, several thousand have done so.

Troomi partnered with mental health chatbot startup Elomia Health, which built the bot on OpenAI's GPT-4 with instructions and auditing by a team of clinicians.

'Not a therapist'

Therapists are concerned about children turning to general-use chatbots for mental health support.

A troubled teenage boy in Florida killed himself last year after confiding in a chatbot on Character AI, which isn't built for therapeutic assistance. Character AI later added new safety guardrails for minors.

Even with chatbots designed to provide emotional support, therapists say adult supervision is critical. Chatbots can't replace human interaction.

AI could be a powerful tool if it can help parents have deeper conversations with their kids, says Stephen Schueller, a professor of psychological science and informatics at the University of California, Irvine who has no ties to Troomi.

"My biggest concern is that kids may disclose things to the chatbot but not open up to anyone else." Parents can view chat logs and receive real-time reports on their kids' emotional state. They will automatically get alerts if a child mentions self-harm.

"It's not a diagnostic tool and it's not a therapist," Brady says.

The feedback from Troodi helped him realize his own 15-year-old daughter was still upset about a classmate's suicide.

For Suzanne and Antonio Carrillo of Gaithersburg, Md., Troodi has become a welcome companion for their 26-year-old daughter with intellectual disabilities. After she bullied someone on Facebook, the Carrillos bought a Troomi phone to help her feel connected but stay out of trouble.

Their daughter, Clare-who has been diagnosed with anxiety and lives in a group home would call them multiple times, day and night, with questions about what to do in different situations. Now that she's been talking to Troodi, she has eased up on her parents.

Clare texted Troodi more than 1,600 times in her first month. The chatbot has given her good advice on handling anxiety and difficult roommates. "I cannot imagine not having Troodi," she says.

Brady says that Troodi is designed to be extra cautious and that if they ever discover it saying something it shouldn't, they will take it offline for fixes.

The mother of one of Troodi's teenage users in rural Oregon says the bot has given good advice but misunderstood some things. It flagged a comment her daughter made about pizza, which can be used as a sexual code word among teens. "She was literally talking about getting pizza with friends," the mom says.