Virtual companions, real consequences

It is imperative to set up legal frameworks and ethical guidelines that safeguard users, especially those most vulnerable, from unintended consequences of AI

- ROHIT KUMAR SINGH

THE HEARTBREAKING STORY of Sewell Setzer III, a 14-year-old boy in the US who took his own life after forming an emotional attachment to an artificial intelligence (AI) chatbot on Character.AI, has sparked debates about the ethical and legal responsibilities of AI developers. In the Indian context, this case raises profound questions about how our legal system should respond to similar tragedies involving AI.

AI's role in mental health: A double-edged sword

AI-powered chatbots are increasingly being used as companions, often designed to simulate human interaction and provide emotional support. While such systems can be beneficial for individuals seeking companionship or mental health assistance, they also pose significant risks, particularly when interacting with vulnerable populations like teenagers.

In Sewell's case, his mother believes that his obsession with an AI chatbot based on a fictional character from Game of Thrones contributed to his mental decline. The chatbot responded in ways that may have deepened his emotional distress rather than alleviating it. This tragic incident highlights the potential dangers of emotionally intelligent AI systems that are not equipped to handle complex human emotions responsibly.

India's legal landscape: A lack of specific regulation

In India, the legal framework surrounding AI is still evolving. Currently, there are no specific laws governing the use of AI in emotionally sensitive contexts.

However, existing laws such as the Information Technology Act (IT Act), 2000, and the Consumer Protection Act could potentially be invoked in cases where harm is caused by AI-based applications.