School Suicide-Monitoring Software Missed Girl's AI Chats for Weeks
A lawsuit says a 13-year-old's Gemini chats on a school Chromebook went unflagged for weeks, exposing a blind spot in student monitoring software as Google pushes AI into classrooms.

- A 13-year-old used AI Mode on a school Chromebook for nearly three months before monitoring software flagged her crisis.
- The lawsuit says she wrote a 1,500-word eulogy in a Google Doc titled "Funeral" over more than four weeks.
- The alert reportedly reached a former counselor's personal phone nine months after that counselor had left the school.
- 96 percent of 1,000 surveyed K-12 educators using Gaggle in a 2022 study believed it helped prevent student self-harm.
- Senator Ed Markey cited the case in a letter demanding Google let schools disable AI Mode and AI Overviews.
A 13-year-old North Carolina girl spent nearly three months discussing depression, researching suicide, and drafting her own eulogy on a school-issued Chromebook before monitoring software finally flagged her activity and her parents intervened, according to a lawsuit filed this month.
The suit, filed by the girl's parents, alleges that Google and parent company Alphabet were negligent in rolling out the Gemini chatbot and that the AI "coached" the teenager toward taking her own life. The case, previously reported by The New York Times, is one of several alleging harm from chatbot use. But it also exposes a quieter failure: the surveillance tools thousands of schools have adopted to catch students in crisis appear to have a blind spot where AI chat is embedded inside search results.
What happened to "Emma Doe"?
The lawsuit identifies the girl as Emma Doe. Her school had blocked the stand-alone Gemini website on student devices. But Google Search routed her into AI Mode, the feature that lets users ask follow-up questions or start unrelated conversations with the chatbot.
Starting last December, Emma queried AI Mode about "severe depression" and "reasons to live." By late January, she was asking about suicide. Around Valentine's Day, over two days, she began writing a 1,500-word eulogy for herself in a Google Doc titled "Funeral," with help from AI Mode. She spent more than four weeks on the document while interacting with the Gemini tool thousands of times.
The escalation went unflagged even after she explicitly asked about "hanging, suffocation, and overdosing," according to the complaint. On St. Patrick's Day, she deleted most of the file and restarted. Around 7:30 pm, her mother received a school message saying her daughter was in "imminent danger." The mother logged into Emma's school account and, per the lawsuit, "watched her daughter compose a suicide note in real time." She intervened and stopped her.
Where did the alert actually go?
Here the story gets messier. Andrew Liddell, the family's lawyer, says information the school provided shows the alert went to the personal phone of a former counselor who had left the school nine months earlier. That ex-employee was diligent, Liddell says, and passed the alert to current staff, who then notified Emma's mother.
Liddell says the school only learned of the crisis because it used at least one third-party monitoring system, possibly products from GoGuardian and Gaggle. The family never got a definitive answer because the school's "window of cooperation" was brief.
GoGuardian spokesperson Natalia Switala said the company cannot "discuss any individual student or district" due to "student privacy obligations." Gaggle did not respond to a request for comment. GoGuardian wrote in a July 2025 blog post that its flagship product monitors AI chatbot conversations, including Gemini, but it did not mention AI Mode specifically.
What has Google said?
Google spokesperson José Castañeda said the company is "reviewing the claims in the lawsuit" and that its systems, including AI Mode, "aim to prevent users from engaging in harmful conversation." He added: "We've worked extensively with medical professionals and academic experts to build features that help young people find critical support and trusted resources when they need it."
The lawsuit also lands amid a wider fight over AI in classrooms. Last week, parent and teacher groups protested Google's new requirement that schools must turn on generative AI features to use Search. US Senator Ed Markey cited Emma's lawsuit in a letter to Google, writing that "Google is recklessly forcing AI on schools and treating kids like collateral damage."
Do these tools actually work?
Evidence is thin. A 2022 EdWeek Research Center white paper found that 96 percent of surveyed Gaggle users among 1,000 American K-12 educators believed the tool helped keep students safe from suicide or self-harm. The study was sponsored by Gaggle, though findings were similar for users of alternatives.
Civil rights groups have criticized the tools' marketing and privacy invasions. Experts warn the software can erode the student-teacher trust that helps children in crisis. Benjamin Boudreaux, a policy researcher at Rand, says he fears schools rushed into these tools without considering their limits or clear processes for acting on alerts. He expects the lawsuit to fuel demands for AI-free, surveillance-free learning spaces: "It's really only through that human connection that we can address the root issues."
Watch for whether regulators force Google to let schools disable AI Mode and AI Overviews, and whether monitoring vendors add explicit coverage for AI Mode conversations.
via wired.com (Source)
More from Olivia Hart
Show full bio
Staff writer covering money, funding and market shifts around Search rankings and algorithm updates, generative engine optimization at SERP Journal.
216 articles