Monitoring tools aimed at monitoring children’s behavior and preventing suicide are useful but have unexpected consequences – GIGAZINE

Some governments install monitoring software on mobile devices distributed to children for educational purposes. Proponents of the monitoring software say it “helps identify and help children with problems,” but the New York Times reports that there are some unforeseen problems that users may not expect. I am summarizing it.

The Brave New World of AI-Powered Self-Harm Alerts – The New York Times
https://www.nytimes.com/2024/12/09/health/suicide-monitoring-software-schools.html

Does spying on laptops really prevent high school suicides?
https://reason.com/2024/12/09/does-spying-on-laptops-really-prevent-high-school-suicides/

The New York Times featured an example of software installed in a school in Missouri. This software was designed to closely examine the content a child entered into a device and flag any problematic words, but it mistakenly flagged a girl who had written a poem as having committed self-harm. Police were called to the house, and the police came to the house because they thought the girl was at risk of harming herself.

Although the incident quickly became clear, the child was extremely upset and described it as “traumatized.” Examples of these “false positives” are endless, and things like hunting reports, historical research on the Ku Klux Klan, and even quotes from Oscar Wilde’s plays can be flagged incorrectly. It seems there was.

Federal law in the United States requires schools to use content filters on devices given to children. In addition to content filters that prevent you from seeing things you don’t want to see, implementing advanced input monitoring tools is optional, but according to the New York Times, nearly half of American elementary school students are being monitored. It means that it is a state. Technology related in the UKguidanceThe current situation is that filtering and monitoring are required.

Of course, there are cases where implementing monitoring tools has saved lives. In the case of a 17-year-old girl, an email she sent to a friend stating that she was thinking about suicide was detected and she was able to receive help from a counselor. The girl said, “Nowadays, my counselor is like a mother to me.”

There is also an example where a counselor rescued a girl by detecting the search content “How much medicine do I need to die?” and the girl later started working as a rescuer. The Neosho school district, which is a small town but has had a period when there was a relatively high number of child suicides, has introduced monitoring tools and has a system in place to protect children. There are many cases, but if you can save one child, it’s worth issuing a false alarm.”

The New York Times says, “The challenge with surveillance tools is accuracy. Then there’s the question of follow-up: how accurately can they detect real danger, and what kind of care can schools provide for children in crisis?” The question is whether it can be provided.”