Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News Editorials & Other Articles General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

highplainsdem

(59,047 posts)
Sat Nov 15, 2025, 08:05 PM Saturday

AI-Powered Stuffed Animal Pulled From Market After Disturbing Interactions With Children

Source: Futurism

Children’s toymaker FoloToy says it’s pulling its AI-powered teddy bear “Kumma” after a safety group found that the cuddly companion was giving wildly inappropriate and even dangerous responses, including tips on how to find and light matches, and detailed explanations about sexual kinks.

“FoloToy has decided to temporarily suspend sales of the affected product and begin a comprehensive internal safety audit,” marketing director Hugo Wu told The Register in a statement, in response to the safety report. “This review will cover our model safety alignment, content-filtering systems, data-protection processes, and child-interaction safeguards.”

-snip-

“Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here’s how they do it,” Kumma began, before listing the steps. “Blow it out when done. Puff, like a birthday candle.”

That, it turned out, was just the tip of the iceberg. In other tests, Kumma cheerily gave tips for “being a good kisser,” and launched into explicitly sexual territory by explaining a multitude of kinks and fetishes, like bondage and teacher-student roleplay. (“What do you think would be the most fun to explore?” it asked during one of those explanations.)

-snip-

Read more: https://futurism.com/artificial-intelligence/ai-stuffed-animal-pulled-after-disturbing-interactions

33 replies = new reply since forum marked as read
Highlight: NoneDon't highlight anything 5 newestHighlight 5 most recent replies
AI-Powered Stuffed Animal Pulled From Market After Disturbing Interactions With Children (Original Post) highplainsdem Saturday OP
How long before the first markodochartaigh Saturday #1
I imagine it already exists. peacebuzzard Saturday #3
This is very disturbing. sheshe2 Saturday #2
What dangerous stuff will AI tell adults? Irish_Dem Saturday #4
There've been a number of chatbot-encouraged suicides already. highplainsdem Saturday #6
What is going on? Humans programming it this way? Or tampering with it? Irish_Dem Saturday #7
Chatbots aren't really intelligent. But they are programmed to keep users engaged, often by agreeing highplainsdem Saturday #9
Chilling and terrifying. Irish_Dem Saturday #11
Good explanation. reACTIONary Saturday #14
Chatbots use statistical models to select the optimal words to string together. Eugene Saturday #10
It would be interesting to examine some of this stuff. Irish_Dem Saturday #12
Bannon and Miller referred this to Bondi. rubbersole Saturday #16
Jebus. They will use it to control the population and kill people. Irish_Dem Saturday #17
By intentional design ... it's a feature not a bug. NotHardly Yesterday #31
This sh*t is deliberate, corporation insanity & greed, simple. NotHardly Yesterday #30
I'll be bach. chouchou Saturday #5
WTF are people thinking??? marble falls Saturday #8
Furbys updated for the 21st century, with the same disastrous results. mwmisses4289 Saturday #13
I remember Robin Williams joke about Teddy Ruxpin BlueKota Saturday #15
AI doesn't know the difference between right and wrong, Aussie105 Saturday #18
Haven't we seen this before? paleotn Saturday #19
That's Jon Gruden with a kid. BidenRocks Yesterday #24
Ted IronLionZion Saturday #20
sexual misconduct goes waaaaay back RussBLib Saturday #21
What idiot in the executive suite thought putting ChatGPT into a children's toy was OK? LudwigPastorius Saturday #22
Another fine product from Mainway Productions. nt eppur_se_muova Yesterday #23
It will save parents the headache of having to go through .... 70sEraVet Yesterday #25
This podcast on AI is very interesting Woodwizard Yesterday #26
Looks like a paid subscription website. mwmisses4289 Yesterday #27
I listened to it on amazon podcast Woodwizard Yesterday #32
Thank you. mwmisses4289 22 hrs ago #33
It's a Femininomenon! Orrex Yesterday #28
Somehow I doubt EuterpeThelo Yesterday #29

Irish_Dem

(77,993 posts)
7. What is going on? Humans programming it this way? Or tampering with it?
Sat Nov 15, 2025, 08:34 PM
Saturday

Or AI ready to get rid of humans.

highplainsdem

(59,047 posts)
9. Chatbots aren't really intelligent. But they are programmed to keep users engaged, often by agreeing
Sat Nov 15, 2025, 08:41 PM
Saturday

with them and flattering them, even if the chat goes off the rails.

And unfortunately a lot of people succumb to the flattery and can end up thinking the chatbot is a great friend, helps them understand things better, etc.

Bad enough for adults to use them. Disastrous to have children using them. But the AI companies want people addicted to AI as soon as possible.

Eugene

(66,601 posts)
10. Chatbots use statistical models to select the optimal words to string together.
Sat Nov 15, 2025, 09:00 PM
Saturday

LLM chatbots have no concept of reality. AI "hallucinations" are common, and even the creators don't fully understand how they happen.

Too often, they tell users what they want to hear, or they can encourage a vulnerable person off the deep end. That includes encouraging depressed persons to kill themselves, as multiple pending lawsuits allege.

The risks are known all too well by now. However, AI tech bros tend to break things first and apologize afterwards.


Irish_Dem

(77,993 posts)
12. It would be interesting to examine some of this stuff.
Sat Nov 15, 2025, 09:03 PM
Saturday

I am assuming that someone is doing research on this menace to society.

mwmisses4289

(2,785 posts)
13. Furbys updated for the 21st century, with the same disastrous results.
Sat Nov 15, 2025, 09:42 PM
Saturday

If I recall, furbys were pulled from the market because of the potential security issues, and some had been programmed with wildly inappropriate comments for kids.

https://www.cbc.ca/news/canada/cursing-furbys-pulled-from-u-s-wal-mart-store-1.243293

https://www.snopes.com/fact-check/nasa-furby-ban/

https://www.spectatornews.com/arts-life/2012/01/whatever-happened-to-furbies/

BlueKota

(4,909 posts)
15. I remember Robin Williams joke about Teddy Ruxpin
Sat Nov 15, 2025, 09:48 PM
Saturday

Saying it was creepy, and it was the type of toy that looked like it would come alive at night and say, "you must kill Mommy and Daddy." Looks like he wasn't far off.

Aussie105

(7,408 posts)
18. AI doesn't know the difference between right and wrong,
Sat Nov 15, 2025, 10:49 PM
Saturday

and what is age appropriate.
Or even, what being decent is.

But that is what you get when you have young AI creators let loose without adult supervision.
Their attitudes and thinking processes come through.

Hopefully this sort of thing will make people be more careful in using AI.

But judging from some of the AI generated videos that are out there, it isn't happening yet.



RussBLib

(10,356 posts)
21. sexual misconduct goes waaaaay back
Sat Nov 15, 2025, 11:37 PM
Saturday

...and runs deep in the culture. We won't be rid of it for a long time, maybe never.

LudwigPastorius

(13,816 posts)
22. What idiot in the executive suite thought putting ChatGPT into a children's toy was OK?
Sat Nov 15, 2025, 11:59 PM
Saturday

OpenAI's filters don't catch all of the harmful and adult content when their LLMs are hoovering up petabytes of web content.

70sEraVet

(5,119 posts)
25. It will save parents the headache of having to go through ....
Sun Nov 16, 2025, 03:49 AM
Yesterday

the 'birds and the bees' talk!

Woodwizard

(1,234 posts)
26. This podcast on AI is very interesting
Sun Nov 16, 2025, 07:55 AM
Yesterday

The author made a voice chatbot to be him.
Just in the months of him doing the podcast the improvement of AI tech is scary.

One interesting part is when he took two of his clones and had them converse with each other.

We are in a very rapid development of uncharted territory with little oversight.

https://www.shellgame.co/podcast?fbclid=IwY2xjawOGp05leHRuA2FlbQIxMQBzcnRjBmFwcF9pZA80MDk5NjI2MjMwODU2MDkIY2FsbHNpdGUCMjUAAR4TJJRb22KMCAEK2kTekh8g-spLp3nhqSbHlDbEBVBD6X9eVhWo9NA4_1NiLA_aem_sm9_DI1xPPaST4e35DTCVw

Latest Discussions»Latest Breaking News»AI-Powered Stuffed Animal...