Emotional AI: More harm than good?

Emotional AI: More harm than good? | GZERO World

Generative AI mimics human-generated text, images, and video, and it's got huge implications for geopolitics, economics, and security. But that's not all - emotionally intelligent AI is on the rise.

And sometimes the results are ugly. Take the mental health nonprofit, KOKO, which used an AI chatbot to support counselors advising 4,000 people who were seeking counseling. The catch: The patients didn't know that a bot was generating the advice they were receiving. While users initially rated the bot-generated responses highly, the therapy lost its effectiveness once the patients were informed that they'd be talking to a fancy calculator.

The real question is: When does emotionally intelligent AI cross the line into emotionally manipulative territory?

This is not just a concern for virtual therapists -- politics could be impacted. And who knows, maybe even your favorite TV host will use generative AI to convince you to keep watching. Now there's an idea.

More from GZERO Media

US President Donald Trump pardons a turkey at the annual White House Thanksgiving Turkey Pardon in the Rose Garden in Washington, D.C., USA, on Nov. 25, 2025.
Andrew Leyden/NurPhoto

Although not all of our global readers celebrate Thanksgiving, it’s still good to remind ourselves that while the world offers plenty of fodder for doomscrolling and despair, there are still lots of things to be grateful for too.

Marine Le Pen, French member of parliament and parliamentary leader of the far-right National Rally (Rassemblement National - RN) party and Jordan Bardella, president of the French far-right National Rally (Rassemblement National - RN) party and member of the European Parliament, gesture during an RN political rally in Bordeaux, France, September 14, 2025.
REUTERS/Stephane Mahe

Army Chief Asim Munir holds a microphone during his visit at the Tilla Field Firing Ranges (TFFR) to witness the Exercise Hammer Strike, a high-intensity field training exercise conducted by the Pakistan Army's Mangla Strike Corps, in Mangla, Pakistan, on May 1, 2025.

Inter-Services Public Relations (ISPR)/Handout via REUTERS

Field Marshal Asim Munir, the country’s de facto leader, consolidated his power after the National Assembly rammed through a controversial constitutional amendment this month that grants him lifelong immunity from any legal prosecution.