(Correct Alvarez & Marsal in paragraph to be a consulting firm, not a law firm)
By Elvira Pollina and Martin Coulter
MILAN/LONDON (Reuters) – Italy’s data protection agency said on Friday it banned artificial intelligence (AI) chatbot company Replika from using Italian users’ personal data, citing unintended consequences. Adults are at risk and emotionally vulnerable.
Replika, a San Francisco startup founded in 2017, provides users with custom avatars that they can talk to and listen to.
It leads among English speakers and is free to use, though it brings in about $2 million a month selling bonus features like voice chat.
“Virtual Friends” are marketed as improving the emotional well-being of users.
But Italian regulators said that by interfering with users’ emotions, “may increase the risk for individuals who are still developing or in a state of emotional vulnerability”.
Jen Persson, director of children’s privacy advocacy group Defend Digital Me, told Reuters that tools designed to affect children’s emotional or mental health should be classified as health products and therefore subject to strict safety standards.
“These tools are being used on children without much supervision or protection against potential abuse,” she said.
Italian regulators highlighted the lack of age verification mechanisms, such as filters for minors or blocking devices if users don’t explicitly state their age.
Replika breached European privacy regulations and processed personal data illegally because it could not be based, or even implied, on contracts that minors could not sign, the watchdog said.
Replika did not immediately respond to an email from Reuters seeking comment.
Robert Grosvenor, managing director of consultancy Alvarez & Marsal, said Italian regulators were unlikely to be the only European regulators considering action against companies like Replika.
“While age verification can provide a means of protecting some of the most vulnerable, it does not address the risks and harms that AI-based services and solutions may pose when left unregulated , potentially because of unconscious bias and discrimination,” he said.
Replika’s developer, the American company Luka Inc, must inform the Italian authorities for the implementation of its in Measures taken as a result of the requirements in ) days, up to 20 millions of euros ($80.21 million dollars), or, as stated, up to 4% of its annual worldwide turnover.
(This story has been corrected to say that Alvarez & Marsal is a consulting firm, not a law firm, in paragraph )
