Privacy experts sound the alarm over Microsoft’s latest AI tool

A demonstration of Microsoft's Recall feature on a Surface Pro is pictured following the Microsoft Briefing event in Redmond, Washington on May 20, 2024. (Jason Redmond / AFP)

Microsoft’s buzziest new AI feature is raising concerns that it could potentially be misused in the wrong hands.

This week, the company showed off a new tool called Recall for Windows computers that acts as a personal “time machine,” allowing users to quickly pull up anything that’s ever been on screen, such as documents, images and websites. It’s different from a keyword search; the tool regularly saves screenshots of the user’s screen and stores them directly on the device. It then uses AI to process the data and make it searchable.

For example, if someone previously searched for a green dress or the name of a local ice cream shop, they can ask the feature to “recall” anything in their history that was shown on screen.

Although so-called semantic search is a big step forward for AI, it comes at a time when the industry is moving so quickly and government regulators, companies and consumers are still figuring out how to use the technology responsibly.

Jen Golbeck – a professor of AI at the University of Maryland who focuses on privacy – said the recall feature could pose a potential “nightmare” if the device falls into the wrong hands.

“Stuff may stay on your device, but that doesn’t mean people can’t get to it,” she said. “You won’t have an option to protect yourself even if you use incognito mode or clear your history because the tool has access to everything that’s been on your screen.”

The UK’s independent regulator for Data Protection and Freedom of Information, the Information Commissioner’s Office (ICO), told CNN it is investigating the tool “to understand the safeguards in place to protect user privacy.”

“We expect organisations to be transparent with users about how their data is being used and only process personal data to the extent that it is necessary to achieve a specific purpose,” the ICO said in a statement.

Microsoft did not immediately respond to a request for comment. CEO Satya Nadella told The Wall Street Journal in an interview ahead of Monday’s launch that web searches must only be done on Microsoft’s Edge web browser and that the screenshots never leave the user’s computer.

“You have to put two things together: This is my computer and this is my Recall – and it’s all being done locally,” he said.

Geoff Blaber, CEO of market research firm CCS Insight, said that makes the issue less concerning.

“The backlash by some to this feature isn’t surprising, but it’s an overreaction given that the data stays exclusively on the device and the user has full control,” Blaber said.

Someone can decide whether to turn the feature on during the device setup process and can customize and blacklist which apps and websites Recall can access.

“These controls suggest the feature has been built with security and privacy at its core,” he said. “Recall won’t appeal to everyone but the utility provided is likely to be significant.”

Potential for malicious uses

But Golbeck cited times when that protection might not be enough, such as a journalist in a hostile country, a person trying to get out of an abusive relationship, someone searching for sensitive medical information or an employee whose boss wants to track them. She believes issues around the tool will likely impact the workplace the most.

“We know jobs monitor what we’re doing on their devices, but a lot of people will do a personal thing on their work computer and all of that will be archived and visible to an IT department, even if it just stays on device,” she added.

Golbeck said technology, despite the benefits, is too often developed without the consideration of malicious uses.

“What we’ve seen over the last 15 years is every single one of those potential malicious use cases comes to fruition in some way or another,” she said. “People will want to use this cool feature without understanding the privacy risk – that there will be a permanent record of everything you do.”

Similar reactions to features such as biometric passwords, such as fingerprints, facial recognition and iris scans, have emerged in the past. Blaber said to overcome some of the initial concerns, Microsoft and its partners will need to prove the value, security and privacy of the product in the real world.

Still, Michela Menting, a senior research director at ABI Research, believes the feature is a “step backwards” for privacy.

“The argument that hackers need physical access to even be able to manipulate Recall is short-sighted at best because there are plenty of highly ingenious ways threat actors could exploit such a valuable tool,” she said. “All it takes is time and effort to find a way.”