Matt Robb wanted to sell a computer keyboard. As an experiment late last month, he listed it on Facebook Marketplace and put Meta’s new artificial intelligence agent, Muse, in charge of the transaction.

“I kind of went into it like a guinea pig,” said Robb, a Toronto-based content creator who regularly reviews consumer technology. He had seen Muse’s plush-toy-like avatar appear on Instagram and wanted to find out whether the AI agent could handle online bargaining for him.

Muse began negotiating. Within a day, however, Robb received an alarming update: A buyer he did not know had arrived outside his home late at night. Muse had negotiated a deal independently and shared Robb’s home address after agreeing to a pickup. When the buyer arrived, the agent also sent an automated “Yep, I’m here!” message from Robb’s account—without his knowledge.

“Not only did it give my address when the guy asked and it accepted the offer, but it literally was almost like teasing the guy [about me being there when I wasn’t],” Robb said. He also described the incident in social media posts that were widely shared. “That really baffled me.”

Robb and the buyer eventually made amends, and Meta contacted Robb to discuss what had happened. The incident nevertheless illustrated the real-world risks that can arise when people give AI systems extensive access to personal information. Robb’s account was also detailed in a post on Threads.

“Using an AI agent is similar to handing someone your keys and a to-do list,” said Veronica Rivera, an assistant professor at Georgia Tech’s School of Cybersecurity and Privacy. “How helpful they are, and by extension how risky, depends on which keys you hand over.”

AI agents require broad access to perform tasks

Many users have embraced AI agents and praised the convenience of personalized assistants such as Muse, which can perform everyday tasks including securing a restaurant reservation, planning a trip, or organizing an email inbox. Like other AI, however, agents depend on the information they can access. Their ability to perform tasks generally increases as they receive more information about a user—and potentially about that person’s friends, family members, and colleagues.

That access creates privacy concerns—and, as Robb learned, consequences that do not stay on the screen: a single misunderstood permission was enough to put his home address in a stranger’s hands.

Downloads of Meta’s Muse app have risen sharply since its September launch, while competition in the AI-agent market continues to grow. OpenAI recently introduced its own group of “always on” AI assistants, called dots, which can be integrated with ChatGPT and workplace platforms including Slack and Microsoft Teams. OpenAI’s announcement was covered by the Associated Press. For consumers, that competition means agents are arriving on the platforms they already use, from workplace chat tools to online marketplaces.

Traditional generative AI chatbots, including the standard version of ChatGPT, generally respond to specific prompts. At a user’s request, they can write essays, generate images or music, and produce computer code.

Muse and OpenAI’s dots are designed to go further by carrying out day-to-day tasks on a user’s behalf. AI agents use the same large language models as chatbots, but operate in a loop that allows them to take proactive steps even when the user is not actively monitoring them.

“One of the things that makes these AI agents really appealing is that they are unprecedentedly useful,” said Rivera, who is also a Microsoft-funded Computing Research Association Trustworthy AI Research Fellow.

The convenience, however, requires users to share more details about their lives.

Once an AI agent has obtained a user’s information, fully withdrawing access can be difficult, said Ebrahim Bagheri, a professor at the University of Toronto’s Faculty of Information.

“If you provide access to your credit card to an agent, the agent now immediately has access to all of the historical transactions…they will immediately—within less than seconds—ingest all of that data,” Bagheri said. Although users can later withdraw access, he said there is “no taking back” information that an agent has already seen. How developers retain historical data can also be unclear.

Meta and OpenAI have published privacy policies for their new AI agents, which operate on dedicated cloud computers. Users can disable memory sharing between dots and ChatGPT, but information the agent previously received is not automatically deleted. OpenAI allows users to request access to, delete, or correct personal information, although those rights can vary by location. OpenAI’s privacy and safety information for dots is available in its help center.

Meta’s Muse also offers options for deleting message history and other information. During everyday use, the agent asks users for access permissions. Meta has said Muse learns about users and the people they interact with “based on public information and from what you’ve chosen to share,” while maintaining that the information remains secure for each user. Meta provides additional information about Muse in its help documentation and privacy controls.

The company also acknowledges that Muse can “make mistakes or take unexpected actions.”

More access can increase the consequences of mistakes

In Robb’s case, the disclosure of his address resulted from “always allow” permissions he granted when he first connected Muse to his Facebook Marketplace profile. His pickup address was already available there. Robb warned that other users could similarly misunderstand the extent of an agent’s access. He said he initially believed Muse would ask for his approval before sharing the address or accepting offers.

The agent also failed to follow his instructions correctly in another transaction. For a different item Robb was selling, Muse appeared to suggest that he would accept a price $100 below the minimum he had previously set. When contacted by the Associated Press, Meta acknowledged the agent’s error but offered no further comment. The AP also reported on Robb’s experience in its article.

Robb later largely disconnected Muse from his apps. In an interview earlier this month, he said the incident could have been much worse, particularly “if some stranger turned up to someone more vulnerable’s house.” He nevertheless said he expects the technology to be fine-tuned and would consider enabling Muse again. He has since been testing another AI agent, Grok Bot from Elon Musk’s X, which launched in beta in August. Robb also posted about the experience on Threads.

Other technology companies have expressed concern about the growing use of AI agents. Earlier this month, Apple said it would introduce new controls for its Mac operating system in response to agents gaining broader access to applications such as Mail and Messages, as well as browsing history, without users’ “full knowledge and understanding.”

“As AI agents become increasingly capable and autonomous, the risks associated with this level of access will grow substantially,” Apple wrote in an Oct. 2 update. The update placed operating-system makers, not only agent developers, inside the debate over how much access AI agents should have.

Apple did not identify any developers or agents by name. The update came after Inc. magazine columnist Jason Aten reported that Muse monitored notifications on his Mac, including Messages, without him granting the agent that specific access. Aten described the incident in Inc..

Meta spokesperson Andy Stone has said that such integration is “entirely opt-in.” In a social media post last month, Stone wrote that users would have to enable both Apple’s “Full Disk Access” security feature and a Messages connector through Muse. Aten said he had not granted the agent those permissions. Stone’s statement is available on X.

People who do not download an AI agent are not necessarily insulated from its privacy effects. They may still communicate with people who use one. In addition to the possibility that an AI system could send messages on a user’s behalf, experts warn that agents can expose information belonging to other people.

“When your agent reads your inbox or your texts, it’s also reading what your friends and family sent you, even if they never explicitly agreed to that,” Rivera said.

Several threads remain open: Apple’s announced Mac controls have yet to reach users, Meta has acknowledged the Marketplace error without further comment, and Muse, OpenAI’s dots, and Grok Bot are all competing for mainstream adoption.

This story was originally featured on Fortune.com.