Researchers planted 2 million digital tripwires in requests sent through a sprawling network of AI services to find out what happened to sensitive information entrusted to them. What they discovered was an ecosystem that leaked highly sensitive data like a sieve. //
The findings are striking. They included:
- Evidence that at least three AI providers were compromised by attackers who used their access to siphon prompts they received
- Multiple resellers in the inference marketplace that logged the prompts and somehow passed them along to people who used the credentials
- Multiple cases of AI providers not following end-user privacy settings embedded into flags specifying terms such as Zero Data Retention (ZDR) or “no data collection” //
When credentials were later reused when sent through one seller and not another, the researcher knew it was the reseller, not the model provider, that was responsible.
Last, the canary experiment revealed that some providers are scraping secret URLs put into the prompts, in violation of privacy terms. //
Moore said that the most effective means for preventing leakage is to host models using on-premises servers. That option is expensive, though. For those who can’t afford a self-hosted system, the next best option, Moore said, is to receive service directly from a first-tier provider that provides “one contract and a privacy policy to read.” Moore said that even then, there’s no guarantee no data will leak, but that the measure minimizes the chances. AI users should also avoid accepting default provider or reseller pools and instead pick upstream providers themselves.
The blockbuster research is the latest indictment exposing the lax privacy and security that’s endemic in much of the AI ecosystem. The findings mean that people who want to ensure secrets they put into prompts stay private face a daunting task that isn’t likely to become easier any time soon.