We explore the privacy of machine learning systems, and show how many standard components of the ML pipeline create side-channel vulnerabilities that leak private user data.

We propose an evaluation methodology for AI systems operating in domains without ground truth. The key idea is to test whether the AI’s outputs violate consistency constraints of the problem domain. We show that this allows finding clear failures in state-of-the-art models, including GPT-4 on tasks where it is hard to evaluate, and superhuman chess engines.

We reflect on the discrepancies between the attack goals and techniques developed in the adversarial examples literature, and the current landscape of attacks on chatbot applications.