In this position paper, we argue that membership inference attacks are fundamentally unsound for proving to a third party, like a judge, that a production model (e.g., GPT-4) was trained on your data.

We show that LLMs often reproduce short snippets of training data even for natural and benign (non-adversarial) tasks.

We demonstrate the feasibility of poisoning LLMs during pre-training with effects that measurably persist even after safety fine-tuning.

We introduce finetuning as an effective way to extract larger amounts of training data from production language models. This attack extracts 5x more training documents than our previous divergence attack and enables targeted reconstruction of specific documents.

We discuss the security of the Glaze tool, and how the authors’ actions may not be in the best interest of their users.

We organized the LLM CTF and a Trojan Detection Competition on Aligned LLMs at IEEE SaTML 2024. This post summarizes the main findings.

We find that empirical evaluations of heuristic privacy defenses can be highly misleading, and propose a new evaluation protocol that is reliable and efficient.

We reverse-engineer the Claude 3 tokenizer. Just ask Claude to repeat a string and inspect the network traffic.

We present a novel attack that poisons RLHF data to enable universal jailbreak backdoors. Unlike existing work on supervised fine-tuning, our backdoor generalizes to any prompt at inference time.

We explore the privacy of machine learning systems, and show how many standard components of the ML pipeline create side-channel vulnerabilities that leak private user data.

We propose an evaluation methodology for AI systems operating in domains without ground truth. The key idea is to test whether the AI’s outputs violate consistency constraints of the problem domain. We show that this allows finding clear failures in state-of-the-art models, including GPT-4 on tasks where it is hard to evaluate, and superhuman chess engines.

We reflect on the discrepancies between the attack goals and techniques developed in the adversarial examples literature, and the current landscape of attacks on chatbot applications.