We have reverse-engineered the (secret) Claude 3 tokenizer by inspecting the generation stream. Check our blog post, code and Twitter thread.