Your pretty-printed JSON is costing you almost 2x the tokens. I ran the same 5… · @ReignSwords · TwKit

@ReignSwords

Your pretty-printed JSON is costing you almost 2x the tokens. I ran the same 5 rows of data through tiktoken tonight in 5 formats. GPT-4o tokenizer. Here's the damage: 1. Pretty JSON: 137 tokens 2. Minified JSON: 72 3. YAML: 95 4. Markdown table: 69 5. CSV: 46 Pretty-printing nearly doubles the bill. That's the whitespace tax, measured. Two more from the same session: - The same sentence in Chinese costs 1.5x the tokens of English on GPT-4's tokenizer. On GPT-4o's it's 1.15x. They quietly cut CJK token cost by 23.5%. - A 12-line Python retry function: 59 tokens. Code is cheap. Formatting is the thief. Honest caveats: tiktoken only covers OpenAI's tokenizers. Claude and Gemini don't publish theirs, so your mileage varies there. And I measured price, not quality — uglier input might cost you accuracy. That's a test for another night. My rule from now on: machine-to-machine payloads go out minified, tables go out as markdown, and nothing a model reads gets pretty-printed ever again. I run these tear-downs so you don't have to — follow along if that's your thing. Now be honest: how much of your context window is just indentation? 🔵 Fellow Premium members, let's connect! I post about AI & tech — follow for more 👥

曝光 28 · 评论 6 · 点赞 0 · 书签 0