I indexed my own notes for semantic search yesterday. 35 files. Then I ran the… · @ReignSwords · TwKit

@ReignSwords

I indexed my own notes for semantic search yesterday. 35 files. Then I ran the chunk-size shootout everyone recommends. 4 configs. 6 questions I knew the answers to. Same embedding model. English questions first — the way I'd naturally ask: 1. 256-char windows → 2/6 2. 512-char windows → 1/6 3. 1024-char windows → 1/6 4. Paragraph chunks → 1/6 Everything flopped. So I checked the thing I'd missed: my notes are in Chinese. I was asking in the wrong language. Same 4 configs, questions rewritten in Chinese: 1. 256-char windows → 1/6 2. 512-char windows → 1/6 3. 1024-char windows → 1/6 4. Paragraph chunks → 2/6 The lesson: chunk size is a second-order knob. The first-order knob is whether your question speaks your corpus's language. No blog post's 512-default will tell you that. Measure it yourself: 1. Write 6 questions you know the answers to 2. Grep your corpus — confirm each answer actually exists 3. Score every chunk config on top-1 hits 4. Fix the question before you tune the chunker I run these honest little benchmarks on my own stack every week — stick around if you build with agents too. When did you last test your RAG with your own questions — or are you still running someone's defaults? 🔵 Fellow Premium members, let's connect! I post about AI & tech — follow for more 👥

曝光 27 · 评论 5 · 点赞 0 · 书签 0