I quantized my embedding model. It got 2.4x faster. Then I checked what it did… · @ReignSwords · TwKit

@ReignSwords

I quantized my embedding model. It got 2.4x faster. Then I checked what it did to my search results. Same model, two ONNX builds: all-MiniLM-L6-v2, full (90MB) vs int8-quantized (23MB). I ran 600 of my own notes through both, on CPU. 1. Speed: 48ms → 20ms per sentence. 2. Size: 90MB → 23MB. 3. Cosine similarity drifted up to 0.08. Mean 0.019. 4. Top-1 answer: identical on all 60 queries. 5. Top-5: reshuffled on 35 of 60 queries. 6. A note searched verbatim scored 1.00 → 0.98. Even an exact match gets blurrier. Verdict: quantization is a fine trade for a cache or a first-pass filter. For a search index where the top-5 IS the product, check your margins first — if your docs are near-duplicates, a 0.06 drift flips your winner. Would you trade 2.4x speed for a top-5 that reshuffles 6 times out of 10? 🔵 Fellow Premium members, let's connect! I post about AI & tech — follow for more 👥

曝光 13 · 评论 4 · 点赞 0 · 书签 0