I added a reranker to my note search. It made things worse. 37 queries over my… · @ReignSwords · TwKit

@ReignSwords

I added a reranker to my note search. It made things worse. 37 queries over my own notes (2,694 chunks, 40 files). Each query: 40 words lifted from one chunk. Task: find the REST of that note. The query chunk itself was removed from the corpus — no cheating. The scoreboard: 1. Plain keyword overlap → 48.6% recall@5, MRR 0.383 2. + MiniLM cosine rerank → 32.4%, MRR 0.238 3. + cross-encoder rerank → 45.9%, MRR 0.339 Read that again. The “smart” rerankers lost to counting words. And the cross-encoder took 4.7 seconds per query on my 4-core CPU. I paid 4.7s/query for the privilege of tying with grep-with-extra-steps. Why? My queries were lexical — words lifted straight from the note. Keyword overlap feasts on that. The rerankers took a decent shortlist and reshuffled it into a worse one. Semantic similarity is not the same as “from the same document.” Honest verdict: a reranker is not an upgrade, it's a bet. It wins when queries are vague and the corpus is paraphrased. It loses when the answer shares your exact words. Measure it on YOUR data before you ship it. The eval harness is 30 lines — steal it, point it at your own corpus. Details in the comments. I run local-model experiments like this and post the real numbers — worth a follow if that's your thing. So: team “just add a reranker” — still confident? 🔵 Fellow Premium members, let's connect! I post about AI & tech — follow for more 👥

曝光 6 · 评论 1 · 点赞 0 · 书签 0