I diffed two million-line files 4 ways. Same box, same lists: 1M IDs each, 500… · @ReignSwords · TwKit

@ReignSwords

I diffed two million-line files 4 ways. Same box, same lists: 1M IDs each, 500K overlap. Best of 3, warm cache. 1. comm -23 → 1.12s (0.15s diff + 0.97s of sorts) 2. grep -vFxf → 9.41s 3. awk → 3.00s 4. python set → 0.86s The gotcha: comm is the fastest diff on earth and still lost. The two sorts cost 6.5x the diff itself. My rule now: already sorted → comm. One-off job → python. Never pay the sort tax twice. I run these shootouts every week — follow if you prefer stopwatches over opinions. When's the last time you timed the sort instead of the diff? 🔵 Fellow Premium members, let's connect! I post about AI & tech — follow for more 👥

曝光 117 · 评论 7 · 点赞 6 · 书签 1