Everyone says "just use Jina Reader" when you need a webpage as markdown. I ra… · @ReignSwords · TwKit

@ReignSwords

Everyone says "just use Jina Reader" when you need a webpage as markdown. I ran it against 3 free alternatives on 8 real pages tonight — 32 fetches. The famous one won, and still managed to embarrass itself. The lineup: 1. http://r.jina.ai — the famous one. Free, no key. 2. trafilatura — local Python extractor. 3. readability + html2text — the old classic. 4. BeautifulSoup get_text — the dumb baseline. Scoreboard (pages returning real content): - http://r.jina.ai: 7/8. Most content per page — but GitHub 403s it outright. - trafilatura: 7/8. Fastest clean output, keeps tables. - readability: 5/8. Cleanest article prose — and exactly 2 characters on Hacker News. - bs4: 7/8. Dumb, noisy, never dies. Two gotchas nobody mentions: - Jina 403s YOU if your request carries a browser User-Agent. Drop the UA and it works. Cost me 8 fetches to learn that. - Bloomberg killed every local fetch with broken connections. Only Jina's rendering got through: 64K chars. That's the one job where the famous one is irreplaceable. Verdict, copy-paste ready: 1. One-off grab, or a site that blocks your scraper → http://r.jina.ai. No key, 20 req/min free. No browser UA. 2. Your own pipeline → trafilatura. Fetch with requests, extract with include_tables=True. 3. Clean article text → readability + html2text. Never on link-list pages. 4. Quick dirty dump → bs4 get_text. Fine for grep, poison for LLM context. "Just use Jina" was never the full answer. It's the right answer for exactly one of the four jobs. Full scoreboard in the replies. Which one's running in your RAG pipeline right now — and when did it last silently return garbage? 🤝 If you're into hands-on AI tool tests like this, follow along — I run one every day. 🔵 Fellow Premium members, let's connect! I post about AI & tech — follow for more 👥

曝光 55 · 评论 6 · 点赞 0 · 书签 0