One robots.txt line now tells four AI labs to keep your words out of training.… · @harden_369741 · TwKit

@harden_369741

One robots.txt line now tells four AI labs to keep your words out of training. OpenAI, Google, Microsoft and Anthropic agreed on October 30 to honor the same token, __usage_ai-training, so a publisher stops maintaining a separate blocklist for GPTBot, ClaudeBot, Google-Extended and the rest. I have done that blocklist dance before and it always rotted, one crawler rename away from silently failing. The catch is right there in the fine print. This is a voluntary preference, not a lock. It covers training only, not crawling or search, and the rollout is staggered: Anthropic says it has started, Microsoft starts February 1, 2027, Google aims for 2027, OpenAI says the coming months. A shared opt-out is real progress, but I will only trust it once I can see a crawler actually honor it on a live site. https://www.thebrief.news/en/standard/article/38414/four-ai-companies-agree-on-a-shared-opt-out-signal-for-web-scraping #AITraining #RobotsTxt

曝光 11 · 评论 0 · 点赞 0 · 书签 0