OpenAI will honor a shared robots.txt signal for AI training preferences
OpenAI said it will honor, within the coming months, a common signal in robots.txt that lets site owners tell multiple participating AI providers at once whether their content may be used for generative AI training. The approach comes from the IETF's AI Preferences working group, and OpenAI's existing per-crawler controls stay in place. My take: blocking crawlers one company at a time was a chore, and now there's a shared rule. If you own a website or blog, spend five minutes deciding whether AI may learn from your content, instead of letting someone else decide for you.