Indexed Claude Chats Reveal Robots.txt Limits

Researchers discovered publicly indexed Claude AI conversation pages, highlighting that using Disallow in robots.txt does not prevent already accessible pages from appearing in search results. Robots.txt only restricts crawling and should not be considered a replacement for proper indexing controls. Publishers wanting confidential or temporary pages excluded from search should use noindex, authentication, or other access restrictions. The incident serves as another reminder that crawl directives alone cannot guarantee content remains hidden from search engines.