# robots.txt for https://www.mattressguides.com/ User-agent: Googlebot User-agent: Googlebot-Image Disallow: User-agent: Bingbot Disallow: User-agent: Applebot Disallow: # ----- LLM PROVIDERS (full access) ---------------------------- # Explicitly allow major LLM search and retrieval bots. # Full access (no path restrictions) so AI-powered search # tools can surface product pages in recommendations. # OpenAI (https://developers.openai.com/api/docs/bots) User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: OAI-AdsBot User-agent: GPTBot Disallow: # Anthropic (https://support.claude.com/en/articles/8896518) User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot Disallow: # Google (https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) User-agent: Google-Extended User-agent: GoogleOther Disallow: # Perplexity (https://docs.perplexity.ai/docs/resources/perplexity-crawlers) User-agent: PerplexityBot User-agent: Perplexity-User Disallow: # Common Crawl (https://commoncrawl.org/ccbot) User-agent: CCBot Disallow: # Apple (https://support.apple.com/en-us/119829) User-agent: Applebot-Extended Disallow: # ----- AMAZON BOTS (selective) -------------------------------- # Amazonbot: general crawler, used to train Amazon AI models. # AmazonAdBot: intelligence gathering, no benefit without # Amazon ad network participation. # Amzn-SearchBot and Amzn-User are NOT blocked -- they power # Rufus (250M+ users) and Alexa, and do not train models. User-agent: Amazonbot Disallow: / User-agent: AmazonAdBot Disallow: / Sitemap: https://www.mattressguides.com/sitemap.xml