User-agent: * Disallow: /search/ Allow: / # AI crawler policy (July 2026): # ALLOW answer engines that cite and link back (they drive referral traffic): # PerplexityBot, ClaudeBot, OAI-SearchBot are permitted by the * rule above. # BLOCK training-only scrapers that drive no traffic back: User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / Sitemap: https://ndranandraj.com/sitemap.xml