# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content # robots.txt — Aditya Mallah # Last updated: 2026-07-10 # Policy: fully open. All crawlers — search engines, AI search agents, # and AI training bots — may scrape, index, and use site content # (including for model training and fine-tuning). # === Default: open to everyone === User-agent: * Allow: / # Machine-readable surfaces (also linked from the site) # Sitemap: https://adityamallah.com/sitemap-index.xml # LLM index: https://adityamallah.com/llms.txt # LLM full content: https://adityamallah.com/llms-full.txt # RSS: https://adityamallah.com/rss.xml Sitemap: https://adityamallah.com/sitemap-index.xml # === AI training & foundation-model crawlers (explicitly allowed) === User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: OAI-AdsBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Code Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-Web Allow: / User-agent: Google-Extended Allow: / User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-News Allow: / User-agent: Googlebot-Video Allow: / User-agent: Google-InspectionTool Allow: / User-agent: AdsBot-Google Allow: / User-agent: Mediapartners-Google Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / User-agent: Bytespider Allow: / User-agent: TikTokSpider Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: FacebookBot Allow: / User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / User-agent: Diffbot Allow: / User-agent: DeepSeekBot Allow: / User-agent: PanguBot Allow: / User-agent: Amazonbot Allow: / User-agent: Amzn-SearchBot Allow: / User-agent: Amzn-User Allow: / User-agent: Omgilibot Allow: / User-agent: omgili Allow: / User-agent: Panscient Allow: / User-agent: iAskBot Allow: / User-agent: iaskspider Allow: / User-agent: Yandex Allow: / User-agent: YandexAdditional Allow: / User-agent: YandexAdditionalBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: DuckAssistBot Allow: / User-agent: YouBot Allow: / User-agent: PhindBot Allow: / User-agent: Andibot Allow: / User-agent: ExaBot Allow: / User-agent: LinkupBot Allow: / User-agent: TavilyBot Allow: / User-agent: Kagi-Fetcher Allow: / User-agent: Bravebot Allow: / User-agent: KunatoCrawler Allow: / User-agent: Bingbot Allow: / User-agent: BingPreview Allow: / User-agent: MSNBot-Media Allow: / User-agent: ia_archiver Allow: / User-agent: archive.org_bot Allow: / User-agent: PetalBot Allow: / User-agent: SemrushBot Allow: / User-agent: AhrefsBot Allow: / User-agent: DotBot Allow: / User-agent: MJ12bot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: Sogou Allow: / User-agent: Slurp Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: Twitterbot Allow: / User-agent: xAI-Grok Allow: / User-agent: Grok Allow: /