# The crawler policy for https://laithalsaadoon.github.io, every project site under it included.
#
# RFC 9309 s2.3 puts robots.txt at the origin root, so this one file governs every GitHub Pages
# project site the account serves (https://laithalsaadoon.github.io/<repo>/). A project site cannot
# publish its own: a file at /<repo>/robots.txt has no protocol meaning. Every site under this origin
# is public documentation for an open-source repository, and being read is the reason each one exists,
# so every class below is allowed, and allowed explicitly. A missing robots.txt would produce the same
# traffic with no record of the decision (RFC 9309 s2.3.1.3: a crawler that gets a 4xx "MAY access any
# resources on the server").
#
# The classes mirror laithalsaadoon/microvms-agentd site/src/lib/robots.ts, where each vendor token was
# checked against that vendor's own crawler documentation on 2026-08-26. Change both together.
#
# HOW A CRAWLER READS THIS:
#
#   * A crawler obeys exactly ONE group: the first whose product token matches it. "User-agent: *" is a
#     fallback used only when no named group matches, so every line that applies to a named token,
#     Content-Signal included, is repeated inside that token's own group.
#   * Product-token matching is case-insensitive (s2.2.1); path values are case-sensitive.
#   * "Sitemap:" is not part of RFC 9309 (s2.2.4, "Other Records"). It must be an absolute URL and
#     belongs to no group.
#
# CONTENT SIGNALS. "Content-Signal:" states how content may be used after it is fetched, in the three
# categories of draft-romm-aipref-contentsignals-00 (https://contentsignals.org): search (a search index
# and its results), ai-input (retrieval, grounding and other real-time use in generated answers), and
# ai-train (training or fine-tuning). It is a stated preference, not access control; RFC 9309 parsers
# skip the line. The draft expired on 2026-04-04 without adoption, so the syntax may change.
# All three are "yes" here, matching the Allow lines: a model that has read these docs answers questions
# about the projects correctly instead of inventing an API. To decline training, change ai-train=yes to
# ai-train=no on every line and say why here.


# CLASS B: user-initiated fetchers. A person asked an assistant to read this page now. The raw Markdown
# twins and llms.txt bundles on the project sites exist for this use.
User-agent: Claude-User
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: DuckAssistBot
User-agent: Amzn-User
User-agent: meta-externalfetcher
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# CLASS A1: corpus collection that feeds model training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Applebot
User-agent: Amazonbot
User-agent: Bytespider
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# CLASS A2: AI search and answer indexes. Each vendor states these do not feed model training.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
User-agent: Amzn-SearchBot
User-agent: Google-CloudVertexBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# CLASS A3: control-only tokens. These never fetch; they govern whether content the operator's ordinary
# crawler already collected may be used for AI training and grounding. Google-Extended affects Gemini
# training and grounding, and not Search inclusion or ranking.
User-agent: Google-Extended
User-agent: Applebot-Extended
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# LEGACY TOKENS. Anthropic's crawler documentation lists three tokens (ClaudeBot, Claude-User,
# Claude-SearchBot) and neither of these. They are listed because older robots.txt files and
# agent-readiness scanners still check for them; a rule naming them matches no current Anthropic crawler.
User-agent: Claude-Web
User-agent: anthropic-ai
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# EVERYTHING ELSE. No Disallow lines: a crawler that renders a page to judge it needs the CSS and JS,
# and a blocked asset is scored as a broken page.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /


# A sitemap index covering the root page and every project site's own sitemap. It is rebuilt daily and
# on every push to this repository; each project republishes its own child sitemap when it deploys.
Sitemap: https://laithalsaadoon.github.io/sitemap.xml