How to Validate llms.txt, ai.txt, robots.txt and security.txt

A text file at the root of your site fails quietly. A broken llms.txt does not throw an error anywhere; an agent just reads less of it. A security.txt that expired last month still serves a 200. A robots.txt rule with a typo is skipped without a word. Validation is how you find out before a crawler does.

This guide walks through validating all six formats our validator supports: llms.txt, llms-full.txt, both ai.txt standards, robots.txt and security.txt. It covers how to feed a file in, when to override what the tool detects, how to read the results, and which checks you still have to do by hand.

Step 1: validate from a URL, or paste the file

There are two ways in, and they check slightly different things.

• Validate from URL checks the file as it is actually served. That adds two things pasting cannot: a warning when the server sends it as something other than text/plain or text/markdown, and relative links resolved against the real address.
• Paste Content checks a draft before it is published, or a file you cannot fetch. Nothing is uploaded; validation runs in your browser.

One served-as warning deserves attention: text/html almost always means your server returned a “page not found” page with a 200 status instead of the file. The validator is then reading your 404 template, and so is every crawler.

Step 2: check the detected format

Every result shows which format was detected and why. From a URL, the filename decides: a path ending in security.txt, robots.txt, llms-full.txt, llms.txt or ai.txt is routed accordingly. Pasted content is identified by its shape: YAML frontmatter, the Contact and Expires fields, robots-only directives, or a Markdown H1 with a blockquote and link lists.

Two cases are worth knowing, because both are correct behaviour that can surprise you:

• A pasted robots.txt can be read as a Spawning ai.txt. The two formats share a grammar, and a file with only User-agent, Allow, Disallow and Sitemap lines is byte-for-byte valid as either. Unless it contains a robots-only line such as Content-Signal or Crawl-delay, pick robots.txt from the format dropdown, or validate by URL. Why two ai.txt formats exist explains the overlap.
• A large llms.txt is checked as llms-full.txt. Past 500 KB, a file named llms.txt is almost certainly the full-content variant, and it gets the lenient rules. If yours really is a link index that big, it is too big: the point of llms.txt is a short read.

The dropdown overrides detection for any format, so a wrong guess never locks you into the wrong rules.

Step 3: read errors, warnings and notes differently

Results come in three lists, and only one of them decides whether the file is valid.

The split matters because the specs are looser than most people assume. The llms.txt spec requires exactly one thing, the H1; a missing blockquote or missing H2 sections are warnings, not errors. A validator that fails you for those is enforcing its own rules. The v2 changes post covers what is and is not required.

What each format gets checked for

llms.txt and llms-full.txt

Errors are rare by design: a missing H1, or a link with a disallowed scheme such as javascript: or data:. Relative links are fine. Warnings cover the things that weaken the file: no blockquote summary, no H2 link sections, a bare URL where a Markdown link belongs, the same URL listed twice, and more than 100 links. Notes flag v2 details, such as a file published at a subpath and so covering only that subtree, or no links to Markdown versions of pages. llms-full.txt is checked leniently: a missing H1 is only a warning there. For the patterns behind these rules, see common llms.txt mistakes.

robots.txt

Syntax first: an Allow or Disallow with no User-agent above it is an error, and a path that does not start with / is a warning because crawlers ignore it. Then intent: blocking a vendor's search crawler along with its training crawler, a Content-Signal declared only under * where named groups never read it, and signals that contradict each other across groups. The result also lists every major AI crawler and whether your file allows it, blocks it or never mentions it. Controlling AI crawlers with robots.txt goes through each vendor's tokens.

security.txt

RFC 9116 is strict, so more findings here are errors: no Contact, no Expires, an Expires that is not an ISO 8601 date or is already in the past, and a single-use field repeated. An expiry more than a year away is a warning. Missing Canonical or Policy, and an unsigned file, are notes. Clearsigned files are handled: the PGP armor is not mistaken for fields.

ai.txt

The two ai.txt standards get separate rule sets. A Spawning file is checked like robots.txt, plus deprecated crawler tokens. An aitxt.ing file is checked for its frontmatter keys (updated, scope, parent) and its Markdown structure. The Your Next Tours case study shows what a clean one looks like in practice.

What no validator can check for you

A clean result means the file matches its specification. It does not mean the file is right. Four checks stay manual:

1. Links resolve. A validator checks that a URL is well-formed, not that it returns a page. Click through, or run a link checker against the file.
2. Your files agree with each other. A path listed in llms.txt but disallowed in robots.txt invites an agent to read a page it has been told not to fetch. Compare them by hand.
3. The v2 link relations are in place. rel="describedby" and the Markdown alternate live in your HTML and HTTP headers, not in llms.txt. Check with curl -I.
4. External identifiers still exist. A Wikidata or registry ID that no longer resolves undermines the file it was meant to anchor, as the First Point case study found.

A pre-publish checklist

1. Paste the draft and fix every error.
2. Read each warning and fix it, or write down why not.
3. Confirm the detected format is the one you meant.
4. Publish, then validate again from the live URL to catch Content-Type and 404 problems.
5. Do the four manual checks above.
6. Put a reminder in your calendar before security.txt's Expires date.

Starting from scratch? Every template on our examples page passes with no errors or warnings, so you begin from a clean result instead of working towards one.