robots.txt correct variant
A well-formed robots.txt: directives inside User-agent groups, known keys, proper key: value syntax.
Site-wide file — one per host.
This signal isn't a page; the container fetches it at a fixed path, and the
correct/broken/authbroken variant is chosen by the request host. On this host the canonical path(s)
below serve the correct variant. Add
?mode=… to override locally.
Canonical path(s)
Broken variant
A rule before any group, a bare User-agent:, a missing colon, and an unknown directive.
Catalogued flags
| Flag | Kind | Severity |
|---|---|---|
robotsTxtDirectiveBeforeGroup | defect | high |
robotsTxtMissingColon | defect | medium |
robotsTxtEmptyUserAgent | defect | medium |
robotsTxtOversized | defect | medium |
robotsTxtUnknownDirective | informational — fails nothing | — |
Fetch the live file: https://agentic-mcp-modern.crawler-test.com/robots.txt