manifesto.md

ConventionRepo Conventions

Robots Exclusion Protocol (robots.txt)

The rules a service publishes at /robots.txt to tell crawlers which URIs they may access, as standardised by the IETF in RFC 9309 from Martijn Koster's 1994 method.

Canonical text: https://www.rfc-editor.org/rfc/rfc9309.html. In use on this site: manifesto.md/robots.txt.

The rules

  1. This document specifies the rules originally defined by the "Robots Exclusion Protocol" that crawlers are requested to honor when accessing URIs.
  2. The rules MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service, UTF-8 encoded and served as text/plain.
  3. A group is one or more user-agent lines that are followed by one or more rules; a rule is a line with a key-value pair that defines how a crawler may access URIs.
  4. Crawlers MUST use case-insensitive matching to find the group that matches the product token; if no group matches, they follow the "*" group if one exists.
  5. Allow and disallow lines indicate whether accessing a URI that matches the corresponding path is allowed or disallowed; the matching MUST start with the first octet of the path.
  6. The most specific match found MUST be used; the most specific match is the match that has the most octets.
  7. If no matching rule exists, the URI is allowed.
  8. Crawlers MUST support the special characters "#" (line comment), "$" (end of the match pattern) and "*" (0 or more instances of any character).
  9. The crawlers SHOULD follow at least five consecutive redirects, even across authorities.
  10. A 4xx status code means robots.txt is unavailable and the crawler MAY access any resources on the server.
  11. A 5xx status code means robots.txt is unreachable and the crawler MUST assume complete disallow.
  12. Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable.
  13. The parsing limit MUST be at least 500 kibibytes.

Facts

Maintainer
IETF; authors Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman
Licence of the text
IETF Trust Legal Provisions (BCP 78)
First published
2022-09
Status of the text
stable
Listing
listed, not yet claimed

Read with

Badge

manifesto.md: listed

[![manifesto.md](https://img.shields.io/endpoint?url=https://manifesto.md/badge/robots-txt.json)](https://manifesto.md/robots-txt/)

Maintainers claim a listing with a pull request setting listing: claimed, or a yes on the listing issue.