Polite Auditing

RFC 9309 Compliance Documentation

Technical implementation of the Robots Exclusion Protocol (REP) within the HumangoBot engine.

Crawl-Delay & Rate Limiting

HumangoBot strictly follows the Crawl-delay directive in robots.txt. If no delay is specified, we maintain a default minimum of 5 seconds between requests to a single domain.

Backoff Strategy

Upon receiving 429 or 503 status codes, we apply exponential backoff to reduce server load.

Disallow Protocol

Paths listed under Disallow for HumangoBot or * are strictly excluded from auditing.

User-Agent Identification

To avoid being flagged as generic traffic, our User-Agent is unique and informative. We do not use generic headers (like Go-http-client or node-fetch).

HumangoBot/1.0 (+https://humango.app)