RankLabs Observe

RankLabsBot

Deterministic public-web evidence acquisition for RankLabs.

This is the authoritative page for site operators and machine consumers. It explains how RankLabsBot discovers pages, what it records, how it differs from provider-operated AI crawlers, and how to verify or restrict it.

What RankLabsBot is

RankLabsBot is RankLabs' deterministic first-party crawler agent for public-web evidence acquisition. It supports RankLabs' Observe layer by retrieving publicly available site representations and preserving evidence for inspection and deterministic analysis. RankLabsBot is not an AI model, an answer engine, or a crawler operated by an AI provider.

User agent

RankLabsBot identifies its primary HTTP acquisition requests with this User-Agent header:

RankLabsBot/1.0 (+https://ranklabs.ai/bot)

A User-Agent string can be copied by another actor and is not proof of origin by itself. Use the verification process below for suspicious traffic.

Evidence-based page discovery

URL classification helps RankLabs describe observed pages. It never authorizes a guessed crawl target. RankLabsBot uses this discovery contract:

Starts with the site's homepage and URLs declared through discovered sitemaps.

Adds same-site links extracted from fetched HTML when robots and locale policy allow them.

Does not guess platform routes such as collection, product, account, or about paths.

Keeps unsupported representations, blocked URLs, and failed fetches outside successful page evidence.

Evidence recorded

A successful crawl produces stored observations, not a claim about how another machine interpreted the site. Depending on availability and configured evidence limits, RankLabs can preserve:

Requested and final URLs, redirect history, HTTP status, response context, fetch mode, and observation time.

Bounded raw HTML and extracted page signals such as metadata, canonical references, internal links, and JSON-LD when available.

Validation issues and diagnostics linked to the exact stored page observation.

Explicit unavailable or failed states when the crawler cannot establish page evidence.

Crawl behavior

Robots policy before acquisition

RankLabsBot fetches and evaluates robots.txt for the host. It applies the most specific matching rules for RankLabsBot, falls back to User-agent: *, and resolves Allow and Disallow rules by specificity. Supported Crawl-delay values are applied with a documented 30-second safety maximum.

HTTP first

Primary acquisition uses bounded HTTP GET requests, follows only HTTP or HTTPS redirects within configured limits, records response status, and rejects oversized evidence objects. Later crawls can use conditional request headers when prior ETag or Last-Modified evidence exists.

Polite rate and backoff

Requests are limited per host. Retry policy distinguishes rate limits, transient server errors, and browser challenges. A 403, 429, or 503 response alone does not authorize browser rendering.

Evidence-gated rendering

RankLabs may use bounded Chromium rendering when stored delivery evidence supports a client-rendering or verified challenge condition. Rendering is an acquisition mode, not a simulation of an AI provider crawler.

Public-session continuity

RankLabsBot may retain cookies set by the host during one crawl run so subsequent requests remain in the same public session. It does not sign in, submit credentials, or use that session to perform customer-facing actions.

RankLabsBot compared with provider-operated AI crawlers

Provider-operated AI crawler

A third-party agent controlled by an AI, answer, or search provider. Its fetching, indexing, training, retrieval, and citation policies are established by that provider, not by RankLabs.

RankLabsBot

A first-party acquisition agent controlled by RankLabs. It records what RankLabs fetched from the site under the crawl policy described here. It does not prove what a provider fetched or how a model used a page.

Provider behavior is measured separately through enabled, stored AI observations.

Public-web crawl evidence is separate from commercial or analytics data obtained through explicitly connected integrations.

No external actions

RankLabsBot does not intentionally submit forms, add items to a cart, complete checkout, change account state, or publish content. Rendered pages may load the normal subresources required to present the page. Crawling is evidence acquisition and does not authorize RankLabs to change the site.

Verifying RankLabsBot

To verify traffic claiming to be RankLabsBot, contact RankLabs with the source IP addresses, timestamps, requested URLs, and User-Agent values. RankLabs can check whether the requests match its infrastructure records.

RankLabs does not currently publish a static IP allowlist. If a verified allowlist becomes available, this page will be its canonical location.

Restricting or blocking RankLabsBot

Site operators can restrict or block RankLabsBot with standard robots.txt directives. RankLabsBot fetches robots policy before each crawl session.

Block entirely

User-agent: RankLabsBot
Disallow: /

Disallow specific paths

User-agent: RankLabsBot
Disallow: /cart
Disallow: /checkout
Disallow: /account

Set a supported crawl delay

User-agent: RankLabsBot
Crawl-delay: 5

Allowlisting RankLabsBot

A site operator who has authorized a RankLabs crawl can use the exact User-Agent as one criterion in a WAF, CDN, or bot-management rule:

RankLabsBot/1.0 (+https://ranklabs.ai/bot)

User-Agent allowlisting is not cryptographic verification. Use the contact process below when origin verification is required.

Contact

Questions, abuse reports, verification requests, allowlisting requests, or rate-limit complaints can be sent to hello@ranklabs.ai. Include timestamps, requested URLs, source IP addresses, and relevant request headers when available.