crawlreplay

crawlreplay checks how a proposed robots.txt file or candidate deployment would affect URLs that crawlers have actually requested.

It supports two checks:

• Policy check: compare old and new robots.txt rules against access logs.
• Release check: fetch the public and candidate versions of sampled URLs, then compare status, redirects, canonical links, indexability, hreflang, schema types, and content fingerprints.

This is pre-release software. Review the privacy and network-safety sections before using production logs.

Install from source

You need Rust 1.86 or newer.

cargo install --locked --path .
crawlreplay --version

Nothing in this repository publishes a crate, image, release, or GitHub repository automatically.

Check a robots.txt change

crawlreplay policy \
  --old robots.production.txt \
  --new robots.proposed.txt \
  --format markdown \
  access.log

Use --verified-only when findings should include only Google common crawlers and Bingbot requests whose source addresses match the bundled vendor CIDR snapshots. Other recognized crawler user agents remain claimed, not verified.

Check a candidate deployment

crawlreplay release \
  --baseline-origin https://www.example.com \
  --origin https://preview.example.net \
  --max-urls 250 \
  --format json \
  access.log

--origin is the candidate deployment. --baseline-origin is the current public site. Both are required, so an untrusted host field in a log cannot choose a network destination. Run separate checks when one log set covers more than one public origin.

By default, replay requests retain query parameter names and duplicate order but send empty values. This prevents a production token or personal value from being forwarded to the candidate. --replay-query-values restores the original values for both origins; use it only with logs and a candidate environment you are authorized to share that data with.

Supported input

• Nginx combined access logs
• line-delimited Nginx JSON logs using the documented default field names
• line-delimited Cloudflare Logpush HTTP request logs
• plain text and gzip data detected by magic bytes
• multiple files in the order provided
• standard input via -

Only GET and HEAD records are checked. Each input record is limited to 1 MiB. Malformed records stop the run and report the source and line without printing the raw record.

Reports

Use --format with one of:

• terminal for a concise operator summary
• json for the versioned machine-readable report
• markdown for a CI job summary
• sarif for SARIF 2.1.0 consumers

Query values are redacted in every report. Terminal output uses color only when stdout is an interactive terminal and NO_COLOR is not set.

Client IP addresses are used in memory when a supported crawler can be checked against its published address ranges, then discarded by default. Reports never include them. Use --drop-client-ip=false only when the address needs to stay in memory until the command finishes.

Threshold configuration is documented in docs/README.md.

Exit codes

Code	Meaning
0	The check completed and no configured threshold failed.
1	One or more configured thresholds were reached.
2	CLI arguments or configuration are invalid.
3	A log or policy input could not be imported.
4	No baseline/candidate pair could be compared.
5	An internal error prevented a valid result.

HTTP failures for individual URLs appear as findings. A release check exits with 4 only when every sampled comparison failed.

Network safety

Release checks:

• send no captured cookies, authorization headers, bodies, or client headers;
• use an identifiable crawlreplay/ user agent;
• accept only HTTP and HTTPS URLs without embedded credentials;
• keep baseline and candidate redirects within their exact allowed origins;
• resolve hostnames through a resolver that rejects loopback, private, link-local, documentation, metadata, and other special-use addresses;
• disable ambient HTTP proxies so DNS validation remains in the same trust boundary as the connection;
• limit concurrency, request starts, redirects, time, and response bytes; and
• request an uncompressed body and enforce the size limit while streaming.

--allow-private relaxes the address restriction only for the exact candidate origin. It is intended for a named local or staging service. It does not allow a private baseline or cross-origin redirect.

See docs/threat-model.md for assumptions and residual risks.

Crawler identity data

The repository includes exact-union snapshots of:

• Google common crawler CIDRs, source timestamp 2026-07-31; and
• Bingbot CIDRs, source timestamp 2024-01-03.

Snapshot sources, checksums, and the update procedure are recorded in the crawler data notes. A matching address verifies only the supported crawler product associated with that snapshot. User-agent text by itself never produces a verified verdict.

Docker

docker build -t crawlreplay:local .
docker run --rm -v "$PWD:/work:ro" crawlreplay:local \
  policy \
  --old /work/robots.production.txt \
  --new /work/robots.proposed.txt \
  /work/access.log

The builder and runtime base images are pinned by digest. The runtime image uses a non-root user and contains only the binary, CA certificates, and this project's license.

GitHub Action

The composite action lives in action/. It builds this locked source tree before running a check. GitHub-hosted runners already provide Rust; a self-hosted runner needs Rust 1.86 or newer.

After the v0.1.0 tag is published, use:

- uses: lame13/crawlreplay/action@v0.1.0
  with:
    command: policy
    old-robots: config/robots.production.txt
    new-robots: public/robots.txt
    files: logs/crawlers.log
    format: json
    thresholds: config/crawlreplay-thresholds.json

For more than one log, provide one path per line in files. Omit thresholds when the workflow should report findings without acting as a release gate.

Development

cargo fmt --all -- --check
cargo check --workspace --all-targets --locked
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo test --workspace --locked

The CI workflow runs compilation and tests with Rust 1.86, checks dependencies against the RustSec advisory database, and uses Cargo 1.90 to assemble the interdependent workspace crates. It also smoke-tests the local action and builds the Docker image. See CONTRIBUTING.md before adding fixtures or changing report contracts.

Security and license

Report vulnerabilities using SECURITY.md. The source code is licensed under Apache-2.0; see LICENSE.