Concurrent Web Crawler

A production minded web crawler written in Go. It explores HTTP and HTTPS pages within a single hostname using bounded concurrency, deterministic output, graceful cancellation, and explicit partial failure reporting.

The project is implemented as a reusable crawler package with a thin command line interface.

Highlights

• Bounded worker pool with configurable concurrency
• Single owner coordination for the queue, visited set, and active work count
• Same hostname scope enforcement, including redirect validation
• Context cancellation and HTTP request timeouts
• Deterministic output regardless of request completion order
• Partial results when individual pages fail
• Comprehensive unit and integration style tests
• Race detector coverage in continuous integration

Architecture

flowchart LR
    CLI[Command line interface] --> Coordinator[Coordinator]
    Coordinator --> Queue[Pending URL queue]
    Queue --> Workers[Bounded worker pool]
    Workers --> Fetcher[HTTP fetcher]
    Fetcher --> Parser[HTML link parser]
    Parser --> Workers
    Workers --> Results[Fetch results]
    Results --> Coordinator
    Coordinator --> Report[Crawl report]

Loading

The coordinator is the only component that mutates crawl state. Workers fetch and parse pages, then return immutable results. This ownership model avoids shared state locking around the queue and visited URL set.

Requirements

• Go 1.26 or newer

Quick Start

git clone https://github.com/azamir911/concurrent-web-crawler.git
cd concurrent-web-crawler
make check
make build
./bin/crawler https://example.com/

The crawler can also be run without building a binary:

go run ./cmd/crawler https://example.com/

Configure worker count and request timeout:

go run ./cmd/crawler -workers=8 -timeout=30s https://example.com/

Option	Default	Description
-workers	4	Maximum concurrent fetch operations
-timeout	10s	Timeout for each complete HTTP request

The process listens for SIGINT and SIGTERM. Cancellation stops new scheduling, waits for in flight workers to finish, and exits cleanly.

Output

Each successfully visited page is written to standard output:

URL: https://example.com/
Links:
  /about
  https://external.example.net/

External links are preserved in the output but are not crawled.

Errors and partial crawl information are written to standard error.

Exit code	Meaning
0	Crawl completed successfully
1	Crawl failed or completed with partial page failures
2	Invalid command line usage or configuration

Behavioral Guarantees

Scope

• Only HTTP and HTTPS URLs are eligible for crawling
• Only the seed hostname is in scope
• Different subdomains are excluded
• Redirect targets are validated before they are followed
• Relative links are resolved against the final URL after redirects

Deduplication

• URLs are marked as seen before scheduling
• Scheme and hostname casing are normalized
• Fragments are removed
• Path casing, query values, ports, and trailing slashes remain significant

Concurrency

• Active fetches never exceed the configured worker count
• The coordinator owns scheduling state
• Workers do not mutate the queue or visited set
• A crawl finishes only when the pending queue is empty and no jobs are active

Failure Handling

• One page failure does not stop unrelated pages
• Successful pages and failures are returned together
• Cancellation and timeout errors remain detectable with errors.Is
• No automatic retries are performed

Retries are deliberately excluded because a correct retry policy requires decisions about backoff, retryable status codes, server load, and request budgets.

Traversal Model

The crawler uses an iterative FIFO traversal that is conceptually breadth first. Network requests complete concurrently, so completion order is not breadth first. Pages and failures are sorted before output to keep results deterministic.

For a crawl graph with V scheduled pages and E discovered links, coordination and deduplication are approximately O(V + E). Memory usage grows with the visited set, pending queue, and accumulated report.

Validation

make check

The validation target runs:

go vet ./...
go test ./...
go test -race ./...

Coverage includes:

• URL validation, resolution, and scope rules
• Redirect handling
• HTML parsing
• HTTP statuses and content types
• Cycles and duplicate discovery
• Bounded concurrency
• Context cancellation and timeouts
• Partial failures
• Deterministic ordering
• Command line exit codes and output failures

Tradeoffs

In memory crawl state

This keeps coordination simple and fast for a single process. It is not suitable for crawls that exceed available memory.

Accumulated output

Sorting the final report provides deterministic output, but requires storing results until the crawl finishes.

Hostname scope

Restricting the crawl to one hostname is predictable and safe. Registrable domain crawling would require public suffix handling and a more explicit policy.

No distributed coordination

A multi process implementation would need a durable frontier, atomic URL claiming, retry and dead letter policies, persistent results, rate limiting, metrics, and tracing.

Responsible Use

This project does not implement robots.txt, crawl delay, authentication, or per host rate limiting. Those controls should be added before crawling third party websites at meaningful scale.

Project Layout

cmd/crawler/       CLI entrypoint, flags, signals, exit codes, and output
internal/crawler/  URL policy, coordination, HTTP fetching, and HTML parsing

License

MIT