Skip to content

example_aggregate_output.md

Latest commit

 

History

History
151 lines (118 loc) · 4.71 KB

example_aggregate_output.md

File metadata and controls

151 lines (118 loc) · 4.71 KB
Raw

Quicrawl LLM-Friendly Aggregate Format

This document demonstrates the new LLM-friendly aggregate format that has been added to Quicrawl.

What's New

The crawl methods now support returning HTML content in an aggregate format that is specifically designed to be LLM-friendly. This format includes:

  1. Clear Separations: Each page is clearly separated with distinctive headers and separators
  2. Page Details: Each page includes URL, status code, timestamp, and error information
  3. Structured Layout: The format follows a consistent structure that makes it easy for LLMs to parse and understand
  4. Aggregate Summary: A header section provides an overview of all crawled pages

Example Output Format

WEBSITE CRAWL AGGREGATE REPORT
================================================================================
ROOT URL: https://example.com
TOTAL PAGES: 3
CRAWL TIMESTAMP: 2024-01-15T10:30:00Z
================================================================================

ROOT PAGE
================================================================================
PAGE: https://example.com
STATUS: 200
TIMESTAMP: 2024-01-15T10:30:00Z
================================================================================

HTML CONTENT:
----------------------------------------
<!DOCTYPE html>
<html>
<head><title>Example</title></head>
<body><h1>Welcome</h1></body>
</html>
----------------------------------------


LINKED PAGES
================================================================================

LINKED PAGE 1/2
================================================================================
PAGE: https://example.com/about
STATUS: 200
TIMESTAMP: 2024-01-15T10:30:01Z
================================================================================

HTML CONTENT:
----------------------------------------
<!DOCTYPE html>
<html>
<head><title>About Us</title></head>
<body><h1>About</h1></body>
</html>
----------------------------------------

LINKED PAGE 2/2
================================================================================
PAGE: https://example.com/contact
STATUS: 404
TIMESTAMP: 2024-01-15T10:30:02Z
ERROR: Content is not HTML (Content-Type: text/plain)
================================================================================


END OF CRAWL REPORT
================================================================================

Usage

HTTP API

Single Page Crawl with Aggregate Format

curl -X POST http://localhost:8080/api/v1/crawl \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "aggregate": true}'

Parallel Crawl with Aggregate Format

curl -X POST http://localhost:8080/api/v1/crawl-parallel \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "aggregate": true, "max_concurrency": 5}'

Dedicated Aggregate Endpoints

# Single page aggregate
curl -X POST http://localhost:8080/api/v1/crawl-aggregate \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

# Parallel aggregate
curl -X POST http://localhost:8080/api/v1/crawl-parallel-aggregate \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "max_concurrency": 5}'

CLI Commands

Single Page Crawl with Aggregate Format

# Display aggregate format in terminal
./quicrawl crawl https://example.com --aggregate

# Save aggregate format to file
./quicrawl crawl https://example.com --aggregate --output aggregate.txt

Parallel Crawl with Aggregate Format

# Display aggregate format in terminal
./quicrawl crawl-parallel https://example.com --aggregate --concurrency 5

# Save aggregate format to file
./quicrawl crawl-parallel https://example.com --aggregate --concurrency 5 --output aggregate.txt

Benefits for LLMs

  1. Structured Data: Clear headers and separators make it easy to identify different pages
  2. Context Preservation: Each page retains its URL and metadata context
  3. Error Handling: Failed crawls are clearly marked with error information
  4. Unified Format: All pages are presented in a consistent, parseable format
  5. Comprehensive View: LLMs can analyze the entire website structure in one document

Response Structure

The aggregate format returns an AggregateHTMLResult structure:

{
  "url": "https://example.com",
  "aggregate_html": "WEBSITE CRAWL AGGREGATE REPORT\n...",
  "page_count": 3,
  "success_count": 2,
  "error_count": 1,
  "timestamp": "2024-01-15T10:30:00Z",
  "errors": [
    "https://example.com/contact: Content is not HTML (Content-Type: text/plain)"
  ]
}

This format makes it much easier for LLMs to understand and process website content while maintaining all the important contextual information.