ProductGPT Scraper Service

Containerized microservice for Amazon product scraping.

Architecture

• REST API for job control
• Shared MongoDB for data storage
• Docker containerization
• Puppeteer + Cheerio hybrid scraping

Quick Start

Development

# Install dependencies
bun install

# Set environment variables
cp .env.example .env
# Edit .env with your MongoDB URI

# Run development server
bun run dev

Docker

# Build and run with docker-compose
docker-compose up -d

# Or build manually
docker build -t productgpt-scraper .
docker run -p 3001:3001 --env-file .env productgpt-scraper

API Endpoints

Health Check

GET /health

Create Job

POST /api/jobs
Headers: X-API-Key: your-api-key
Body: {
  "type": "all" | "target-categories" | "one",
  "config": {
    "listType": "bestsellers",
    "categories": [["Beauty & Personal Care", "Skin Care", "Serums"]],
    "enableEnhancement": true
  }
}

Get Job Status

GET /api/jobs/:jobId
Headers: X-API-Key: your-api-key

Cancel Job

DELETE /api/jobs/:jobId
Headers: X-API-Key: your-api-key

Environment Variables

MONGO_URI=mongodb://...
PORT=3001
SCRAPER_API_KEY=your-secret-key
SCRAPER_MAX_CONCURRENT=15
SCRAPER_HEADLESS=true
SCRAPER_TIMEOUT=45000

Integration with Main API

Update your main API to call the scraper service:

// Instead of calling scraper directly:
const result = await hybridScraperService.scrapeAllCategories();

// Call the scraper service:
const response = await fetch('http://scraper-service:3001/api/jobs', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-API-Key': process.env.SCRAPER_API_KEY,
  },
  body: JSON.stringify({
    type: 'all',
  }),
});

const { data } = await response.json();
const jobId = data.jobId;

// Poll for status
const statusResponse = await fetch(`http://scraper-service:3001/api/jobs/${jobId}`, {
  headers: { 'X-API-Key': process.env.SCRAPER_API_KEY },
});

Resource Requirements

• Memory: 2-4GB recommended
• CPU: Multi-core for parallel scraping
• Network: Stable connection for Amazon scraping

Notes

• Scraper writes directly to shared MongoDB
• Jobs run asynchronously
• Supports job cancellation
• Health checks for container orchestration