US Government Contract Scraper & Manager

A full-stack application to scrape US government contracts from SAM.gov, store them in MongoDB, and provide a web interface for browsing and organizing contracts into lists.

Tech Stack

• Backend: Node.js + Express
• Frontend: React + Tailwind CSS
• Database: MongoDB
• Scraper: Node.js with SAM.gov API integration
• Deployment: Docker

Getting Started

Prerequisites

• Node.js 18+
• Docker and Docker Compose (for MongoDB)
• SAM.gov API key (free, register at https://sam.gov/content/entity-registration)

Development Setup

1. Start MongoDB using Docker:

   docker-compose -f docker-compose.dev.yml up -d

2. Configure environment:

   cd backend
   cp .env.example .env
   # Edit .env and add your SAM_API_KEY

3. Install and run backend:

   cd backend
   npm install
   npm run dev

4. Install and run frontend (in a new terminal):

   cd frontend
   npm install
   npm run dev

5. Open http://localhost:3000 in your browser

Running the Scraper

Run the scraper manually via CLI:

cd backend
npm run scrape

Or trigger it via API:

curl -X POST http://localhost:3001/api/scraper/run

API Endpoints

Contracts

• GET /api/contracts - List contracts (paginated, filterable)
• GET /api/contracts/:id - Get single contract
• GET /api/contracts/stats - Dashboard statistics

Lists

• GET /api/lists - Get all lists
• POST /api/lists - Create new list
• GET /api/lists/:id - Get list with contracts
• PUT /api/lists/:id - Update list
• DELETE /api/lists/:id - Delete list
• POST /api/lists/:id/contracts - Add contract to list
• DELETE /api/lists/:id/contracts/:contractId - Remove from list

Scraper

• POST /api/scraper/run - Trigger scrape
• GET /api/scraper/status - Check scraper status

Production Deployment

Build and run with Docker Compose:

# Set your SAM API key
export SAM_API_KEY=your_api_key_here

# Build and start all services
docker-compose up -d --build

The application will be available at:

• Frontend: http://localhost:3000
• Backend API: http://localhost:3001

Project Structure

contracting/
├── backend/
│   ├── src/
│   │   ├── config/      # DB connection, env config
│   │   ├── models/      # MongoDB schemas
│   │   ├── routes/      # API endpoints
│   │   └── scraper/     # SAM.gov scraper
│   ├── server.js
│   └── Dockerfile
├── frontend/
│   ├── src/
│   │   ├── components/  # Reusable UI components
│   │   ├── pages/       # Main views
│   │   ├── hooks/       # Custom React hooks
│   │   └── services/    # API client
│   └── Dockerfile
├── docker-compose.yml
└── README.md