Note

This project is archived. OpenLib has shifted its focus toward showcasing high-quality, verified applications — curated by the community and admins, not automated bots. Going forward, only community-submitted and admin-approved apps will be listed on OpenLib. As a result, this crawler is no longer in active use or development.

The repository is preserved here for reference and for anyone who may find the architecture or approach useful for their own projects.

OpenLib Crawler

Production-ready GitHub crawler and AI-powered enrichment system built for discovering, analyzing, moderating, and syncing open-source applications into OpenLib.

Designed specifically for low-resource systems like MX Linux laptops, the crawler uses Node.js, SQLite, and local Ollama AI models to process repositories efficiently with minimal RAM and CPU usage.

---

Features

• GitHub repository discovery and crawling
• AI-powered metadata enrichment using local LLMs
• README parsing and screenshot extraction
• Quality scoring and duplicate detection
• Moderation workflow before publishing
• Sync engine for pushing approved apps to OpenLib
• SQLite-based lightweight architecture
• Restartable worker-based processing
• Optimized for low-end hardware
• PM2 and cron support for automation
• Browser GUI for live operations, data files, logs, queues, and moderation

---

Architecture

GitHub API
   ↓
Crawler Workers
   ↓
Repository Filters
   ↓
README + Metadata Parser
   ↓
Screenshot Extractor
   ↓
Local AI Enrichment
   ↓
Duplicate Detection
   ↓
Quality Scoring
   ↓
Moderation Queue
   ↓
Sync Queue
   ↓
OpenLib API

The crawler database acts as the source of truth. OpenLib data is updated only through authenticated sync workers.

---

Tech Stack

• Node.js
• SQLite
• Ollama
• PM2
• GitHub REST API
• Browser GUI served by the Node.js control server

---

Project Structure

openlib-crawler/
├── docs/
├── bin/
├── logs/
├── scripts/
├── src/
│   ├── gui/
│   ├── services/
│   ├── workers/
│   ├── config.js
│   ├── db.js
│   └── server.js
├── data/
├── crawler.js
├── updater.js
├── sync.js
├── ai.js
└── ecosystem.config.js

Important Components

File / Folder	Purpose
src/config.js	Loads environment configuration
src/db.js	Initializes SQLite database and WAL mode
src/gui/public/	Browser GUI assets
src/services/	Core services for crawling, AI, parsing, scoring, syncing
src/workers/	One-shot worker jobs
src/server.js	Browser GUI, moderation API, job controls, data API
docs/schema.sql	Reference database schema
ecosystem.config.js	PM2 process configuration
bin/openlib-control-panel	Launches the local GUI server

---

OpenLib Control GUI

The operator UX is now a browser GUI served by src/server.js. It provides dashboard metrics, worker controls, queue views, SQLite statistics, Ollama monitoring, GitHub API usage, system telemetry, log streaming, scheduler controls, repository moderation, data file downloads, app export, and .env editing.

npm run control

Then open:

http://127.0.0.1:3020

The GUI reads the existing openlib.db, .env, data/, and logs/ assets, so crawler data does not need migration. If MODERATION_API_KEY is set, enter it in the GUI API key field.

---

Installation

Requirements

• Node.js 18+
• npm
• Git
• Ollama (optional but recommended)

---

Setup

git clone https://github.com/AHS-Mobile-Labs/openlib-crawler.git && \
cd openlib-crawler && \
cp .env.example .env && \
npm install && \
npm run setup

---

Smoke Test

npm run smoke

---

Ollama Setup

Install Ollama and pull a lightweight model:

ollama pull tinyllama

Alternative lightweight models:

ollama pull qwen2.5:0.5b
ollama pull phi

Recommended settings for low-end systems:

CRAWLER_CONCURRENCY=2
AI_CONCURRENCY=1
TARGET_APPS_PER_RUN=50

---

Running Jobs

Crawl Repositories

npm run crawl -- --limit 50

Update Existing Apps

npm run update

Run AI Enrichment

npm run ai

Refresh Screenshots

npm run screenshots

Sync Approved Apps

npm run sync

Start Control GUI And Moderation API

npm run control

---

Scheduler

Run all jobs automatically:

npm run scheduler

---

PM2 Setup

Install PM2:

npm install -g pm2

Start workers:

pm2 start ecosystem.config.js

Save configuration:

pm2 save
pm2 startup

---

Cron Jobs

Example cron configuration:

0 */6 * * * cd /home/Linox/openlib-crawler && /usr/bin/npm run crawl -- --limit 50
20 2 * * * cd /home/Linox/openlib-crawler && /usr/bin/npm run update
40 1 * * * cd /home/Linox/openlib-crawler && /usr/bin/npm run ai
10 3 * * 0 cd /home/Linox/openlib-crawler && /usr/bin/npm run screenshots
*/30 * * * * cd /home/Linox/openlib-crawler && /usr/bin/npm run sync

---

Moderation API

Get Pending Apps

curl -H "X-API-Key: $MODERATION_API_KEY" \
"http://127.0.0.1:3020/api/apps?status=pending&limit=20"

Update App Metadata

curl -X PUT \
-H "X-API-Key: $MODERATION_API_KEY" \
-H "Content-Type: application/json" \
-d '{"category":"Developer Tools","tags":["git","developer-tools"]}' \
"http://127.0.0.1:3020/api/apps/1"

Approve App

curl -X POST \
-H "X-API-Key: $MODERATION_API_KEY" \
"http://127.0.0.1:3020/api/apps/1/approve"

Reject App

curl -X POST \
-H "X-API-Key: $MODERATION_API_KEY" \
-H "Content-Type: application/json" \
-d '{"reason":"not a user-facing app"}' \
"http://127.0.0.1:3020/api/apps/1/reject"

---

Quality Rules

Repositories are rejected before expensive processing if they are:

• Forks
• Archived
• Disabled
• Unlicensed
• Inactive
• Below minimum star threshold
• Missing useful descriptions

Quality scoring considers:

• Stars
• Activity
• Releases
• Screenshots
• Website availability
• Organization verification
• License quality
• README quality
• Documentation
• Topic richness

Only repositories above the configured quality threshold enter moderation.

---

Local AI Enrichment

OpenLib Crawler uses local Ollama models only.

If Ollama is unavailable, the system falls back to deterministic enrichment logic.

Supported lightweight models:

• tinyllama
• qwen2.5:0.5b
• phi
• lightweight Mistral variants

---

Project-Local Ollama

Start local Ollama:

npm run ollama:start

Pull models:

npm run ollama:pull

Check status:

curl http://127.0.0.1:11434/api/tags

Run AI test:

npm run ai -- --limit 1

Stop Ollama:

npm run ollama:stop

---

Logs & Storage

Resource	Location
SQLite Database	openlib.db
Main Logs	logs/openlib-crawler.log
PM2 Logs	logs/pm2-*.log
Screenshot Cache	data/cache/screenshots
Ollama Models	data/ollama-models

SQLite runs in WAL mode for better reliability during concurrent workers.

---

Goals

OpenLib Crawler aims to:

• Build a scalable open-source app discovery engine
• Reduce manual app submission work
• Improve metadata quality using local AI
• Support privacy-friendly self-hosted infrastructure
• Run efficiently on low-end hardware

---

License

MIT License

---

OpenLib

Discover open-source alternatives and apps on:

OpenLib Website