A from-scratch AI inference service built to answer one question: how cheap can you make text embeddings before quality degrades?
The answer turned out to be pretty cheap. By stacking three optimisation layers — INT8 quantization, Redis caching, and dynamic request batching — the final system runs on free-tier cloud hardware and handles concurrent load at 2.17x the throughput of a naive single-inference setup.
Built and documented as a five-phase series, each phase adding one layer of the stack.
Accepts text via a POST request, returns a 768-dimensional embedding from a quantized DistilBERT model. Before touching the model, it checks Redis for a cached result. If the cache is cold and multiple requests arrive simultaneously, they get batched into a single ONNX forward pass rather than processed one by one.
A React dashboard polls the server every 3 seconds and shows latency, cache hit rate, throughput, batch size distribution, and estimated cost savings in real time.
Client
|
POST /v1/embed
|
+--> Redis lookup (Upstash, Mumbai)
| |
| HIT --> return cached embedding (~42ms)
| |
| MISS --> BatchWorker
| |
| classify queue depth
| |
| LOW (< 3) --> single ONNX inference
| MED/HIGH --> collect 50ms --> batched ONNX call
| |
| cache result with TTL --> return
|
GET /v1/dashboard --> Vite analytics dashboard
| Phase | What changed | Key number |
|---|---|---|
| Phase 1 | INT8 quantization | 254MB → 63MB, 45ms → 16ms inference |
| Phase 2 | Redis caching | 42ms cache hits, 2.6x speedup on repeated queries |
| Phase 3 | Dynamic batching | 2.17x throughput, ~6ms per item in a batch of 8 |
| Phase 4 | Analytics dashboard | Live metrics, cost savings ticker |
| Phase 5 | Cloud deployment | Running free on HF Spaces + Vercel |
Backend
- FastAPI - request handling, routing, middleware
- ONNX Runtime - INT8 quantized inference
- HuggingFace Transformers + Optimum - model download and ONNX export
- Upstash Redis - HTTP-based caching, no persistent TCP connection
- asyncio - event loop, queue-depth load classification, Future-based batching
Frontend
- Vite + React
- Recharts - latency, throughput, batch size, tier distribution charts
Infrastructure
- Docker - containerised backend
- Hugging Face Spaces - free CPU hosting, persistent /data volume for model artifacts
- Vercel - frontend hosting, proxies /v1/* to HF Space
# Backend
cd server
python -m venv venv && venv\Scripts\activate
pip install -r requirements.txt
python local/quantize.py
python local/benchmark.py
uvicorn api.main:app --reload --port 8000
cd cloudinfer-ui
npm install
npm run dev python local/benchmark_cache.py --requests 100 --unique-ratio 0.3
python local/benchmark_batch.py --waves 10 --wave-size 12| Variable | Description |
|---|---|
UPSTASH_REDIS_REST_URL |
Upstash database REST URL |
UPSTASH_REDIS_REST_TOKEN |
Upstash REST token |
ONNX_MODEL_PATH |
Path to INT8 model (default: models/distilbert_int8/model_int8.onnx) |
BATCH_LOW_THRESHOLD |
Queue depth below which requests are processed immediately (default: 3) |
BATCH_WINDOW_MS |
Collection window for batching (default: 50) |
CACHE_TTL_DEFAULT |
Default Redis TTL in seconds (default: 3600) |
GET /health liveness + readiness check
POST /v1/embed text embedding inference
GET /v1/metrics cache hit rate, latency averages
GET /v1/batcher/metrics batch size, load tier distribution
GET /v1/batcher/config current threshold settings
GET /v1/dashboard aggregated payload for the frontend
Backend is on Hugging Face Spaces (free Docker tier). On first boot the container downloads DistilBERT, exports to ONNX, and applies INT8 quantization — this takes about 5 minutes and is cached to the /data volume for subsequent restarts.
Frontend is on Vercel. The vercel.json rewrite rule proxies all /v1/* calls to the HF Space so the dashboard works without CORS configuration.
Live: https://shridhar23-cloudinfer.hf.space
Quantization makes inference viable on free CPU hardware. Without it the model is 254MB and takes ~280ms per call — too slow to build anything useful around.
Caching eliminates compute for repeated queries entirely. At a 70% hit rate, 70% of requests cost nothing. This is where the actual cost savings come from.
Batching handles the cache miss path efficiently. When 8 unique queries arrive simultaneously, one batched ONNX call takes ~53ms total instead of 8 sequential calls taking ~352ms. It doesn't reduce cost — it multiplies throughput without adding hardware.
MIT