⚡ Bolt: Optimize asynchronous operations by wrapping synchronous database and network I/O#87
Conversation
Wrap synchronous SQLite database persistence queries in `asyncio.to_thread()` and transition the Groq LLM inference library from synchronous `Groq` to `AsyncGroq`. This optimization prevents synchronous I/O operations from blocking the FastAPI asynchronous event loop, vastly improving concurrent application capability. Co-authored-by: Deepaksingh7238 <110552872+Deepaksingh7238@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
There was a problem hiding this comment.
Pull request overview
This PR updates the FastAPI backend to avoid blocking the event loop by making Groq LLM inference fully async and offloading synchronous SQLite persistence calls to a thread, improving concurrent request handling under load.
Changes:
- Replaced the synchronous Groq client with
AsyncGroqand awaited LLM completion calls in async endpoints. - Wrapped synchronous SQLite persistence operations (
create_transfer_record,set_agent_b,list_transfers,get_transfer) withawait asyncio.to_thread(...). - Added a Jules “Bolt” learning note documenting the async/persistence guidance.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| backend/main.py | Switches Groq calls to AsyncGroq and offloads synchronous persistence operations via asyncio.to_thread to prevent event-loop blocking. |
| .jules/bolt.md | Documents the “Bolt” learning/action about avoiding synchronous I/O in async FastAPI handlers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| from groq import AsyncGroq | ||
| from livekit import api | ||
| from contextlib import asynccontextmanager |
| # ⚡ Bolt: Using AsyncGroq to prevent blocking the FastAPI event loop during LLM inference | ||
| global groq_client | ||
| if groq_client is None: | ||
| groq_client = Groq(api_key=GROQ_API_KEY) | ||
| chat_completion = groq_client.chat.completions.create( | ||
| groq_client = AsyncGroq(api_key=GROQ_API_KEY) | ||
| chat_completion = await groq_client.chat.completions.create( |
| # ⚡ Bolt: Initialized AsyncGroq client here to avoid thread blocking during inference | ||
| global groq_client | ||
| if groq_client is None: | ||
| groq_client = Groq(api_key=GROQ_API_KEY) | ||
| groq_client = AsyncGroq(api_key=GROQ_API_KEY) | ||
|
|
| ## 2025-05-05 - Avoid synchronous persistence blocking FastAPI event loop | ||
| **Learning:** Found multiple synchronous persistence methods (`persistence.create_transfer_record`, `persistence.set_agent_b`, `persistence.list_transfers`, `persistence.get_transfer`) in the FastAPI backend main module. Synchronous operations executed on the main event loop will block async routing for the entire server during those operations. | ||
| **Action:** Always wrap synchronous persistence calls inside the `asyncio.to_thread` coroutine to prevent blocking the FastAPI asynchronous event loop, improving overall concurrent handling performance for the API application. | ||
|
|
||
| ## 2025-05-05 - Use async groq client for performance improvement |
💡 What: Transitioned Groq LLM logic to use
AsyncGroqand wrapped all synchronouspersistencemodule calls (create_transfer_record,set_agent_b,list_transfers,get_transfer) inasyncio.to_thread().🎯 Why: Executing blocking database and network I/O synchronously on the main thread blocks the FastAPI event loop, bottlenecking parallel API requests and restricting concurrency.
📊 Impact: High impact. Eliminates main thread blockage, allowing the system to process incoming concurrent API routes unhindered while processing SQLite writes and Groq LLM network wait times.
🔬 Measurement: Can be verified using performance test load simulators (like Locust or
ab) hitting the/transfersand/initiate-transferendpoints concurrently. Before this change, long network/disk operations would pause responsiveness of parallel calls.PR created automatically by Jules for task 15408712951919183000 started by @Deepaksingh7238