⚡ Bolt: Prevent FastAPI event loop blocking#78
Conversation
…yncio.to_thread This commit prevents the FastAPI event loop from being completely blocked during network calls and database queries. Synchronous operations, which historically halted the event loop preventing concurrent handling, have been refactored. The `Groq` client was switched to `AsyncGroq`, and four blocking SQLite persistence calls (`create_transfer_record`, `set_agent_b`, `list_transfers`, and `get_transfer`) were offloaded to separate threads via `asyncio.to_thread`. Co-authored-by: Deepaksingh7238 <110552872+Deepaksingh7238@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
There was a problem hiding this comment.
Pull request overview
Refactors the FastAPI backend to improve event loop responsiveness by making Groq calls asynchronous and offloading blocking SQLite persistence work to threads.
Changes:
- Switch Groq usage in
/ai-voiceand/initiate-transferfrom syncGroqto asyncAsyncGroqwith awaited completions. - Offload synchronous SQLite persistence calls (
create_transfer_record,set_agent_b,list_transfers,get_transfer) usingasyncio.to_thread. - Add a Jules “bolt” note documenting the async-client anti-pattern.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| frontend/package-lock.json | Updates lockfile metadata for fsevents. |
| backend/main.py | Uses AsyncGroq + await for network calls and asyncio.to_thread for persistence to reduce event loop blocking. |
| .jules/bolt.md | Adds internal note documenting the async Groq client guidance. |
Files not reviewed (1)
- frontend/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| import time | ||
| import asyncio | ||
| from groq import Groq | ||
| from groq import Groq, AsyncGroq |
There was a problem hiding this comment.
Groq is imported but no longer used after switching to AsyncGroq. Please remove the unused import (or keep only one client type) to avoid lint/type-check failures and reduce confusion about which client should be used.
| from groq import Groq, AsyncGroq | |
| from groq import AsyncGroq |
| # Offloaded synchronous SQLite operation to asyncio.to_thread to prevent blocking the FastAPI event loop | ||
| rec_id = await asyncio.to_thread( | ||
| persistence.create_transfer_record, | ||
| room_name=request.room_name or "unknown", |
There was a problem hiding this comment.
asyncio.to_thread was added to offload SQLite calls, but it is only available in Python 3.9+. The README currently states the backend supports Python 3.8+, so this will raise AttributeError: module 'asyncio' has no attribute 'to_thread' on 3.8. Either bump the documented minimum Python version (and any CI/runtime constraints) to 3.9+ or replace this with a 3.8-compatible offload approach (e.g., AnyIO’s thread helper or loop.run_in_executor).
| global groq_client | ||
| if groq_client is None: | ||
| groq_client = Groq(api_key=GROQ_API_KEY) | ||
| chat_completion = groq_client.chat.completions.create( | ||
| # Switch to AsyncGroq from synchronous Groq to prevent network requests from blocking the event loop | ||
| groq_client = AsyncGroq(api_key=GROQ_API_KEY) | ||
| chat_completion = await groq_client.chat.completions.create( |
There was a problem hiding this comment.
The lazy global initialization of groq_client is not concurrency-safe: two concurrent requests can both observe groq_client is None and create separate AsyncGroq instances (with the earlier instance potentially leaked). Consider initializing the client once during app startup (lifespan) or protecting initialization with an asyncio.Lock / helper function that ensures a single instance is created.
💡 What
Refactored the FastAPI application to unblock the main event loop. Switched the synchronous
Groqclient to theAsyncGroqclient to await API network responses asynchronously in the/ai-voiceand/initiate-transferendpoints. Additionally, offloaded the synchronous SQLite persistence operations (create_transfer_record,set_agent_b,list_transfers, andget_transfer) to isolated threads viaasyncio.to_thread.🎯 Why
Synchronous network calls (via the standard
Groqclient) and direct SQLite persistence calls inside ofasync defFastAPI endpoints create significant performance bottlenecks. When the thread executing the event loop blocks, the server essentially halts, meaning it cannot process other concurrent incoming requests until the synchronous call is finished.📊 Impact
Using
AsyncGroqallows the thread executing the event loop to pause execution of that particular task and process other incoming network connections. Similarly,asyncio.to_threadmoves blocking file I/O operations entirely out of the event loop. The event loop responsiveness should see dramatic improvements during concurrent load, avoiding "0 heartbeat" freeze states.🔬 Measurement
Ensure that when executing the endpoints with
httpxconcurrently alongside a background heartbeat loop, the event loop yields properly rather than pausing execution. Backend integration tests (which trigger these APIs) have been confirmed to pass viapytest.PR created automatically by Jules for task 10252513326132528006 started by @Deepaksingh7238