[FEAT]: Interactive Voice Feedback Loop for Missing Incident Data

[yassinezadod]

---

name: 🚀 Feature Request
about: Suggest an idea or a new capability for FireForm.
title: "[FEAT]: Interactive Voice Feedback Loop for Missing Incident Data"
labels: enhancement
assignees: ''

---

📝 Description

Implementation of a "Human-in-the-loop" interactive feedback system. If the LLM (Mistral) fails to extract a mandatory field from the transcript, the system will use Text-to-Speech (TTS) to verbally ask the responder for the missing information.

💡 Rationale

In high-stress emergency environments, responders need a hands-free experience. If a recording is unclear, the current system simply leaves a "-1" or empty field. This feature ensures 100% data integrity without requiring the user to stop and type manually.

🛠️ Proposed Solution

• Integrate a lightweight TTS engine (like pyttsx3 or gTTS) in src/backend.py.
• Logic change in textToJSON to trigger a question if a value is returned as "-1".
• Update to requirements.txt for audio dependencies.
• New prompt for Mistral to generate concise clarification questions.

✅ Acceptance Criteria

• Feature works in Docker container (with audio device access).
• System successfully triggers a voice prompt when a field is missing.
• JSON output is updated based on the voice response.

📌 Additional Context

This leverages local AI processing to maintain privacy while ensuring the final PDF is fully populated before it is sent to the UN HQ.

[Gopisokk]

I have implemented the core logic and backend infrastructure for this feature in PR #200.

How it was implemented:
Rather than tightly coupling local Python TTS libraries (pyttsx3/gTTS) inside the backend container (which scales poorly and causes Docker audio-passthrough issues), I implemented a robust, stateful "Human-in-the-Loop" API architecture. This allows the frontend/client to handle the actual TTS/microphone interaction, which is the standard pattern for decoupled web apps.

Key Changes in this PR:

1. Stateful Database Schema: Upgraded FormSubmission to store partial form states (status="missing_data", extracted_data, and missing_fields).
2. Decoupled Extraction Core: Refactored llm.py and file_manipulator.py to identify missing -1 fields without crashing or forcing an incomplete PDF build.
3. Interactive Feedback Endpoint: Added POST /forms/{id}/feedback. When the frontend asks the user for the missing fields, it sends the follow-up text here. The LLM processes only the missing fields, merges the JSON data, and finally generates the completed PDF.

Note for Frontend Integration:
To complete this feature, the frontend just needs to look for status: "missing_data" in the API response, use browser-native TTS (or another client-side TTS solution) to read the missing_fields list to the user, and send the user's second audio transcript to the new /feedback endpoint.