fix(security): protect rag streaming endpoints with internal auth

[Namraa310806]

Summary

This PR fixes a security issue where protected RAG service endpoints could potentially be accessed directly, bypassing the intended API gateway protections.

The update strengthens service-to-service authentication by protecting streaming endpoints, introducing prefix-based route protection, normalizing route matching behavior, and adding comprehensive regression tests to prevent future authentication bypasses.

---

Changes Made

Internal Authentication Protection

• Added explicit protection for /ask/stream.
• Added protection for additional sensitive RAG endpoints.
• Enforced validation of X-Internal-Token for protected routes.

Prefix-Based Route Protection

Added subtree-level protection for:

/ask/*
/processing-status/*

This ensures any future nested endpoints are automatically protected without requiring manual updates.

Examples:

/ask/v2
/ask/v2/stream
/ask/new-feature
/processing-status/123

are now protected automatically.

Route Normalization

Added path normalization to prevent authentication bypass through trailing slash variations.

Examples:

/ask/stream
/ask/stream/
/process-pdf
/process-pdf/

are now treated identically by the middleware.

Secure Token Validation

• Uses secrets.compare_digest() for constant-time token comparison.
• Prevents timing-based token validation attacks.

Fail-Closed Security Behavior

When INTERNAL_RAG_TOKEN is not configured:

• Protected endpoints return HTTP 503.
• Requests are never processed accidentally.
• Service remains secure by default.

Regression Test Coverage

Added automated tests covering:

• Missing token rejection
• Invalid token rejection
• Valid token acceptance
• /ask/stream protection
• Prefix-based protection
• Trailing slash normalization
• Missing configuration behavior
• Protected endpoint enforcement

---

Security Impact

This change prevents direct access to internal RAG service endpoints when the service is exposed outside the intended gateway architecture.

The fix reduces the risk of:

• Gateway authentication bypass
• Rate-limit bypass attacks
• Unauthorized direct access to RAG endpoints
• Future route protection regressions
• Misconfiguration-related exposure

---

Files Modified

• rag-service/main.py
• rag-service/tests/test_main.py

---

Verification Checklist

• /ask/stream protected
• Prefix-based protection added
• Trailing slash normalization implemented
• Constant-time token validation preserved
• Fail-closed behavior implemented
• Regression tests added
• Existing functionality preserved
• No breaking API changes introduced

---

Related Issue

Fixes: #500

[coderabbitai]

Warning

Review limit reached

@Namraa310806, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 25 minutes and 5 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: df76381d-7b30-4454-81f1-bf98e1dbf9bc

📥 Commits

Reviewing files that changed from the base of the PR and between 5590b87 and 6733da2.

📒 Files selected for processing (2)

• rag-service/main.py
• rag-service/test_main.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Namraa310806]

Suggested GSSoC Labels: level:critical quality:exceptional type:security type:testing

[vercel]

@Namraa310806 is attempting to deploy a commit to the firefistisdead's projects Team on Vercel.

A member of the Team first needs to authorize it.