[Bug] /crawl/job and /llm/job return HTTP 500 when jwt_enabled=true

[tuheyuan]

Bug Summary

When security.jwt_enabled: true is set in config.yml, the async job endpoints (POST /crawl/job, POST /llm/job, GET /crawl/job/{task_id}, GET /llm/job/{task_id}) return HTTP 500 Internal Server Error with AttributeError: 'Depends' object has no attribute 'credentials'.

All synchronous endpoints (/crawl, /md, /html, /screenshot, /pdf) work correctly with the same JWT.

Environment

• crawl4ai: 0.8.9 (Docker image unclecode/crawl4ai:latest)
• Source file: deploy/docker/job.py, line 100
• Deployed via docker compose; bind mount config.yml; env SECRET_KEY set

Reproduce

config.yml:

security:
  enabled: true
  jwt_enabled: true
  api_token: "<64-char hex>"

# 1. Get JWT
JWT=$(curl -sS -X POST http://localhost:8890/token \
  -H 'Content-Type: application/json' \
  -d '{"email":"admin@gmail.com","api_token":"<above>"}' \
  | jq -r .access_token)

# 2. Sync endpoint — works ✅
curl -X POST http://localhost:8890/crawl \
  -H "Authorization: Bearer $JWT" \
  -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com"]}'
# → 200 OK

# 3. Async endpoint — fails ❌
curl -X POST http://localhost:8890/crawl/job \
  -H "Authorization: Bearer $JWT" \
  -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com"]}'
# → 500 Internal Server Error

Stack Trace

[ERROR] Exception in ASGI application
Traceback (most recent call last):
  ...
  File "/app/server.py", line 263, in add_security_headers
    resp = await call_next(request)
  ...
  File "/app/auth.py", line 64, in verify_token
    if not credentials or not credentials.credentials:
                            ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Depends' object has no attribute 'credentials'

Root Cause Analysis

In deploy/docker/job.py:

@router.post("/crawl/job", status_code=202)
async def crawl_job_enqueue(
        payload: CrawlJobPayload,
        background_tasks: BackgroundTasks,
        _td: Dict = Depends(lambda: _token_dep()),   # ← line 100, BUG
):

Compare with the working pattern in deploy/docker/server.py (e.g., line 341):

async def get_markdown(
    request: Request,
    body: MarkdownRequest,
    _td: Dict = Depends(token_dep),   # ← works
):

The other endpoints pass token_dep (a function reference) directly to Depends(). FastAPI then calls token_dep() to get the jwt_required function, which FastAPI further resolves as a sub-dependency so its internal Depends(security) is evaluated and credentials becomes a real HTTPAuthorizationCredentials.

The job.py endpoints use Depends(lambda: _token_dep()) — the lambda is the dep, returns the jwt_required function object. FastAPI does not recursively resolve that returned function as a sub-dependency, so when jwt_required runs, Depends(security) is still a raw Depends object instead of the resolved HTTPAuthorizationCredentials. Hence credentials.credentials blows up.

Same bug exists at lines 59 (/llm/job), 90 (/llm/job/{task_id} status), 126 (/crawl/job/{task_id} status).

Suggested Fix

Replace the Depends(lambda: _token_dep()) pattern with Depends(_token_dep) (no parens):

# job.py, lines 59, 90, 100, 126
_td: Dict = Depends(_token_dep),

This passes the dep function directly, letting FastAPI resolve it normally — same as how Depends(token_dep) works in server.py.

A small refactor is needed because _token_dep is set as a module-global by init_job_router(). The fix may need to capture _token_dep in a closure or use functools.partial.

Impact

• Severity: Medium — async task endpoints are unusable when JWT auth is enabled
• Workaround: Use synchronous /crawl (returns full result inline, not a task_id). Same parameters, just no async/polling pattern.
• Affects: All 4 endpoints in deploy/docker/job.py

Tested Versions

• crawl4ai 0.8.9 (Docker image unclecode/crawl4ai:latest, 2026-06-16)
• Did not test on older versions, but the pattern appears in the latest source

[nightcityblade]

Hi, I'd like to work on this. I'll submit a PR shortly.

[nightcityblade]

I checked current main and this appears to already be fixed there: deploy/docker/job.py now uses the request principal dependency for the async job endpoints instead of the old Depends(lambda: _token_dep()) pattern present in v0.8.9. Since there isn't a remaining code change for me to make on main, I'm going to step back/unclaim this.

[samuelpert]

Hello, I've talked to the maintainer Som and he said this is a good first issue for a first time contributor. I am taking a Codepath course AI301 under the lead of professor Jake Ascher, where I need to make OSS contributions. I will take a look and understand the issue first.

[samuelpert]

I checked the current main branch (v0.9.0) and the Depends(lambda: _token_dep()) pattern is no longer present in deploy/docker/job.py. Looks like this was resolved in the upgrade to 0.9.0. Can this issue be closed?

[ntohidi]

This was fixed in v0.9.0 — the job endpoints now use the proper _principal_dep dependency instead of the broken Depends(lambda: _token_dep()) pattern.

Closing this issue. Feel free to reopen or ask if you have any questions!