Test failures: Optional 'chapter rhythm classifier' pass aborts worker pipeline on JSONDecodeError

[CyberSecDef]

Problem

Some recent test failures (visible in CI and PR #154) occur when the "chapter rhythm classifier" agent pass fails to parse its LLM output as JSON. Instead of gracefully degrading, the pipeline aborts early, resulting in missing progress updates and status never being set to "done".

Error symptoms (from CI logs):

• parse_llm_json: no JSON structure found in response (strategy=direct): Processed output from the LLM agent.
• Chapter-agent pass FAILED — pass='chapter rhythm classifier' ... error=JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This causes:

• 0 in-memory step updates (TestProgressSnapshotThrottling.test_set_step_updates_in_memory_always)
• 0 disk writes (test_fewer_disk_writes_than_step_updates)
• status never set to "done" (TestGenerationStatusOrdering.test_done_only_set_after_all_post_manuscript_passes)

Solution

Make the "chapter rhythm classifier" pass robust to JSON parsing failures by catching json.JSONDecodeError (and other relevant exceptions). For optional passes, inject a fallback result using PASS_FAILURE_KEY and _log_pass_failure, including human-readable error details and minimal safe defaults. This ensures the worker pipeline can proceed, recording normal progress and updating status at the correct times.

Example patch pattern

from novelforge.agents.chapter._helpers import PASS_FAILURE_KEY, _log_pass_failure
from json import JSONDecodeError

try:
    ...
    data = parse_llm_json(raw)
    ...
except (JSONDecodeError, Exception) as exc:
    return {
        PASS_FAILURE_KEY: _log_pass_failure(
            exc, pass_name="chapter rhythm classifier", chapter_num=chapter_num, chapter_title=title, optional=True,
        ),
        "rhythm": "unknown", "notes": [],
    }

Once fixed, CI should record step updates and eventually set status to "done", allowing test passes as intended.