Skip to content

test(e2e): accept 422 alongside 404 for unfetchable page - #48

Closed
ilya-bogin-keenable wants to merge 1 commit into
mainfrom
fix/dead-page-422
Closed

test(e2e): accept 422 alongside 404 for unfetchable page#48
ilya-bogin-keenable wants to merge 1 commit into
mainfrom
fix/dead-page-422

Conversation

@ilya-bogin-keenable

Copy link
Copy Markdown
Contributor

What

Last night's e2e run failed on all three platforms (Linux/macOS/Windows) at a single test: test_fetch.py::test_dead_page.

AssertionError: assert 'Unprocessable entity' == 'Not found'

Why

The fetch backend now rejects an unfetchable page (https://example.com/nonexistent-xyz-12345) with a 422 "Unprocessable entity" instead of a 404 "Not found", depending on how far the upstream fetch got. Reproduced locally against the live API — it returns either string. The CLI contract (exit code 1 + structured YAML error) is unchanged; only the backend's error label shifted.

Fix

test_dead_page now accepts either "Not found" or "Unprocessable entity" instead of pinning the exact string. Both are valid "this page can't be fetched" outcomes.

Verified locally: pytest tests/e2e/test_fetch.py → 7 passed.

🤖 Generated with Claude Code

The fetch backend now rejects a dead page with a 422 ("Unprocessable
entity") instead of a 404 ("Not found") depending on how far the upstream
fetch got. Both are valid "this page can't be fetched" outcomes, so
test_dead_page now accepts either error string instead of pinning the
exact one — the CLI contract (exit code 1 + structured error) is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

test(e2e): accept 422 or 404 for unfetchable page

🧪 Tests 🕐 Less than 10 minutes

Grey Divider

AI Description

• Relax test_dead_page to accept either 404 or 422 backend error labels.
• Prevent e2e flakes when upstream fetch failures map to different HTTP outcomes.
• Keep validating the CLI contract (exit code 1 + YAML error payload).
Diagram

graph TD
  A["E2E test"] --> B["CLI fetch"] --> C["Fetch backend"] --> D["Target URL"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Assert on a stable error code (not localized/worded message)
  • ➕ Less brittle than matching human-readable strings
  • ➕ Better forwards-compatibility if backend wording changes again
  • ➖ Requires a stable machine-readable field in the YAML payload (or adding one)
  • ➖ Potentially larger cross-component change than a test-only fix
2. Normalize backend errors in the CLI layer
  • ➕ Keeps tests strict while insulating users from backend label drift
  • ➕ Centralizes mapping of equivalent error conditions
  • ➖ Adds maintenance burden to keep mappings aligned with backend semantics
  • ➖ May hide meaningful backend distinctions if over-normalized

Recommendation: The PR’s approach (accept either "Not found" or "Unprocessable entity") is appropriate for an e2e test whose intent is "unfetchable page returns exit code 1 and a structured error". If this error label is meant to be stable long-term, consider introducing/asserting a machine-readable error code (or normalizing in the CLI) to avoid future string drift.

Files changed (1) +4 / -1

Tests (1) +4 / -1
test_fetch.pyAllow either 404 or 422 error label for dead-page fetch +4/-1

Allow either 404 or 422 error label for dead-page fetch

• Updates 'test_dead_page' to accept both "Not found" (404) and "Unprocessable entity" (422) as valid error strings for an unfetchable URL. Adds clarifying comments explaining the upstream-dependent variability.

tests/e2e/test_fetch.py

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Qodo Logo

@qodo-code-review

Copy link
Copy Markdown

CI Feedback 🧐

A test triggered by this PR failed. Here is an AI-generated analysis of the failure:

Action: fmt · clippy · test · audit

Failed stage: audit [❌]

Failed test name: ""

Failure summary:

The action failed during the cargo audit step because RustSec reported a security vulnerability in
the dependency tree, causing cargo audit to exit with code 1.
- Vulnerability found in quinn-proto
0.11.14 (RUSTSEC-2026-0185): remote memory exhaustion via unbounded out-of-order stream reassembly;
solution is to upgrade to >=0.11.15.
- There was also 1 allowed warning for anyhow 1.0.102
(RUSTSEC-2026-0190) marked as unsound, but the job failed due to the vulnerability.

Relevant error logs:
1:  ##[group]Runner Image Provisioner
2:  Hosted Compute Agent
...

157:  path: ~/.cargo/registry
158:  ~/.cargo/git
159:  ~/.cargo/bin
160:  target
161:  
162:  key: cargo-Linux-6a53416847cb553d0aaaed269887a76452774c39b82801298b7d3361606dfaf2
163:  restore-keys: cargo-Linux-
164:  enableCrossOsArchive: false
165:  fail-on-cache-miss: false
166:  lookup-only: false
167:  save-always: false
168:  ##[endgroup]
169:  (node:2331) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
170:  (Use `node --trace-deprecation ...` to show where the warning was created)
171:  Cache hit for: cargo-Linux-6a53416847cb553d0aaaed269887a76452774c39b82801298b7d3361606dfaf2
172:  (node:2331) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
173:  Received 0 of 321681281 (0.0%), 0.0 MBs/sec
...

186:  �[36;1mcargo clippy --locked --all-targets --all-features -- -D warnings�[0m
187:  shell: /usr/bin/bash -e {0}
188:  ##[endgroup]
189:  Checking keenable-cli v0.1.22 (/home/runner/work/keenable-cli/keenable-cli)
190:  Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.80s
191:  ##[group]Run cargo test --locked --all-features
192:  �[36;1mcargo test --locked --all-features�[0m
193:  shell: /usr/bin/bash -e {0}
194:  ##[endgroup]
195:  Compiling keenable-cli v0.1.22 (/home/runner/work/keenable-cli/keenable-cli)
196:  Finished `test` profile [unoptimized + debuginfo] target(s) in 0.93s
197:  Running unittests src/main.rs (target/debug/deps/keenable-102c11edab353715)
198:  running 20 tests
199:  test api::tests::app_title_defaults_when_env_absent_or_blank ... ok
200:  test api::tests::app_title_uses_trimmed_override ... ok
201:  test api::tests::auth_error_covers_401_403_and_400_auth ... ok
202:  test api::tests::display_joins_error_and_message ... ok
203:  test api::tests::rate_limit_is_429_only ... ok
204:  test commands::ide::tests::conflicting_name_is_substring_and_case_insensitive ... ok
205:  test commands::ide::tests::extract_entry_api_key_from_headers_and_legacy_args ... ok
206:  test commands::ide::tests::extract_url_handles_all_styles ... ok
207:  test commands::ide::tests::http_entry_shape_per_ide ... ok
208:  test commands::ide::tests::keenable_url_matches_prod_and_test ... ok
209:  test commands::ide::tests::codex_toml_entry_uses_http_headers ... ok
210:  test commands::search::tests::config_mode_accepts_valid_and_legacy_standard ... ok
211:  test commands::search::tests::to_json_includes_only_set_filters ... ok
212:  test commands::search::tests::to_json_omits_unset_filters ... ok
213:  test update::tests::newer_only_when_strictly_greater ... ok
214:  test update::tests::rejects_non_numeric ... ok
215:  test update::tests::parses_plain_and_v_prefixed ... ok
216:  test update::tests::shorter_version_compares_lexicographically ... ok
217:  test update::tests::unparseable_is_never_newer ... ok
218:  test commands::search::tests::config_mode_ignores_invalid_or_missing ... ok
219:  test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
220:  ##[group]Run command -v cargo-audit >/dev/null 2>&1 || cargo install cargo-audit --locked
221:  �[36;1mcommand -v cargo-audit >/dev/null 2>&1 || cargo install cargo-audit --locked�[0m
222:  shell: /usr/bin/bash -e {0}
223:  ##[endgroup]
224:  ##[group]Run cargo audit
225:  �[36;1mcargo audit�[0m
226:  shell: /usr/bin/bash -e {0}
227:  ##[endgroup]
228:  Fetching advisory database from `https://github.com/RustSec/advisory-db.git`
229:  Loaded 1145 security advisories (from /home/runner/.cargo/advisory-db)
230:  Updating crates.io index
231:  Scanning Cargo.lock for vulnerabilities (212 crate dependencies)
232:  error: 1 vulnerability found!
233:  warning: 1 allowed warning found
234:  Crate:     quinn-proto
235:  Version:   0.11.14
236:  Title:      Remote memory exhaustion in quinn-proto from unbounded out-of-order stream reassembly
237:  Date:      2026-06-22
238:  ID:        RUSTSEC-2026-0185
239:  URL:       https://rustsec.org/advisories/RUSTSEC-2026-0185
240:  Severity:  7.5 (high)
241:  Solution:  Upgrade to >=0.11.15
242:  Crate:     anyhow
243:  Version:   1.0.102
244:  Warning:   unsound
245:  Title:     Unsoundness in `Error::downcast_mut()`
246:  Date:      2026-06-25
247:  ID:        RUSTSEC-2026-0190
248:  URL:       https://rustsec.org/advisories/RUSTSEC-2026-0190
249:  ##[error]Process completed with exit code 1.
250:  Post job cleanup.

@IlyaGusev IlyaGusev closed this Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants