Conversation
…ve stack - Add prerequisites section defining `hey` (brew install hey, now installed) - Replace all placeholder endpoints with real values from tf-outputs.json - Replace token_type:taj with correct token_type:raja; add admin key header - Use pinned package hashes for all scale tiers (1k/10k/100k/1m) - Create scale packages via Quilt Packaging Engine (SQS) in data-yaml-spec-tests - Replace localhost:9901 admin stats with `aws ecs execute-command` approach - Replace auth-disable vagueness with `terraform apply -var auth_disabled=true` - Remove docker-compose echo server step (no local testing) - Remove pytest performance marker approach; hey is the benchmark tool - Move CI regression gate to Out-of-scope - Fix security audit results: downgrade docker-compose finding to Low severity Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…peline - Add scale/1k-1m to seed-config.yaml with pre-built URIs pointing to data-yaml-spec-tests; seed_packages.py skips S3/quilt push and only creates DataZone listings + subscription grants for packages with a uri field - Add perf_test_bucket Terraform variable (default: data-yaml-spec-tests); grant RALE router Lambda read access and include bucket in RAJEE_PUBLIC_PATH_PREFIXES so the auth-disabled baseline can reach it - Add verify_perf_access.py: reads principal + package URI from .rale-seed-state.json, probes /token then Envoy end-to-end; gates deploy - Wire _verify-perf-access into deploy sequence; expose as ./poe verify-perf - Fix performance spec URLs from Quilt catalog format (/b/.../packages/...) to RALE USL format (/<bucket>/<author>/<name>@<hash>) - Add lesson-learned note to spec documenting the two gaps (IAM + DataZone) that caused all 2026-03-23 benchmark requests to fail - Remove stale ernest-test from RAJA_USERS (.env); was never an IAM user Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… ECS exec
_extract_principal in the RALE authorizer called json.loads directly on the
x-raja-jwt-payload header. Envoy's forward_payload_header forwards the JWT
payload as base64url, not plain JSON, so the parse silently failed and the
function fell through to the ECS task role ARN — which is not in any DataZone
project — causing every authorized request to return 403.
Fix mirrors the guard already in authorize.lua: check if the value starts with
'{'; if not, base64url-decode it first. Removes the x-raja-principal workaround
from verify_perf_access.py; the benchmark now uses real SigV4-issued tokens.
Also enable ECS execute-command on the RAJEE service so the Envoy admin stats
step in the performance spec can be completed via 'aws ecs execute-command'.
Update 03b-live-performance-results.md with root-cause analysis and resolution
notes for both blockers from the 2026-03-23 run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors the retry logic in tests/integration/test_rale_end_to_end.py. Transient 503s occur while ECS replaces tasks after a service update. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add ECSExec policy statement (ssmmessages:*) to rajee_task_permissions so ECS execute-command works for Envoy admin stats collection - Fix verify_perf_access.py to request token_type "rajee" (not "raja") so the issued JWT passes Envoy's jwt_authn filter validation All three verify_perf_access.py checks now pass: ✓ /token → 200 ✓ Envoy GET /data-yaml-spec-tests/scale/1k@40ff9e73 → 200 ✓ ECS execute-command → 200 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…enchmarking Adds a dedicated per-route bypass of the jwt_authn + lua filters, scoped to the perf test bucket, so the baseline can be measured without toggling auth_disabled on the live stack. Terraform passes PERF_DIRECT_BUCKET from var.perf_test_bucket to the ECS task automatically. verify_perf_access.py now checks direct (no-token) access first, making IAM and route config failures immediately visible before the auth checks. Removes the --exercise-auth-toggle path and all terraform-apply logic. The performance audit spec is updated to use the direct route for the baseline instead of the auth_disabled toggle cycle. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…UCKET unset The env var is injected into the ECS task by Terraform but is not present in .env when the verify script runs locally during ./poe deploy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous route matched /{perf_bucket}/ which hijacked the auth-enabled
test URL. Switch to a dedicated /_perf/ prefix with prefix_rewrite "/" so
normal auth paths are unaffected.
Add s3_perf_upstream cluster with aws_request_signing (service_name: s3)
so the direct route can access private buckets using the ECS task role.
Add rajee_task_perf_bucket IAM policy granting the Envoy task role
s3:GetObject / s3:ListBucket on perf_test_bucket (mirrors the existing
rale_router_perf_bucket policy).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… check URL
/_perf/ now routes to s3_perf_upstream (aws_request_signing service=s3),
not rale_router_cluster. No auth filters. prefix_rewrite "/" strips the
/_perf prefix before forwarding to S3.
verify_perf_access.py direct check hits /_perf/{bucket}/ (bucket root).
Accepts 200 or 403 as success — both prove S3 was reached, not Envoy auth
(which would return 401).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix path description: /{perf_bucket}/... → /_perf/{perf_bucket}/...
- Fix token_type in both hey benchmark examples: raja → rajee
- Add live performance results doc
- Update tf-outputs.json from latest deploy
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Closed
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review.
Tip: disable this comment in your organization's Code Review settings.
…l host - Add request_headers_to_remove to envoy.yaml.tmpl so clients cannot supply a forged JWT payload header; Lua now only sees the value written by jwt_authn after successful verification - Fix deny response metadata leak in rale_authorizer (manifest_hash, package_name, registry no longer exposed) - Fix hard-coded /tmp in Lambda handlers (use tempfile.gettempdir()) - Fix strict mypy: Lambda handler dirs promoted to packages - Upgrade dependency lockfile: fastapi, starlette, mangum, boto3, ruff, pydantic-core and others brought to current releases - Update audit summary and changelog per review feedback Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drernie
changed the title
drernie
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full audit results across three dimensions — performance, security, and quality — for the RAJA MVP. See
specs/23-audits/04-audit-summary.mdfor the canonical summary.Performance Audit
Direct S3 baseline (
scale/1k, n=100): P50 0.924 s · P95 1.096 s · P99 7.106 sscale/1mscale/100kscale/10kscale/1k† P99 baseline (7.1 s) was inflated by a single outlier; these tiers beat it by chance.
P50 is flat across all tiers at ~0.91 s. The JWT+Lua filter chain adds negligible median cost — S3 fetch dominates.
P99 is noise, not tier-driven. Tail values are driven by ECS task cycling (IMDS credential-refresh bursts), not package size or auth overhead.
503s only on
scale/1k. Correlated with IMDS bursts during credential refresh, not a routing problem.Decision: no optimization required. The 15 % P99 threshold cannot be reliably evaluated from this data set. Recommended follow-up: re-run baseline at n=1 000 to reduce noise; investigate IMDS 503 recurrence as a separate issue.
Security Audit
infra/terraform/main.tfauthorization = "NONE"on all resources; no resource policy, throttling, or access logginginfra/terraform/main.tfprincipal = "*"constrained only bysource_accountinfra/terraform/main.tfs3:*over both buckets; control plane can mutate Lambda config and write secretsinfra/terraform/main.tfThese are infrastructure hardening gaps, not flaws in the core authorization design.
Quality Audit
src/raja/enforcer.py,src/raja/token.pylambda_handlers/rale_authorizer/handler.py.github/workflows/ci.ymlbandit,pip-audit,vulture, or coverage thresholdsArtifacts
specs/23-audits/01a-code-audit-results.md— quality findingsspecs/23-audits/02a-security-audit-results.md— security findingsspecs/23-audits/03f-live-performance-results.md— raw performance numbersspecs/23-audits/04-audit-summary.md— consolidated findings and decisions🤖 Generated with Claude Code