fix: Python SDK canonical encoding (ensure_ascii=True) to match TypeScript#91
fix: Python SDK canonical encoding (ensure_ascii=True) to match TypeScript#91catchmeifyoucaan wants to merge 1 commit into
Conversation
…rue) The Python SDK used ensure_ascii=False in json.dumps during canonicalization, while the TypeScript SDK uses JSON.stringify which defaults to escaping non-ASCII characters. This caused cross-SDK hash disagreement for any protocol message containing non-ASCII Unicode (accents, CJK, emoji, etc.), breaking event hash chain integrity. JA-TM-01 in the Jarvis threat model.
📝 WalkthroughWalkthrough
ChangesProtocol canonicalization
Estimated code review effort: 1 (Trivial) | ~2 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 063797eb75
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| def canonicalize_protocol_value(value: Any) -> str: | ||
| return json.dumps(_sort_protocol_value(value), separators=(",", ":"), ensure_ascii=False) | ||
| return json.dumps(_sort_protocol_value(value), separators=(",", ":"), ensure_ascii=True) |
There was a problem hiding this comment.
For protocol records containing non-ASCII text (for example a worker display name, review note, or evidence description), this changes Python canonicalization away from the TypeScript helper: packages/typescript/src/index.js uses JSON.stringify(sortProtocolValue(value)), which emits {"text":"Café ☕"}, while ensure_ascii=True emits {"text":"Caf\u00e9 \u2615"}. Because hash_protocol_value hashes these bytes, Python and TypeScript now produce different event hashes for valid Unicode protocol records, breaking cross-SDK hash-chain verification.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/python/src/jarvis_protocol/__init__.py`:
- Line 1733: Update the protocol JSON serialization call in the canonicalization
function to use ensure_ascii=False so Unicode characters remain literal and
match JSON.stringify hashing behavior. Add regression coverage for BMP
characters, astral characters, and U+2028/U+2029, while preserving the existing
sorting and compact separator behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: e0d19e8d-34ae-4d97-8c86-ad323761fbce
📒 Files selected for processing (1)
packages/python/src/jarvis_protocol/__init__.py
|
|
||
| def canonicalize_protocol_value(value: Any) -> str: | ||
| return json.dumps(_sort_protocol_value(value), separators=(",", ":"), ensure_ascii=False) | ||
| return json.dumps(_sort_protocol_value(value), separators=(",", ":"), ensure_ascii=True) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate protocol serialization functions =="
rg -n "canonicalize_protocol_value|hash_protocol_value|JSON.stringify|ensure_ascii" packages -S
echo
echo "== map python file shape =="
ast-grep outline packages/python/src/jarvis_protocol/__init__.py --view expanded | sed -n '1,220p'
echo
echo "== map possible TypeScript protocol files =="
fd -a -t f ".*" packages | rg "typescript|ts|jarvis_protocol|protocol"Repository: Flow-Research/jarvis
Length of output: 12495
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
samples = [
{"text": "café"},
{"text": "𝄞"},
{"text": "é😀"},
{"text": "\u2028"},
]
for v in samples:
py = json.dumps(v, separators=(",", ":"), ensure_ascii=True)
py_no = json.dumps(v, separators=(",", ":"), ensure_ascii=False)
print(v, "ensure_ascii=True ", py)
print(v, "ensure_ascii=False", py_no)
PY
node - <<'JS'
const samples = [
{text: "café"},
{text: "𝄞"},
{text: "é😀"},
{text: "\u2028"},
];
for (const v of samples) {
console.log(v, JSON.stringify(v));
}
JSRepository: Flow-Research/jarvis
Length of output: 715
Do not canonicalize protocol JSON with ensure_ascii=True.
Python escapes non-ASCII characters (café → caf\u00e9, astral chars → surrogate escapes), while JSON.stringify keeps them literal. Since this string is hashed, Python and TypeScript will diverge for Unicode payloads. Use ensure_ascii=False (or make both SDKs use the same escaping) and add regression tests for BMP, astral, and U+2028/U+2029 cases.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/python/src/jarvis_protocol/__init__.py` at line 1733, Update the
protocol JSON serialization call in the canonicalization function to use
ensure_ascii=False so Unicode characters remain literal and match JSON.stringify
hashing behavior. Add regression coverage for BMP characters, astral characters,
and U+2028/U+2029, while preserving the existing sorting and compact separator
behavior.
Source: MCP tools
1 participant
Summary
The Python SDK used \ensure_ascii=False\ in \json.dumps\ during \canonicalize_protocol_value(), while the TypeScript SDK uses \JSON.stringify\ which defaults to escaping non-ASCII characters. This caused cross-SDK hash disagreement for any protocol message containing non-ASCII Unicode (accents, CJK, emoji, etc.), breaking event hash chain integrity.
Change
\\diff
\\
Why
Verification
Finds: JA-TM-01
Summary by CodeRabbit