Add hanzi factorization script - #866
Open
klei22 wants to merge 2 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a vendored “hanzi-factor” utilities/documentation bundle (under data/hanzi_factor/hanzi-factor-utils-0.3.0/) containing scripts and docs for Hanzi IDS factorization, normalization, and coverage auditing, and adds a new exploration config for comparing factorized Chinese datasets.
Changes:
- Added a new exploration YAML (
explorations/zh_factorization_compare.yaml) to run comparative training sweeps over two “factored_zhongwen” dataset variants. - Added a
hanzi-factor-utils-0.3.0subtree with CLI scripts (scripts/*.py), demo instructions (demos/*), format documentation (docs/FORMAT.md), and licensing/third-party notes. - Added pre-generated coverage artifacts and accompanying documentation under
artifacts/.
Reviewed changes
Copilot reviewed 16 out of 18 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| explorations/zh_factorization_compare.yaml | New exploration spec for comparing two Chinese factorization datasets across optimizer/norm/attention variants. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/THIRD_PARTY.md | Documents external decomposition sources and optional OpenCC dependency licensing. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/scripts/text_to_ids.py | CLI to transform UTF-8 text by replacing covered Han characters with prefix IDS. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/scripts/normalize_chinese.py | CLI wrapper around OpenCC-based Simplified/Traditional normalization. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/scripts/list_uncovered.py | CLI to extract uncovered characters from a JSON coverage report. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/scripts/ids_to_text.py | CLI to restore prefix-IDS mixed text back to ordinary text via catalogue reverse lookup. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/scripts/fetch_ccd.py | Helper to fetch and integrity-verify a pinned CCD dataset snapshot from npm. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/scripts/coverage_utf.py | Coverage audit driver over Unicode targets and local datasets, with report outputs and fail gates. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/README.md | Comprehensive README describing concepts, CLI usage, APIs, and audit methodology. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/LICENSE | MIT license for the implementation bundle. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/examples/roundtrip.py | Example script demonstrating Hanzi → IDS → bytes → IDS → Hanzi round trips. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/docs/FORMAT.md | Specifies the binary codec payload and framing format (version 1). |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/demos/setup_and_demo.sh | End-to-end demo automation (venv, fetch catalogue, normalize, factorize, restore, verify). |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/demos/SETUP_AND_DEMO.md | Step-by-step instructions and explanation for the demo workflow. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/artifacts/README.md | Explains how included coverage artifacts were produced and how to regenerate them. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/artifacts/coverage_ccd_u4e00_u9fff_strict.json | Pre-generated strict JSON coverage report for U+4E00–U+9FFF with pinned CCD snapshot. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/artifacts/uncovered_ccd_u4e00_u9fff_bijective.tsv | Extracted uncovered list (TSV) derived from the strict JSON coverage report. |
| data/hanzi_factor/hanzi-factor-utils-0.3.0/.gitignore | Ignore rules intended for the vendored hanzi-factor utilities workspace outputs/caches. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -0,0 +1,113 @@ | |||
| # comparison_adam_muon_hsnorm_relu2_softmax_peri.yaml using named group mechanisms from sample.yaml | |||
Comment on lines
+55
to
+60
| with tarfile.open(fileobj=io.BytesIO(tarball), mode="r:gz") as archive: | ||
| member = archive.getmember(MEMBER) | ||
| stream = archive.extractfile(member) | ||
| if stream is None: | ||
| raise ValueError(f"{MEMBER!r} is not a regular archive member") | ||
| return stream.read() |
Comment on lines
+19
to
+20
| SAMPLE_FILE="${REPO_ROOT}/demos/sample_chinese.txt" | ||
|
|
Comment on lines
+43
to
+47
| ```bash | ||
| python -m pip install -e . | ||
| python examples/roundtrip.py | ||
| python -m unittest discover -s tests -v | ||
| ``` |
Comment on lines
+33
to
+35
| step "Installing hanzi-factor with phrase-aware normalization support" | ||
| "${VENV_PYTHON}" -m pip install -e "${REPO_ROOT}[normalize]" | ||
|
|
Comment on lines
+3
to
+5
| These reports were generated by `hanzi-factor` 0.1.0 over the inclusive range | ||
| U+4E00–U+9FFF using the pinned external CCD snapshot fetched by | ||
| `scripts/fetch_ccd.py` (`chinese-characters-decomposition` 0.1.0). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds comprehensive documentation, licensing, and configuration files for the
hanzi-factorPython package. It includes a detailedREADME.mdexplaining the package's purpose, usage, and APIs, as well as supporting documentation for third-party data, demo setup, coverage artifacts, and licensing. Additionally, a.gitignoreis provided to standardize repository hygiene.Documentation and Usage Guides:
README.mddescribing the package's features, usage examples, CLI commands, Python APIs, data formats, audit methodology, and references.demos/SETUP_AND_DEMO.mdwith step-by-step instructions for running an end-to-end demo, including environment setup, normalization, IDS conversion, and round-trip verification.artifacts/README.mddocumenting coverage reports, how to regenerate them, and interpretation of audit results.Third-party Data and Licensing:
THIRD_PARTY.mdclarifying the use of external decomposition catalogues and optional OpenCC normalization dependencies, along with their licenses and data quality considerations.LICENSEfile for the implementation, specifying terms and conditions.Repository Configuration:
.gitignoreto exclude OS/editor files, Python artifacts, virtual environments, build outputs, caches, and generated data from version control.