Add BNF support package for bidirectional DS/Dsp syntax conversion - #68
Conversation
Co-authored-by: hzhangxyz <11623447+hzhangxyz@users.noreply.github.com>
Co-authored-by: hzhangxyz <11623447+hzhangxyz@users.noreply.github.com>
Co-authored-by: hzhangxyz <11623447+hzhangxyz@users.noreply.github.com>
Co-authored-by: hzhangxyz <11623447+hzhangxyz@users.noreply.github.com>
|
package.json 和 pyproject.toml 都应直接放在 bnf/ 下面, 包名就用 apyds-bnf 和 atsds-bnf 吧. 另外, 在package.json中调用antlr, 比如 而python这边最好也要用pythonic的方式实现,但是我不知道应该怎么做,你自己看着来 @copilot |
…yds-bnf Co-authored-by: hzhangxyz <11623447+hzhangxyz@users.noreply.github.com>
Done! Restructured the package as requested: Changes in commit 1d53c6a:
The Python setup now automatically generates ANTLR parsers when running |
There was a problem hiding this comment.
Pull request overview
This PR implements a comprehensive BNF support package that enables bidirectional conversion between DS's lisp-like syntax and a traditional infix notation (Dsp). The implementation follows a mono-repo layout within the bnf/ directory and provides both JavaScript/TypeScript and Python implementations using ANTLR4-generated parsers.
Key Changes:
- ANTLR4 grammar definitions for both Ds (lisp-like) and Dsp (traditional infix) syntaxes
- Complete JavaScript/TypeScript implementation with ES modules and type definitions
- Complete Python implementation with CLI tools (
ds-unparse,ds-parse) - Comprehensive documentation, examples, and test suites for both languages
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| bnf/grammars/Ds.g4 | ANTLR grammar for lisp-like DS syntax |
| bnf/grammars/Dsp.g4 | ANTLR grammar for traditional infix syntax |
| bnf/grammars/README.md | Grammar design notes and trade-offs documentation |
| bnf/javascript/package.json | NPM package configuration with dependencies and scripts |
| bnf/javascript/tsconfig.json | TypeScript compiler configuration |
| bnf/javascript/src/index.ts | Main export file for JavaScript package |
| bnf/javascript/src/unparse.ts | Ds to Dsp converter implementation |
| bnf/javascript/src/parse.ts | Dsp to Ds converter implementation |
| bnf/javascript/tests/conversion.test.js | JavaScript test suite |
| bnf/javascript/README.md | JavaScript-specific documentation |
| bnf/python/pyproject.toml | Python package configuration |
| bnf/python/ds_bnf/init.py | Python package initialization |
| bnf/python/ds_bnf/unparse.py | Ds to Dsp converter implementation |
| bnf/python/ds_bnf/parse.py | Dsp to Ds converter implementation |
| bnf/python/ds_bnf/cli.py | Command-line interface implementation |
| bnf/python/tests/init.py | Python test package initialization |
| bnf/python/tests/test_conversion.py | Python test suite |
| bnf/python/README.md | Python-specific documentation |
| bnf/setup.sh | Setup script for generating ANTLR parsers |
| bnf/examples/example.ds | Example file in DS syntax |
| bnf/examples/example.dsp | Example file in Dsp syntax |
| bnf/examples/README.md | Examples documentation |
| bnf/README.md | Main package documentation |
| bnf/IMPLEMENTATION.md | Implementation summary and development notes |
| bnf/.gitignore | Git ignore rules for generated files |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| rule | ||
| : term | ||
| | (term (',' term)*)? '->' term |
There was a problem hiding this comment.
The rule grammar allows (term (',' term)*)? '->' term which permits zero terms before the arrow (e.g., -> conclusion). This creates an ambiguous parse: is -> conclusion a rule with zero premises, or is it a binary expression? This could lead to unexpected parsing behavior.
Consider making at least one premise mandatory when using the arrow syntax, or add explicit disambiguation logic in the parser.
| | (term (',' term)*)? '->' term | |
| | term (',' term)* '->' term |
| SYMBOL | ||
| : ~[ \t\r\n,()]+ |
There was a problem hiding this comment.
The SYMBOL token definition ~[ \t\r\n,()]+ is overly permissive and will match operators like ->, ==, etc. This creates lexer ambiguities. For example, in the input a->b, the lexer could tokenize this as:
- Three tokens:
a,->,b(desired) - One token:
a->b(SYMBOL matches first)
While ANTLR's maximal munch rule and token ordering may resolve some cases, this can still lead to unexpected tokenization issues. Consider excluding operator characters from the SYMBOL definition or making operators explicit keyword tokens.
| SYMBOL | |
| : ~[ \t\r\n,()]+ | |
| // Operators | |
| ARROW : '->' ; | |
| DOUBLECOLON: '::' ; | |
| DOT : '.' ; | |
| LBRACK : '[' ; | |
| RBRACK : ']' ; | |
| LPAREN : '(' ; | |
| RPAREN : ')' ; | |
| COMMA : ',' ; | |
| ASSIGN : '=' ; | |
| PLUS : '+' ; | |
| MINUS : '-' ; | |
| STAR : '*' ; | |
| DIV : '/' ; | |
| MOD : '%' ; | |
| TILDE : '~' ; | |
| BANG : '!' ; | |
| AND : '&' ; | |
| OR : '|' ; | |
| CARET : '^' ; | |
| DOTSTAR : '.*' ; | |
| LSHIFT : '<<' ; | |
| RSHIFT : '>>' ; | |
| LT : '<' ; | |
| GT : '>' ; | |
| LE : '<=' ; | |
| GE : '>=' ; | |
| EQ : '==' ; | |
| NE : '!=' ; | |
| LAND : '&&' ; | |
| LOR : '||' ; | |
| // Identifiers | |
| SYMBOL | |
| : [a-zA-Z_][a-zA-Z0-9_]* |
hzhangxyz
left a comment
There was a problem hiding this comment.
- 先不要加上examples和tests, 这个pr只完成基本的功能
- setup.sh 为什么还存在, 如果需要就把功能移动到js和python自己的build/package系统中, 如果不需要就删除
- tsconfig.json为什么存在, 这里没有ts只有js
…and Python functionality Co-authored-by: hzhangxyz <11623447+hzhangxyz@users.noreply.github.com>
Done! Simplified the package as requested in commit a7267c9:
The package now contains only core conversion functionality without examples or tests. |
| from .parse import parse | ||
|
|
||
| __all__ = ["unparse", "parse"] | ||
| __version__ = "0.1.0" |
There was a problem hiding this comment.
不要这么些, 使用setuptools-scm来做这个, 参考root目录下那个apyds的pyproject.toml
There was a problem hiding this comment.
Done in d857bd4. Removed hardcoded __version__ and added setuptools-scm configuration to pyproject.toml with dynamic versioning.
| @@ -0,0 +1,38 @@ | |||
| """ | |||
There was a problem hiding this comment.
Done in d857bd4. Deleted cli.py and removed CLI scripts from pyproject.toml.
| @@ -0,0 +1,60 @@ | |||
| # Grammar Design Notes | |||
There was a problem hiding this comment.
这里不需要README, 相关的信息直接合并到 bnf/README.md 那个地方.
There was a problem hiding this comment.
Done in d857bd4. Deleted grammars/README.md and merged grammar design notes into bnf/README.md.
| @@ -0,0 +1,10 @@ | |||
| /** | |||
There was a problem hiding this comment.
bnf/src 这个目录不合适, 用 bnf/atsds_bnf 吧
There was a problem hiding this comment.
Done in d857bd4. Renamed bnf/src to bnf/atsds_bnf and updated all references in package.json.
| @@ -0,0 +1,25 @@ | |||
| # Generated files | |||
There was a problem hiding this comment.
Done in d857bd4. Deleted bnf/.gitignore since root directory already handles ignoring.
| @@ -0,0 +1,188 @@ | |||
| # BNF Package Development Summary | |||
There was a problem hiding this comment.
不用这个, 必要的信息合并到 bnf/README.md 中
There was a problem hiding this comment.
Done in d857bd4. Deleted IMPLEMENTATION.md and merged essential information into bnf/README.md.
| Setup script for apyds-bnf package with ANTLR parser generation | ||
| """ | ||
|
|
||
| import os |
There was a problem hiding this comment.
Done in d857bd4. Removed unused os import from setup.py.
| @@ -0,0 +1,29 @@ | |||
| { | |||
There was a problem hiding this comment.
这里有不少 src/ 但是前面说了目录修改成 atsds_bnf , 所以这里需要同步修改.
There was a problem hiding this comment.
Done in d857bd4. Updated all src/ references to atsds_bnf/ in package.json (exports, main, module, files, scripts).
| "module": "src/index.js", | ||
| "files": [ | ||
| "src/**/*.js", | ||
| "grammars/**/*.g4" |
There was a problem hiding this comment.
Done in d857bd4. Removed grammars/**/*.g4 from the files array in package.json.
|
@copilot 我刚刚做了一些检查和改动
我希望你:
|
…cies Co-authored-by: hzhangxyz <11623447+hzhangxyz@users.noreply.github.com>
Done in 65b4e79:
Key changes:
|
fc04d10 to
733e393
Compare
9723641 to
9d00ffa
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 13 changed files in this pull request and generated 9 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ; | ||
|
|
||
| SYMBOL | ||
| : ~[ \t\r\n,()[\]]+ |
There was a problem hiding this comment.
[nitpick] The SYMBOL token definition in both grammars includes [ and ] in the exclusion set, but the Ds grammar doesn't use these characters. While this doesn't cause issues, it's inconsistent with the actual syntax being parsed. The Dsp grammar needs this exclusion because it uses brackets for subscripts, but Ds uses s-expressions only.
| : ~[ \t\r\n,()[\]]+ | |
| : ~[ \t\r\n,()]+ |
| @@ -0,0 +1,2 @@ | |||
| * | |||
| !index.js | |||
There was a problem hiding this comment.
The .gitignore pattern * followed by !index.js will ignore all files except index.js. However, the actual source file is index.mjs (as seen in the diff), not index.js. This means the .gitignore won't preserve the intended file and will instead preserve a non-existent file.
| !index.js | |
| !index.mjs |
| const conclusion = result.pop(); | ||
| return result.join(", ") + " -> " + conclusion; |
There was a problem hiding this comment.
In the JavaScript implementation, when there's only one premise in a rule (line 19-20), the result is formatted as "----\n{result[0]}". However, in the UnparseVisitor.visitRule (line 69-72), when converting back, result.pop() is always called, which assumes there are at least 2 elements. This means a rule with only a conclusion (no premises) would fail. The parse and unparse operations are not symmetric for edge cases.
| const conclusion = result.pop(); | |
| return result.join(", ") + " -> " + conclusion; | |
| if (result.length === 1) { | |
| // Only a conclusion, no premises | |
| return "----\n" + result[0]; | |
| } else { | |
| const conclusion = result.pop(); | |
| return result.join(", ") + " -> " + conclusion; | |
| } |
| return f"({self.visit(ctx.term(0))} {ctx.getChild(1).getText()} {self.visit(ctx.term(1))})" | ||
|
|
||
|
|
||
| def parse(input): |
There was a problem hiding this comment.
[nitpick] The parameter name is input which shadows the built-in Python function input(). While this works, it's considered poor practice. Consider renaming to input_str, text, or source to avoid shadowing built-ins.
| return visitor.visit(tree) | ||
|
|
||
|
|
||
| def unparse(input): |
There was a problem hiding this comment.
[nitpick] The parameter name is input which shadows the built-in Python function input(). While this works, it's considered poor practice. Consider renaming to input_str, text, or source to avoid shadowing built-ins.
|
|
||
| rule | ||
| : term | ||
| | (term (',' term)*)? '->' term |
There was a problem hiding this comment.
The Dsp grammar rule on line 9 allows an empty premise list with the optional (term (',' term)*)? before '->'. This means -> conclusion is valid syntax. However, the ParseVisitor doesn't handle this case - when result is empty after visiting terms, result.pop() on line 22 will fail with an IndexError. The visitor should check for empty premises.
| { | ||
| input: "atsds_bnf/index.mjs", | ||
| output: { | ||
| file: "dist/index.mjs", |
There was a problem hiding this comment.
The output path in rollup.config.mjs is dist/index.mjs, but package.json references dist/index.mjs for all exports. However, the description mentions the exports should be dist/bnf.mjs (as stated in the PR description). This inconsistency should be resolved - either update rollup.config.mjs output to dist/bnf.mjs or update the PR description.
| file: "dist/index.mjs", | |
| file: "dist/bnf.mjs", |
| conclusion = result.pop() | ||
| return ", ".join(result) + " -> " + conclusion |
There was a problem hiding this comment.
The same asymmetry issue exists in the Python implementation. When there's only one term in a rule (lines 18-19), it's formatted as "----\n{result[0]}", but in UnparseVisitor.visitRule (lines 50-53), result.pop() is always called without checking if there are enough elements. This will fail for rules with only a conclusion.
| conclusion = result.pop() | |
| return ", ".join(result) + " -> " + conclusion | |
| if len(result) == 1: | |
| return result[0] | |
| else: | |
| conclusion = result.pop() | |
| return ", ".join(result) + " -> " + conclusion |
| grammars_dir = base_dir | ||
| output_dir = base_dir / "apyds_bnf" | ||
|
|
||
| # Generate parsers for both grammars | ||
| for grammar in ["Ds.g4", "Dsp.g4"]: | ||
| grammar_path = grammars_dir / grammar |
There was a problem hiding this comment.
[nitpick] The grammars_dir variable is set to base_dir (line 23), but this seems incorrect. Based on the file structure, the grammar files (Ds.g4, Dsp.g4) are in the bnf/ directory, not in a separate grammars/ subdirectory. The code should explicitly look for grammars in the base directory, or the variable should be renamed to avoid confusion (e.g., grammar_dir = base_dir).
| grammars_dir = base_dir | |
| output_dir = base_dir / "apyds_bnf" | |
| # Generate parsers for both grammars | |
| for grammar in ["Ds.g4", "Dsp.g4"]: | |
| grammar_path = grammars_dir / grammar | |
| grammar_dir = base_dir | |
| output_dir = base_dir / "apyds_bnf" | |
| # Generate parsers for both grammars | |
| for grammar in ["Ds.g4", "Dsp.g4"]: | |
| grammar_path = grammar_dir / grammar |
BNF Support Package - Final Refactor
Changes Based on User Feedback
apyds_bnf rewritten 1:1 to match atsds_bnf
package.json updated to match root package.json
pyproject.toml cleaned up
setup.py simplified
Added rollup.config.mjs
Final Structure
Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.