Build spy fmt formatter

[kanin-kearpimy]

Issue: #457

Description

Hello, this PR introduce SPy formatter via spy fmt command. SPy actually extends Python syntax with extra language grammar. Technically, we can use Python formatter. However, we need to handle SPy-specific const and var declaration.

spy format is calling:

1. preprocess .spy code to `python.
2. in preprocess, we mark SPy grammar (e.g. var x, const y with MIDDLE DOT like var·x and const·y.
3. Ruff formatting
4. reintroduce SPy grammar by converting var·x and const·y to var x and const y

Note

This PR cover only single file formatter. Adding multiple files or directory are not that hard. To concentrate only on SPy formatter, I propose we leave those features for another issue (I'll create it later). Such issue is also good for first time contributor.

DEPRECATED DO NOT READ

spy fmt is calling:

1. Read .spy file content and tokenize.
2. Perform scoping tracker SPy-specific declaration const and var << (Please see below for more detail)
3. Cleaning SPy-specific const and var out.
4. Parse (3) to Python tokens.
5. perform ruff formatter.
6. Reintroduce SPy-specific declaration back.
7. change file content.

SPy-specific declaration: Scoping Tracker

We scope source code by function, class, and some control-flow branch such as if, else, while, for. Tracker push/pop those scope into stack. These scopes are not affected by ruff formatter. So, we can track nth-variable by each scope.

In picture above, we have two scope: module_def-main and module_def-main_if-1. So, dictionary tracker has

• module_def-main__x-1
• module_def-main_if-1__y-1

If we found SPy-specific like const, var, we marked them in dictionary. When we perform step (4) to reintroduce SPy-specific, we perform scope tracker against step (1) dictionary.

Please note we also track occurrence of variables and keywords (if, else, while, for) against its scope.

[Copilot]

Pull request overview

Adds a new spy fmt CLI subcommand intended to format .spy sources by preprocessing SPy-specific syntax (var/const) into Python-compatible code, running ruff format, and reinserting SPy-specific tokens afterward.

Changes:

• Introduces spy fmt command wiring and a new SPyFormatter implementation that shells out to ruff format.
• Extends magic_py_parse.py with token-tracking helpers to reinsert var/const after formatting.
• Adds tests for token tracking/reinsertion and a CLI-level formatting test.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
spy/tests/test_fmt.py	New tests for SPy-specific token tracking and reinsertion logic.
spy/tests/test_cli.py	Adds an integration test for spy fmt formatting a file.
spy/magic_py_parse.py	Adds scope/occurrence-based tracking + reinsertion of var/const tokens.
spy/cli/commands/fmt.py	New fmt subcommand implementation.
spy/cli/cli.py	Registers the new fmt subcommand.
spy/analyze/fmt.py	Implements Ruff-based formatting and SPy token reinsertion pipeline.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[JeffersGlass]

HI @kanin-kearpimy! This is really cool! A few little comments dropped in.

My biggest though is that I think this feature should be called format, not fmt, and just have fmt be an alias in the CLI. I think that will make it more clear across the code what the intent is. So, fmt.py -> format.py, test_fmt.py -> test_format.py, etc.

[antocuni]

@kanin-kearpimy sorry for taking so long to review this, the last month has been crazy.
So, first of all: congratulations for having invented this way to solve the problem. It's very clever and I didn't think about it.

That said, I think it's too much clever and I am a bit worried about the extra complexity added to magic_py_parse.

While reviewing this, I got a different idea which would make spy fmt much easier to implement. I prototyped this idea in #540.

Basically, instead of "stripping" var/const and keeping track of varkind_locs in a separate table, we can replace var x with a single identifier var·x, using the special char MIDDLE DOT.

If we have that, spy fmt would become as easy as:

1. use magic_py_parse to produce the "pythonized" source
2. use a regex to replace var·····x with var·x
3. run ruff fmt on it
4. use a regex to replace var·x back to var x
   The nice part is that len('var x') == len('var·x'), so ruff fmt should automatically do the right thing w.r.t. e.g. long lines.

What do you think?

[kanin-kearpimy]

Hi, I refactored code to use new preprocess.

@antocuni

[JeffersGlass]

This might be a nit, but I would love to see the filenames and function names all changed from fmt to format. (fmt.py -> format.py, test_fmt.py -> test_format.py, etc.). There's nothing really gained by the abbreviation here and it would be nice to be consistent with the rest of the command filenames and functions.

Other than that, LGTM!

[antocuni]

thanks @kanin-kearpimy .
This is much easier and simpler than the previous approach, good job :).

I left some comments: many are style issues and nitpicks, but two are real:

1. spy format should operate on filenames, not modnames
2. it should raise SPyError for messages which are meant to be shown to the user, not RuntimeError.

[antocuni]

why do you need cwd=repo_root here?

[kanin-kearpimy]

It's for config in pyproject.toml. However, I adjusted it a bit to explicitly adding config file on command.

            cmd = ["ruff", "format", tmp_path]
            if os.path.exists(config_path):
                cmd.extend(["--config", config_path])
            result = subprocess.run(
                cmd,
                check=True,
                capture_output=True,
                text=True,
            )

[antocuni]

it would be nice to have support for

1. directoris, e.g. spy format /tmp/somedir)
2. multiple files, e.g. spy format /tmp/*.spy)

But I think it's fine to do this in a follow-up PR :)

[kanin-kearpimy]

Of course. Let me share new PR with you soon.

[kanin-kearpimy]

it seems that this improvement has been lost during the refactoring?
Re-opening it.

What do you mean here sorry? @antocuni

[antocuni]

it seems that this improvement has been lost during the refactoring?
Re-opening it.

What do you mean here sorry? @antocuni

the original comment asked to return [str, str], it was marked as resolved but it still returns str.

[paugier]

Side remark: it would be very convenient if spy format could format result of spy redshift. I'm wondering if a trick like var·x could be used for that. Redshift output contains names like "_print::println[str]", which could be turned into equivalent with other special uncommon characters. I will test this idea with a dedicated Python script.

[antocuni]

Side remark: it would be very convenient if spy format could format result of spy redshift. I'm wondering if a trick like var·x could be used for that. Redshift output contains names like "_print::println[str]", which could be turned into equivalent with other special uncommon characters. I will test this idea with a dedicated Python script.

that's an interesting idea.

Not 100% it would make the code more readable though, because redshifted code tends to have very long identifiers. Maybe we can make it working by using a longer line length than usual?

[github-actions]

Documentation Preview
Preview documentation for PR #496.
Last Updated: 2026-06-11 15:01 UTC

[kanin-kearpimy]

Hello @antocuni ,

most of comments has been resolved. For multiple files / dictionary, New PR is coming.