GSoC 2026 - Unify the test suite

[jdesrosiers]

Unify the test suite

Brief Description

In the official test suite, we don't manage the tests for different versions of JSON Schema very well. Each time there's a new release, we make a copy of the previous version and make updates to that copy. Because of this, we have multiple copies of most of the tests and every time we need to add a new test, we usually have to add a copy in several places.

The annotations test suite takes a different approach. Tests have a compatibility property indicting which versions of JSON Schema the test applies to. That allows test runners to filter the tests to which one's apply to them.

The goal of this project is to collapse the per version validation tests into one test suite using a similar approach to the annotations test suite. This will allow us to manage tests in a single suite with no more copies of identical tests.

Expected Outcomes

• A single test suite for all versions of JSON Schema
• No more copies of tests
• Tests can be easily filtered by test runners to the ones that apply to them
• Update any scripts/action in the test suite repo to use the new suite
• Update bowtie to use the new suite
• Update bowtie implementation test runners to use the new suite

Skills Required

• Good attention to detail
• Ability to work in unfamiliar programming languages. (Updating bowtie implementation test runners.)

Mentors

@jdesrosiers and @Julian as co-mentor

Expected Difficulty

small

Expected Time Commitment

90 hours

[AnujAga2005]

Hi @jdesrosiers, this project idea looks very exciting!

Could you please expand a bit on the intended technical approach? I’d love to understand the workflow and requirements better to ensure it aligns with my current skillset.

[Sourya07]

Hi @jdesrosiers,
This project looks feasible for me, and I’m genuinely excited about the idea of unifying the test suite. The focus on reducing duplication and improving long-term maintainability.

I’d love to work on a proposal for this and start exploring the test suite structure and related scripts. Please let me know if there’s anything specific you’d recommend I look into first

[Surajiitmjnu]

Hi @jdesrosiers
I am highly motivated to work on the "Unify the Test Suite" project because it addresses a real maintainability challenge in the JSON Schema ecosystem. I have a strong interest in developer tooling and system design, and I am comfortable working with unfamiliar codebases. I am confident that I can successfully design and implement a unified, compatibility-based test suite and update Bowtie and its runners accordingly.

[jviotti]

@jdesrosiers I love this idea. I wonder if we could also consolidate the https://github.com/python-jsonschema/referencing-suite in the same way. I remember that suite being instrumental for the foundations of Blaze back then.

[AnirudhJindal]

@jdesrosiers hi,
This might be a dumb question, but I want to make sure I’m understanding the intended direction correctly

When an existing test changes over time (for example, semantics being updated, new keywords added, or behavior clarified in a newer draft), is the expectation to:

• keep a single test definition and adjust its compatibility / version constraints, or

• introduce a new test case for the updated behaviour while retaining the old one for earlier drafts?

Related to that, if a test applies to multiple versions but requires a small tweak for one specific version, should that difference be expressed within the same file (e.g. via version-specific annotations), or is it preferable to split it into separate tests?

[jdesrosiers]

I wonder if we could also consolidate the https://github.com/python-jsonschema/referencing-suite in the same way.

I'm not terribly familiar with that suite, but I don't see why it wouldn't work there as well.

[jviotti]

It is pretty simple: given a set of schemas (which might be interconnected) + an input URI (given an optional base), you need to resolve it to the given target. This is useful for making sure things like $id, $anchor, $ref, etc properly work in non-trivial cases. i.e. the whole identification and referencing aspect of JSON Schema

[Honyii]

@jdesrosiers kindly add @Julian as a co-mentor for this project idea.

[shaydaj]

Hi @jdesrosiers and @Julian - I’m Shayda

I’m a full-stack engineer with a strong testing and CI background, and I’m really interested in working on #994. I like the idea of reducing duplication and making the suite easier to maintain long-term - it feels like the kind of structural improvement that pays off across the whole ecosystem.

My current thinking is:

• Move to a single unified validation test suite as the source of truth.
• Introduce a compatibility (or similar) field per test, inspired by the annotations suite, so runners can filter by draft.
• When behaviour genuinely changes between drafts, prefer splitting into separate test cases (same intent, different expected result) rather than embedding version-specific logic inside one test - to keep things readable and simpler for runners.

Just a couple of clarifiers so I align with your intent:

1. For tests where semantics changed across drafts, would you prefer a single test with draft constraints when the expected result is the same, and separate tests when it differs?
2. Do you have a preferred shape in mind for the compatibility metadata (explicit list of drafts vs something more expressive)?

If that direction makes sense, I’d like to start with a small “thin slice” PR this week:

• define the compatibility field structure + documentation
• migrate one small duplicated subset of tests end-to-end
• prove the filtering works with existing scripts

Also — I tried joining Slack via the website but the invite link seems to be expired. Is there a specific Slack channel for GSoC discussions that I should join?

Thanks!

[jdesrosiers]

How about you start by familiarizing yourself with the test suite. The answers to your questions should be apparent when you see how it works.

If that direction makes sense, I’d like to start with a small “thin slice” PR this week:

Don't do that. This has been selected as a GSoC project. You'll have to submit your proposal like everyone else and if you're selected, then we'll talk about the execution plan and you can make PRs.

[Surajiitmjnu]

Hi @jdesrosiers sir

Thanks for the clarification! I’ll start by going through the test suite and understanding the workflow first.
I’ll prepare my proposal accordingly and reach out if I have specific questions.