Possibly inconsistent `$schema` for optional `format` tests ?

[zx80]

While implementing $vocabulary, I noticed that the $schema of optional format tests for draft 2020 point to https://json-schema.org/draft/2020-12/schema, which contains a $vocabulary section with:

{
    "https://json-schema.org/draft/2020-12/vocab/format-annotation": true
}

Which seems to explicitely tell that format are annotations rather than assertions, hence any conformant validator should ignore these, thus all tests values should yield true.

It seems that it would be more logical to use some https://json-schema.org/draft/2020-12/schema-with-format (or similar) $schema with:

{
    "https://json-schema.org/draft/2020-12/vocab/format-assertion": true
}

So that validators are at least expected to consider formats, thus reject some of the test values?
There is a similar issue with draft 2019.

Am I missing something?

[jviotti]

Hey @zx80 ,

This is intentional. Implementations are required to implement format as an annotation to NOT validate (as per https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft2020-12/format.json), while implementations MAY optional implement format-assertion for actual validation (as per https://github.com/json-schema-org/JSON-Schema-Test-Suite/tree/main/tests/draft2020-12/optional/format)

[jviotti]

Ah, wait, I just understood your overall comment: the optional ones have the wrong $schema. That is a good point. They should be pointing at https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/remotes/draft2020-12/format-assertion-true.json. Good catch 👍🏻

I guess this is good proof of the fact almost nobody really implements format-assertion. Most will only stick with the annotation variant, as extremely few users will create their own dialect to enable format-assertion.

[zx80]

Ah, wait, I just understood your overall comment: the optional ones have the wrong $schema.

Indeed.

Most will only stick with the annotation variant, as extremely few users will create their own dialect to enable format-assertion.

ISTM that people should not have to create their own dialect to use an official feature… there should be an official schema available that they could point to.

[jviotti]

Feel free to send a PR yourself fixing the $schema's if you want!

ISTM that people should not have to create their own dialect to use an official feature… there should be an official schema available that they could point to.

Yeah, I remember discussions around it and hopefully the entire format situation gets improved on the next spec release. The main problem is that even if we do support such a dialect, most implementations don't support it or support it badly given the lack of coverage on format assertion in the tests.

I'm actually trying to fix the latter in a GSoC project this year: json-schema-org/community#965

In my experience, most people atm prefer NOT doing format-assertion given inconsistencies in results, or in coverage, which in a way led to the self-fulfilling prophecy I guess.

[zx80]

Feel free to send a PR yourself fixing the $schema's if you want!

Are there official schema URLs with formats as assertions? If not could they be added? It seems to me that it is the real starting point, pointing to a custom localhost test schema is kind of missing the point from my point of view. Another improvement would be to have a way to enable bowtie to also report on formats, which does not seem to be the case for now.

[...] most implementations don't support it or support it badly given the lack of coverage on format assertion in the tests.

It does not seem that bad, with about 30 test values per formats, even if it could be improved, obviously. I think that for most types the standard expectations are clear enough. Also, some formats are more relevant than others: people are more likely to have a date or an email that they really want to be correct somewhere in their data than a json pointer or a duration.

In my experience, most people atm prefer NOT doing format-assertion given inconsistencies in results, or in coverage, which in a way led to the self-fulfilling prophecy I guess.

Hmmm… From a data validation point of view, limiting checks to the structural side and ignoring leaf values seems like a half-baked approach which would catch half of the underlying type issues.

[jdesrosiers]

The dialect of the tests are correct as is. It is intended for them to use standard dialect URI.

Using the format-assertion vocabulary isn't the only way the spec allows enabling format assertion. Implementations can provide a configuration option to enable assertion. The rule is just that they have to default to being disabled if the dialect uses format-annotation.

The README talks about this case ...

Test cases found within special subdirectories may require additional configuration to run. In particular, when running tests within the optional/format subdirectory, test runners should configure implementations to enable format validation, where the implementation supports it.

I think the tests relying on configuration instead of vocabularies makes them applicable to more implementations because not all support vocabularies.

[zx80]

Hello,

Thanks for the explanation.

I agree that an implementation can find another way to enable a feature than doing it per spec.

However, that does not make it a very good reason for an official test suite case not to rely on the spec way of doing thing: It seems strange to require some --please-ignore-spec setting to run official tests.

Moreover, it seems that there is no instance of enabling format assertions with a vocabulary in the test suite, which means that it offers no coverage for the feature. This is no encouragement for implementing it.

So I still think that there should exist an official schema URL to enable assertions and that it should be used by some, if not all, test cases about formats.

[jdesrosiers]

No, you misunderstand. Enabling format validation by configuration is part of the spec. We are not asking people to do something against the spec. Not only is it part of the spec, it's what most implementations do and what most users use. I haven't heard of a single case of anyone actually using vocabularies to enable format validation. It was an idea that never worked out and no one ever used. I'm going to go ahead and close this because the tests are the way they were intended to be and I don't think it needs to change.

[jviotti]

@jdesrosiers Is it too bad to still change the $schema though? See it this way:

• For implementations that switch behaviour based on the vocabularies, like me and @zx80 (proof there is more than 0?), we get exactly what we expect
• For implementations that don't understand $vocabulary, their $schema just points to a meta-schema that still has $schema set to the official 2020-12. They ignore $vocabulary, rely on their configured setting, and it also works fine for them?

It seems we don't truly lose anything. And I still disagree that nobody uses vocabulary to enable format validation. I don't think vocabularies are even the problem, just that we don't release a dialect with the other vocabulary, as @zx80 pointed out.

[zx80]

No, you misunderstand. Enabling format validation by configuration is part of the spec.

I'm sorry that I misuderstood.

After re-reading the relevant part of the spec, there is indeed a requirement in Section 7.2.1 (format annotation vocabulary) for having options to enable or disable format assertions under format annotation for implementations which somehow want to, which is how I interpret Implementations MAY still treat "format" as an assertion in addition to an annotation and attempt to validate the value's conformance to the specified semantics. […].

I understand this paragraph as a guard against such implementation: if you want to do format assertions despite that it is declared as an annotation, you are allowed to do it, but you must provide options to control that. There is a logical condition for the option requirement.

However, I'm not voluntarily in the case of wanting to implement assertions when annotations are required: I simply want the implementation to do by default what is required, when it is required, no more and no less. I've been trying to implement $vocabulary including the OPTIONAL (Section 7.1) format assertion, which seems to be the only place where it matters, hence I may be the only person ever to actually try to run the format tests.

In this context, I seems logical to me that the optional/format directories of the test suite would provide tests to cover Section 7.2.2 of the specification, i.e. tests where format assertion is true, hence this issue.

Moreover, I agree with @jviotti: I do not see a significant downside to providing a meta schema with format assertions, which are part of the spec, nor with the official test suite to actually use it as expected.

[jdesrosiers]

I'll reopen this because there's clear more discussion needed.

For implementations that don't understand $vocabulary, their $schema just points to a meta-schema that still has $schema set to the official 2020-12. They ignore $vocabulary, rely on their configured setting, and it also works fine for them?

Checking the meta-schema to determine the dialect is a vocabulary system behavior. If the implementation doesn't support the vocabulary system, they wouldn't do that. There are three behaviors I've seen.

The first is that the implementation ignores $schema for the purpose of dialect determination and interprets it according to whatever dialect it's configured to use. An example is ajv. For those implementations, changing the $schema doesn't hurt because they ignore $schema anyway.

The second behavior I've seen is that the implementation checks $schema to determine which dialect to use for the schema. If they don't recognize the dialect, they refuse to process the schema. An example is SchemaSafe. If we change $schema to something that isn't an official dialect identifier, it would break the test harness for implementations like this.

The third is full support of vocabularies that reads the meta-schema from $schema and reads $vocabulary to determine how to process the schema. These implementations can handle the $schema change, but they are fairly rare. I don't think there are many out there other than implementations by TSC members.

And I still disagree that nobody uses vocabulary to enable format validation. I don't think vocabularies are even the problem, just that we don't release a dialect with the other vocabulary, as @zx80 pointed out.

I think those are two different things. If we had an official dialect that enables format validation and implementations actually supported it, then I think people would use it. But we don't have such a dialect and it's not very common that it's supported even if someone went to the trouble of constructing their own dialect. Therefore, I'm quite sure it's not common. I'm sure it's not that "nobody" is using vocabularies to enable validation, but I'm sure it's not common. People use what their implementation supports and for most, that means a configuration option. I don't think that simply providing a format validation dialect is going to change much because I don't think most implementations support vocabularies anyway.

hence I may be the only person ever to actually try to run the format tests.

This is very much an incorrect assumption. I expect these tests are used by almost all JSON Schema implementations. Almost all JSON Schema implementations support format validation to some degree and almost all implementations use the test suite. There may be exceptions, but they're very rare.

The format tests weren't created with the aspiration that they might be used some day. Until recently, they were contributed entirely by implementers to fill in gaps they found when implementing format validation and verified by all the other implementations that already support format validation. The recent GSoC candidates that have been around this year is the first time I can ever remember where people contributed to those tests who were not actively implementing format validation.

The format validation tests are very very widely used. Many have used them before you, including me.

However, I'm not voluntarily in the case of wanting to implement assertions when annotations are required: I simply want the implementation to do by default what is required, when it is required, no more and no less.

If you choose to implement format validation without a configuration-based options, you'd be the only implementation I've ever heard of to make that choice.

In this context, I seems logical to me that the optional/format directories of the test suite would provide tests to cover Section 7.2.2 of the specification, i.e. tests where format assertion is true, hence this issue.

These test are the way they are because that's how they've worked long before the vocabulary system existed. I think it's important for them to remain so people can test them with a configuration option.

However, we can talk about how we can provide a way for implementations that only support vocabulary-based activation to run the same tests. I think the unified test suite project is a good opportunity to address this. One of the things the that project aims to do is to remove the $schema keyword for most tests and use a compatibility flag to indicate which versions the test is compatible with. We could make use of that flag to indicate that the test are expected to work with 2020-12 and 2020-12+format-assertion.

[zx80]

I'll reopen this because there's clear more discussion needed.

Thanks!

Checking the meta-schema to determine the dialect is a vocabulary system behavior. If the implementation doesn't support the vocabulary system, they wouldn't do that. There are three behaviors I've seen.

Ok. You are outlining what actual implementations do which is slightly out of sinc with the standard.

The purpose of raising this issue is that I assume that, for the sake of consistency, the logic of an official test suite would be to test the standard expectations, so that implementations that take it as a reference would be driven to comply.

The second behavior I've seen is that the implementation checks $schema to determine which dialect to use for the schema. If they don't recognize the dialect, they refuse to process the schema. An example is SchemaSafe. If we change $schema to something that isn't an official dialect identifier, it would break the test harness for implementations like this.

Ok this is a possible drawback. It seem easily fixable by these implementation: recognize the added schema dialect. Moreover it would only occur whiie running the format test suite.

If we had an official dialect that enables format validation and implementations actually supported it, then I think people would use it.

Ok, this is part of the problem: why specifying the "format" feature without providing the corresponding official meta-schema? What prevents to add one?

I don't think that simply providing a format validation dialect is going to change much because I don't think most implementations support vocabularies anyway.

Currently there is one vocabulary test in the official test suite, so implementations which claim 0 defect must support vocabulary?!

Implementations are much more likely to support the format assertion vocabulary it if there is one test which triggers it.

hence I may be the only person ever to actually try to run the format tests.

This is very much an incorrect assumption [...]

Ok, that's very good! I'm biased because blaze which I use as a reference for performance test does not support formats (yet?) and the bowtie site does not seem to report format tests either.

However, I'm not voluntarily in the case of wanting to implement assertions when annotations are required: I simply want the implementation to do by default what is required, when it is required, no more and no less.

If you choose to implement format validation without a configuration-based options, you'd be the only implementation I've ever heard of to make that choice.

I also have an option. My point is the consistency of the test suite wrt the spec.

In this context, I seems logical to me that the optional/format directories of the test suite would provide tests to cover Section 7.2.2 of the specification, i.e. tests where format assertion is true, hence this issue.

These test are the way they are because that's how they've worked long before the vocabulary system existed. I think it's important for them to remain so people can test them with a configuration option.

Ok, that is an explanation.

This argument seems fine for up to draft7, but unclear for the last two versions which have this system, so the added tests should/could have been updated as well.

I would guess that the draft7 test suite has been copied and updated for the newer versions, but the format assertion/vocabulary implication has been overlooked in the process.

However, we can talk about how we can provide a way for implementations that only support vocabulary-based activation to run the same tests. I think the unified test suite project is a good opportunity to address this. One of the things the that project aims to do is to remove the $schema keyword for most tests and use a compatibility flag to indicate which versions the test is compatible with. We could make use of that flag to indicate that the test are expected to work with 2020-12 and 2020-12+format-assertion.

Yep, that would be a relevant way to adapt the tests cleanly to different contexts.

[jdesrosiers]

The purpose of raising this issue is that I assume that, for the sake of consistency, the logic of an official test suite would be to test the standard expectations, so that implementations that take it as a reference would be driven to comply.

That's true, but config-based activation is just as much part of the spec as vocabulary-based activation. Changing $schema for these tests would fix the gap in covering vocabulary-based activation, but it would create a gap that was previously covered in config-based activation. We should definitely find a way to cover vocabulary-based activation, but not at the expense of config-based coverage.

Implementations are much more likely to support the format assertion vocabulary it if there is one test which triggers it.

We have that in optional/format-assertion.json.

why specifying the "format" feature without providing the corresponding official meta-schema?

I don't remember if there was a reason it wasn't done that way. I think the expectation was that people would create their own dialect, but people found that too complicated and implementers rarely supported it, so it's ultimately rarely done in practice.

What prevents to add one?

We could certainly release one. I wouldn't be opposed. But, it wouldn't be part of the spec because we can't change an already released spec. It would just be a community recommendation for implementations to support it.

The next release will be a little different because the vocabulary system is pulled out into proposed extension and no longer part of the main spec. We can define the new dialect URI in the proposal, but it's unlikely that proposal is going to be adopted. It's more likely going to be replaced by some other mechanism.