Remove `examples` from yaml-schema

[braingram]

Similar to removing style keyword subschemas #256 we would benefit from first having an alternative. The examples are also widely used and an integral part of the pytest-asdf schema testing plugin.

[braingram]

FWIW currently no schemas have examples that aren't at the "top-level" (accessible via "root.examples" in the schema) as tested in asdf-format/asdf#1956

This means the nested examples checking in the yaml schema is unused.

[braingram]

It seems related enough to consider how examples are used.

tldr; drop examples from yaml schema, change their format, handle them differently in the asdf plugin

Examples...:

• provide documentation for schema readers/maintainers about expected (and example) structure(s) that conform to the schema
• are used for effectively schema testing as part of the pytest-asdf plugin where each example is validated (against the provided schema via the tags mentioned in the example)

Some issues are:

• it is only possible to have valid tests whereas having known invalid tests would also be useful for schema testing
• the current format (a 2 or 3 item list) is not intuitive and does not match the current schema which only checks that the first item is a string and the second is either a string or an object (which appears to be unsupported and unused) and doesn't check any additional items (including the actual example text in most cases)
• the use of tags in the examples makes them subject to unhelpful version constraints (which led to the introduction of the asdf-standard requirement
• the inclusion of examples in the schema can significantly bloat the schema (which when printed as part of a traceback creates an imposing wall of text)

Let's take one example from the time schema:

asdf-standard/resources/schemas/stsci.edu/asdf/time/time-1.3.0.yaml

Lines 69 to 77 in 2764c03

	-
	- Example of an array of times
	- asdf-standard-1.6.0
	- |
	!time/time-1.3.0
	value: !core/ndarray-1.1.0
	data: [2000, 2001]
	datatype: float64
	format: jyear

When pytest-asdf checks this example it needs:

• to perform mapping between the !time/time-1.3.0 tag and the schema (the mapping is defined in the manifest making the example not self-sufficient)
• to requires the existence of a !core/ndarray-1.1.0 tag (which is only in some versions of the core-schemas/standard) which leads to a need for a asdf-standard requirement
• to assume the example is valid (it is here since it's a working example)

If we instead rewrite this example to:

   - description: "Example of an array of times"
   - example: |
       value: !core/ndarray-1.1.0 
         data: [2000, 2001] 
         datatype: float64 
       format: jyear
   - valid: true  # assumed value

we fix a few of these issues:

• removal of !time/time-1.3.0 removes the need for the manifest mapping tag to schema
• ndarray issue is unsolved (more on that below)
• this example could be marked as invalid by setting valid: false

How much of an issue !core/ndarray-1.1.0 creates depends on how the example is handled during testing. Time uses a $ref to the ndarray schema so we could just drop the ndarray tag (although the example would be less representative as real files would contain the tag). However if the time schema used a tag validator the tag would be checked (but only via a string match). Given this behavior ($ref doesn't use mapping, tag does a string match) the removal of the !time/time-1.3.0 to schema map effectively decouples this example from the standard (removing any need for an asdf schema standard requirement). Essentially the validation for each example becomes:

asdf.schema.validate(tagged_tree_from_example, schema=asdf.schema.load_schema(schema_uri), reading=True)

which performs validation of the already tagged tree (no need to serialize or deserialize the example) against the schema based on the uri (no need for the tag mapping) and appropriately handles mapping of all $ref and tag schema links.

The remaining issue from above is the schema bloat. I am not going to discuss this here as this comment is already too long and I'll add a tldr; at the top.

[braingram]

To expand on this with a proposed roadmap let's:

• add an examples schema to asdf-standard that can be applied to a file to determine if it contains suitable examples (I have a prototype of this I will push up shortly that works on both existing schemas with examples at the top-level and on non-schema YAML files in case we want to put examples in a separate file)
• update the pytest-asdf plugin to use the examples schema to check if a schema contains examples (and to validate non-schema examples files), extract these examples and run them
• make a new yaml-schema dropping examples (although we may want to wait until we can also drop the style keywords)

This has a few benefits:

• allows examples to be in a separate file
• improves the example format (as described above)
• simplifies the yaml-schema (which would also allow downstream users more flexibility in their schemas)

[perrygreenfield]

I appreciate the implication of comment bloat when addressing schema bloat ;-).

To be honest, I've always had mixed feelings about the inclusion of examples in the schemas, if only for distracting from the core structure of the schema itself besides making changes to that requiring really unnecessary changes to schema versions (it does require that, doesn' it?). So I'm well in favor of your proposal. Is the idea for the schema to reference a uri for examples? If so, will it dispense with specifying a version of the examples (presume the latest? or how will that be handled?).

[braingram]

That's a good point about the version bumping and examples and probably reason enough to keep them outside of the schemas. I think this also precludes referencing a uri with examples (which would also be versioned etc).

One directory structure to encourage would be:

/resources/... # with schemas without examples
/examples/...  # with examples/test files

For example let's say there is:

/resources/schemas/foo-1.0.0.yaml
/resources/schemas/foo-1.1.0.yaml
/examples/schemas/foo-1.0.0.yaml
/examples/schemas/foo-1.1.0.yaml

In this case each schema has a separate example file (which I think makes sense since a different schema version likely needs different examples). With /examples/schemas/foo-1.0.0.yaml containing something like:

schema_id: asdf://.../foo-1.0.0.yaml
examples:
   - description: "some test"
   - example: |
       foo: a
   - valid: true  # assumed value

Where schema_id will be used to look up the schema that will be applied to each example. To make things easier to maintain we might make this also accept a list of URIs (since some schema updates might not impact the examples).

[braingram]

Given the lack of agreement between the current metaschema and the current examples structure and that jsonschema allows extra keywords it seems reasonable to remove examples from the metaschema independent of any external example file format (as long as the new metaschema, like the parent draft 4 metaschema, doesn't prevent the examples in the schema).