Schema version for serialization

[david-pl]

This came out of a discussion with @jon-wurtz and @yzcj105

Should we ever be in a situation where we need to change the serialization format, it may cause kernels that were serialized with the old version to longer be compatible. In order to track this we could add a schema version, so even when deserializing a JSON file we know which version of the (de-)serializer should be used. A further addition to this could be that, should we ever be in a situation where we break the serialization format, we can also automatically migrate old kernels.

This is relevant for data reproducibility etc. since everything going through an API will be serialized.

[Roger-luo]

Yes we should definitely version the JSON file. I believe currently it was somehow versioned in our API or it's not?

cc: @kaihsin, he was one of the author for the serialization feature

[kaihsin]

I don't think its versioned in the serialization frame work. we probably need some header to describe the metadata, so other downstream compilers can put version information there

[david-pl]

we probably need some header to describe the metadata, so other downstream compilers can put version information there

So you're suggesting to just have the ability to pass in a metadata dict that gets serialized into a metadata field in the JSON?
I'd suggest to have a fixed set of fields for that to enforce consistency throughout the pipeline.

This raises another question though: should we start versioning dialects? So you can have e.g. {'language': 'squin', 'version': '1.0'}? Or, more generally, how do we consistently define the version at which a given kernel was serialized?

[kaihsin]

we probably need some header to describe the metadata, so other downstream compilers can put version information there

So you're suggesting to just have the ability to pass in a metadata dict that gets serialized into a metadata field in the JSON? I'd suggest to have a fixed set of fields for that to enforce consistency throughout the pipeline.

This raises another question though: should we start versioning dialects? So you can have e.g. {'language': 'squin', 'version': '1.0'}? Or, more generally, how do we consistently define the version at which a given kernel was serialized?

I think versioning dialect make sense?

[david-pl]

We should tackle this sooner rather than later. There's a few open questions though.

1. Where do we add the version? Should this be stored on the dialect level, or just a keyword when serializing a kernel? In either case, let's make this as explicit and fixed as possible. I'd suggest we make this a language_version which only accepts semantic versions.
2. Which version do we use? While it will quite often be the case that a new version of a package is released without actually breaking the dialect, I feel like it would be good for our sanity to just use the package version tags as the language version, so we don't have to track a separate version for each dialect in the ecosystem.
3. How do we deal with migrations? Specifically, should there be an automatic migration system that reads in a serialized kernel and tries to rewrite it to a new one? This is tricky because: we'd need to keep around an old version of a dialect to rewrite from. Also, we'd somehow have to manage these migrations. That's tightly coupled to the previous point: depending on what we use for versioning the dialects, we need to create corresponding migration logic for each change in the dialect (which is not necessarily coupled to the package version). While this would be great for users, I'm not sure we should go for automatic migrations at all since it is quite a big effort. Also, hopefully languages should not be subject to too many changes in the long run.