Problem
Current snapshot YAML files are created solely by using the FreeCAD Python API. It uses custom logic to identify properties to store in the snapshot, and has a specialized way to represent those properties. This inherently creates some limitations:
- The specialized property representations need to be be changed to include more details (example: Sketcher objects don't include all sub-params)
- Additional properties need to be added to ensure changes are captures (example: Geometry property on Sketcher objects)
- Git commits that do not have snapshots stored do not show any diffs (and it's not currently possible to make snapshots for old commits)
The first two points above are problematic because changing the underlying snapshot structure would produce a "change" in the following snapshots. E.g. adding a Geometry property to new snapshots, would cause a "change" to be registered between old and new snapshot formats (an added property.)
The decision to design it this way was made because the Document.xml is missing a couple of details needed for good UX, discussed below. But a better solution can be designed.
Solution
Ditch the specialized snapshots altogether and use FCStd's underlying Document.xml file (and perhaps GuiDocument.xml if including view parameters) as the source of truth of the document. It includes every value that could possibly cause a change in the document, so we eliminate the possibility of having to change the snapshot to better represent a document, making it the best long-term solution.
It also creates the ability to perform document diffs without having a snapshot created at all -- it would simply be more limited than the full solution, mirroring the functionality in fcinfo, but with a nice UI.
The Document.xml misses two details that are used in the current workbench: hierarchical object tree representation, and property groups. The python API must still be used to get these details.
Snapshot Format
Since the Document.xml and GuiDocument.xml will be used directly, the only additional info needed to store is the property grouping and tree hierarchy, for UI viewing purposes only (not diffing). We can call this a "Extension Snapshot" or similar. It can be stored as json as it's the most efficient at serialization and deserialization.
Flow: Displaying a Tree and Diff
Option 1, no extended snapshot:
- Unzip all fcstd files in the given commit and previous commit
- Parse the xml documents
- Create an in-memory python representation of each object
- Diff the properties of each object
- Display a flat object structure in the GUI (similar to fcinfo output)
Unzipping all files may seem like a lot of overhead, but A) a benchmark showed that the performance is better than the current snapshot YAML deserialization process, so it will actually be a speedup, and B) unzipping and parsing can potentially be run in parallel if further optimization is needed.
Option 2, with extension snapshot:
If both extension snapshots, old and new, are available, the same operation as option 1 can be performed, except it will be displayed in a tree structure, and properties will be grouped nicely.
Migration / Snapshots for Old Commits
Old snapshot files will not work with the new solution. Since the FCStd files contain everything but the tree structure and groups, and we want to be able to support creating extension snapshots for older commits anyway, it makes sense to simply add that functionality now instead of migrating old snapshots. See issue #10 for details.
This will, unfortunately, make existing snapshots that have been committed obsolete, and will require users to recreate snapshots. But this long term solution is more robust and better anyway.
Storage Location
Since it will be possible to backfill snapshots at any time, we should no longer require committing snapshots together with FCStd files. Start storing them in a separate directory that should be gitignored. See issue #5
Problem
Current snapshot YAML files are created solely by using the FreeCAD Python API. It uses custom logic to identify properties to store in the snapshot, and has a specialized way to represent those properties. This inherently creates some limitations:
The first two points above are problematic because changing the underlying snapshot structure would produce a "change" in the following snapshots. E.g. adding a Geometry property to new snapshots, would cause a "change" to be registered between old and new snapshot formats (an added property.)
The decision to design it this way was made because the Document.xml is missing a couple of details needed for good UX, discussed below. But a better solution can be designed.
Solution
Ditch the specialized snapshots altogether and use FCStd's underlying
Document.xmlfile (and perhapsGuiDocument.xmlif including view parameters) as the source of truth of the document. It includes every value that could possibly cause a change in the document, so we eliminate the possibility of having to change the snapshot to better represent a document, making it the best long-term solution.It also creates the ability to perform document diffs without having a snapshot created at all -- it would simply be more limited than the full solution, mirroring the functionality in fcinfo, but with a nice UI.
The Document.xml misses two details that are used in the current workbench: hierarchical object tree representation, and property groups. The python API must still be used to get these details.
Snapshot Format
Since the Document.xml and GuiDocument.xml will be used directly, the only additional info needed to store is the property grouping and tree hierarchy, for UI viewing purposes only (not diffing). We can call this a "Extension Snapshot" or similar. It can be stored as json as it's the most efficient at serialization and deserialization.
Flow: Displaying a Tree and Diff
Option 1, no extended snapshot:
Unzipping all files may seem like a lot of overhead, but A) a benchmark showed that the performance is better than the current snapshot YAML deserialization process, so it will actually be a speedup, and B) unzipping and parsing can potentially be run in parallel if further optimization is needed.
Option 2, with extension snapshot:
If both extension snapshots, old and new, are available, the same operation as option 1 can be performed, except it will be displayed in a tree structure, and properties will be grouped nicely.
Migration / Snapshots for Old Commits
Old snapshot files will not work with the new solution. Since the FCStd files contain everything but the tree structure and groups, and we want to be able to support creating extension snapshots for older commits anyway, it makes sense to simply add that functionality now instead of migrating old snapshots. See issue #10 for details.
This will, unfortunately, make existing snapshots that have been committed obsolete, and will require users to recreate snapshots. But this long term solution is more robust and better anyway.
Storage Location
Since it will be possible to backfill snapshots at any time, we should no longer require committing snapshots together with FCStd files. Start storing them in a separate directory that should be gitignored. See issue #5