Public MPS source code does not include transforms

[rptb1]

When the MPS was under a restrictive licence, Ravenbrook developed an extension for our commercial client, Configura, to allow transformation of certain objects on the heap, using the garbage collector to rewrite them. This code was proprietary to Configura.

Since then, we arranged to publish the MPS under a non-restrictive licence, and Configura have generously agreed that their transforms extension can be shared under that same licence.

The extension is not visible in Git and it might be of benefit to other MPS users.

[rptb1]

Removed "optional" importance because I think this needs review at the next discussion based on our belief that design.mps.walk does not meet Configura's requirements. Resolving this issue is likely essential for version 1.118.

[rptb1]

@thejayps and @rptb1 did a detailed walkthrough of the transforms extension that raised issues that should be dealt with when we incorporate it in the public MPS. The meeting notes are in this internal email message.

[rptb1]

our belief that design.mps.walk does not meet Configura's requirements

Summary from our walkthrough notes:

1. Performance: mps_pool_walk time complexity is O(heap size). Transforms are only upper bounded by O(heap size) but typically O(number of objects to be transformed). See point 23.
2. Function: mps_pool_walk does not scan the roots. Exact roots must be updated to transform objects to meet Configura's requirements of class instance replacement/update. See point 24.
3. Function: mps_pool_walk doesn't deal with ambiguous references and detecting whether it's even possible to do a transform. See point 25.
4. Risk: mps_pool_walk requires the client to implement and maintain subtle pieces of code that could introduce heap corruption and therefore cause very expensive failures in deployment. See point 26, point 27, point 28, and point 29.

[rptb1]

A straightforward merge of the transforms from custom/cet/main may not be feasible until https://github.com/Ravenbrook/mps/milestone/2 because the transforms are explicitly excluded from export by Git Fusion. This will require some thought.

[rptb1]

Documentation of transforms (and mps_pool_walk) should include a comparison of the two. A good starting point is https://info.ravenbrook.com/mail/2020/08/31/16-31-32/0.txt by @gareth-rees :

A comparison between my mps_pool_walk design and RHSK’s transforms design:

1. transforms makes use of MPS summaries to (potentially) reduce the amount of scanning — mps_pool_walk scans everything
2. transforms can only transform references — mps_pool_walk can modify the object in any way it likes
3. transforms is specialized to the hot reloading case — mps_pool_walk has other use cases, e.g. serialization
4. transforms covers references in roots — you’d have to combine mps_pool_walk with mps_arena_roots_walk
5. transforms fails if an ambiguous reference (apparently) needs to be transformed — you can’t detect this case with mps_arena_roots_walk since it doesn’t tell you the rank of the reference.

[rptb1]

5. transforms fails if an ambiguous reference (apparently) needs to be transformed — you can’t detect this case with mps_arena_roots_walk since it doesn’t tell you the rank of the reference.

The implementation of transforms doesn't offer the client program any choice in this matter, or indeed much feedback about what stopped the transform. An implementation using roots walking would provide that, but is much harder for the client to implement. Perhaps there's a compromise.

[rptb1]

Documentation of Transforms is now visible at https://www.ravenbrook.com/project/mps/custom/cet/main/manual/html/custom/cet/transform.html along with other extensions developed for the Configura commercial client.