Build SOCI index file as part of pushing to repository

[bschwedler]

We need to build the SOCI index as part of the build manifest merge process for multi-platform builds.

CloudOps has told us that they will not be able to uses these images in the DataOps cluster unless they have SOCI support.

---

• https://github.com/awslabs/soci-snapshotter/blob/main/docs/soci-index-manifest-v2.md
• Add soci v2 index to workbench-session-init rstudio/rstudio-docker-products#981

Can use soci convert --all-platforms
https://github.com/awslabs/soci-snapshotter/blob/main/docs/cli-usage.md#soci-convert

[bschwedler]

Work evaluation

Design decision

Post-merge SOCI conversion, indexing for all registries. soci convert --all-platforms operates directly on the multi-platform manifest index that the ORAS merge step already produces.

1. New SOCI plugin in posit-bakery

Mirror the ORAS plugin shape (posit_bakery/plugins/builtin/oras/):

• posit_bakery/plugins/builtin/soci/__init__.py — SociPlugin(BakeryToolPlugin) with execute()/results()
• posit_bakery/plugins/builtin/soci/soci.py — find_soci_bin(), SociConvert command wrapper
• Register in the builtin plugin registry

Core operation per target: soci convert --all-platforms <ref> <ref> against each final destination produced by the merge (same tag rewritten with SOCI zTOC layers embedded). Destinations are already available on OrasMergeWorkflowResult.destinations.

2. Wire into the merge pipelines

Native path (bakery ci merge): In cli/ci.py after oras.execute() succeeds, call soci.execute() against the same config.targets, passing dry_run. Reads each target's final tags — no new state needed.

QEMU path (bakery build --push in bakery-build.yml): No merge step exists here — the workflow pushes multi-arch directly. Options:

• Add a --soci-index flag to bakery build that runs soci convert per published tag after push. Preferred — keeps it inside the CLI.
• Skip SOCI for the QEMU path initially and document the gap.

Worth deciding before implementation.

3. CI workflow changes

In both bakery-build-native.yml (merge job) and bakery-build.yml (build job), add a SOCI install step:

• Pull binary from awslabs/soci-snapshotter release assets. Merge runs on ubuntu-latest-4x (amd64), so single amd64 tarball. Prefer a direct SHA-pinned download over the third-party lerentis/soci-installer action used in the reference PR.
• setup-bakery is a plausible home if we want soci available everywhere; otherwise inline in the merge job.

soci convert (v0.10+) is a pure registry-to-registry operation — does not need containerd or ctr, unlike the legacy reference PR. Simplifies setup significantly.

4. Testing

• Unit tests mirroring test/plugins/oras/ with mocked subprocess
• Local-registry integration test via the existing services/registry_management harness that the ORAS merge tests already use
• End-to-end verification via a draft PR in an images-connect worktree pointed at a feature branch of images-shared, pushing to GHCR

5. Open questions / risks

• Runner capacity. soci convert pulls every layer for every platform on the merge runner. Workbench matrix images are multi-GB. ubuntu-latest-4x (16GB/150GB) is likely fine but worth measuring; may need a larger merge runner.
• Merge-time added. Minutes per image. Fine for weekly production, more costly on daily dev builds. Gate SOCI behind --push (dry-run indexing has no value).
• ORAS→SOCI handoff. soci convert --all-platforms expects a multi-platform index; the ORAS merge produces one, but ORAS-created indexes may carry annotations SOCI does not expect. Test early.
• Cleanup. SOCI convert pushes new layers and leaves old manifests referenced by digest. Verify clean.yml does not orphan pre-SOCI manifests.
• QEMU workflow decision (see section 2).

Effort

Small-to-moderate. Plugin is a few hundred lines by analogy; workflow wiring is a handful of steps. Real risk is end-to-end behavior on a real registry with a real multi-arch index. Recommend a spike PR that runs soci convert manually against an existing published tag before investing in the plugin scaffolding.

References

• https://github.com/awslabs/soci-snapshotter/blob/main/docs/cli-usage.md#soci-convert
• https://github.com/awslabs/soci-snapshotter/blob/main/docs/soci-index-manifest-v2.md
• https://github.com/awslabs/soci-snapshotter/releases
• Add soci v2 index to workbench-session-init rstudio/rstudio-docker-products#981