feat(go): add Go packaging for protobuf, ANTLR and extensions

[nielspardon]

Summary

Adds Go packaging for three modules under github.com/substrait-io/substrait-packaging/go/, mirroring the existing Python/Rust/Java structure:

Module	Contents	Codegen
substrait-protobuf	protobuf bindings (substraitpb + substraitpb/extensions), Open (struct) API	buf remote protocolbuffers/go
substrait-protobuf-opaque	same bindings, Opaque (builder) API	buf remote protocolbuffers/go (API_OPAQUE)
substrait-antlr	parsers — substraittype (listener) + functestcase (visitor)	stock ANTLR Go target (no fork)
substrait-extensions	extension/text/test-case data via //go:embed	none — data only (Java model)

How Go publishing differs

Unlike PyPI / crates.io / Maven Central, a Go module is published by pushing a version-control tag — the proxy serves source straight from the tag. So there's no registry upload step and no registry credentials:

• Each per-package workflow generates the code, commits it, pushes a go/<pkg>/vX.Y.Z tag (that is the publish), and warms proxy.golang.org so pkg.go.dev indexes it.
• Generated code and vendored data therefore live only on release tags; the branch keeps only scaffolding (substrait-extensions uses .gitkeep placeholders, so it compiles only after data is vendored).

Alpha publishing (enabled by default, like Rust)

alpha: true is the default so we can publish alpha pre-releases for substrait-go integration testing. scripts/go/next_alpha_version.sh auto-increments -alpha → -alpha.1 → … from the repository's own tags. Semver pre-releases are excluded from @latest, so they don't become the default for consumers. Flip to alpha: false for a stable release once integration is proven.

Design notes

• substrait-extensions ships data only. Go has no canonical JSON-schema code generator, and the extension schema's polymorphic function definitions don't map cleanly onto Go's type system — so typed parsing stays downstream (substrait-go), matching the Java extensions artifact. The package is named substrait and exposes GetSubstraitExtensionsFS()/GetSubstraitTestsFS() (plus a new GetSubstraitTextFS()), making it a drop-in for the legacy spec-repo embed module.
• ANTLR uses the stock Go target (antlr4-go/antlr/v4 runtime) — no fork required, unlike the Rust crate.
• protobuf uses buf remote plugins — no local protoc/protoc-gen-go needed; only buf (already a pixi dependency).
• Two protobuf modules, one codegen flag apart. substrait-protobuf ships the Open (struct) API and substrait-protobuf-opaque ships the Opaque (builder) API — same protos, same runtime/version, differing only by default_api_level=API_OPAQUE. They're wire-compatible but expose distinct Go types, so consumers pick whichever fits. Each is its own module with its own tag namespace, so they version and deprecate independently.

Contents

• Packages under go/ with per-package generate scripts + go-generate-* pixi tasks
• scripts/go/module_exists.sh (proxy idempotency check) and scripts/go/next_alpha_version.sh
• ci_go.yml (PR validation) and go_{protobuf,protobuf_opaque,antlr,extensions}.yml + go_publish.yml, wired into publish_artifacts.yml

Validation

• All generators run against the real spec (v0.94.0); all three modules build, extensions tests pass, protobuf vets clean
• A dry-run migration of substrait-go against these modules builds clean and passes its full test suite (one minor gotcha: a bare baseparser import needs an explicit alias since the module package is functestcase)

Follow-ups (separate, after publishing alphas): migrate substrait-go to these modules, then deprecate the old substrait-protobuf/go module and the spec-repo embed module.

🤖 Generated with AI

[jacques-n]

you might consider two protobuf modules: the old pattern (structs) and the new one (builders).

[nielspardon]

you might consider two protobuf modules: the old pattern (structs) and the new one (builders).

Good call — added a second module, substrait-protobuf-opaque, with the builder API alongside the existing struct-based substrait-protobuf.

[benbellick]

@nielspardon Is this high priority for you to get in?

I ask because at DD we are trying to solve the diamond dependency problem for Go so we can do more graceful substrait-go upgrades in our monorepo. This PR would move Go code from substrait-protobuf to substrait-packaging, but I think we would still be stuck with the same underlying problem. The generated Go protobuf package is still published under one module path using the Substrait spec version, rather than a version that reflects generated Go API compatibility.

If not urgent, would you mind waiting a bit? I am writing up a proposal and will share it in an issue once it is ready.

If this is high priority for you to get in, no worries. We may just need to change the Go versioning scheme later, which could mean using a different namespace/module path. Thanks!

[nielspardon]

If not urgent, would you mind waiting a bit? I am writing up a proposal and will share it in an issue once it is ready.

Not critical for me. I was just trying to help get this work completed since it was sitting dormant for quite some time since @vbarua started it. This is more about ensuring we are leaving a good impression on people evaluating Substrait. Currently, we have SDKs at different spec levels, with breaking builds, different feature levels. I want to make sure we are looking like and becoming a healthy community. That was my main motivation while starting to work on Go.