ci: add C# (NuGet) packaging for Substrait artifacts

[nielspardon]

Summary

Adds a csharp/ tree alongside python/, java/, rust/ and cpp/, publishing three independently versioned NuGet packages built from the spec release:

Package	Contents	Analogue
Substrait.Protobuf	Generated protobuf bindings (Google.Protobuf)	substrait-protobuf, substrait-prost
Substrait.Antlr	Generated ANTLR parsers (Antlr4.Runtime.Standard)	substrait-antlr
Substrait.Extensions	Extension YAMLs, text schemas, test cases and dialect fixtures as embedded resources	java/extensions, cpp/substrait-extensions

All three package ids are unclaimed on nuget.org. The protos already declare option csharp_namespace = "Substrait.Protobuf", so the naming follows what the spec itself chose.

Draft because two org-side settings are needed before a real release can run — see Before this can publish below.

What each package does, and why

All three multi-target netstandard2.0 and net10.0. netstandard2.0 does the real work — it keeps .NET Framework 4.6.2+, Mono and Unity consumers viable and is also what .NET 8/9 consumers resolve. The modern leg is net10.0 rather than net8.0 because .NET 8 leaves support in November 2026 and these packages are published on every spec release for years; nothing is excluded by that choice, since anything older falls back to netstandard2.0. Shared build settings and NuGet metadata live in csharp/Directory.Build.props; each package has a sibling *.Tests project.

Substrait.Protobuf vendors the .proto files and generates C# at build time with Grpc.Tools, which supplies protoc and the C# plugin as a build-only dependency. Nothing generated is committed, and no protoc install is needed. The C++ rationale for build-time generation (ABI coupling to a protobuf runtime) does not apply — generated C# is ordinary managed source — so this is tidiness rather than necessity. The .proto files ship in the nupkg under proto/ for consumers who want to run their own codegen.

Substrait.Antlr commits its generated parsers, as the Rust and C++ targets do: the ANTLR tool is a Java program, and neither consumers nor the publish workflow should need a JDK. Uses the stock ANTLR C# target, so no fork is required unlike Rust. Each grammar set gets its own namespace (Substrait.Antlr.SubstraitType, Substrait.Antlr.FuncTestCase), mirroring the Rust modules and C++ namespaces.

Substrait.Extensions ships data only, matching Java and C++, with zero package dependencies. SubstraitExtensions locates and reads the embedded files; the resource paths are identical to the Java artifact's (substrait/extensions/...).

Why no generated type layer for extensions

I tried this first, since .NET does have a JSON Schema code generator and it would have matched Rust's typify and Python's datamodel-code-generator. It does not survive contact with the schemas.

The Draft 2020-12 $defs keyword needs a rename to Draft-07 definitions for NJsonSchema's reference resolver, which is a harmless keyword swap. The blocker is oneOf. simple_extensions_schema.yaml models a function argument as enumeration_arg | value_arg | type_arg with no discriminator property, and NJsonSchema collapses that to the first branch:

public partial class Arguments : System.Collections.ObjectModel.Collection<Enumeration_arg>

Enumeration_arg has Name, Description and Options — and no Value. Nearly every real extension file uses value_arg (- name: x, value: fp32), so the generated type would silently drop the field that matters. The output also leaks snake_case into public names (Type_variations, Scalar_functions, Type_param_defs) and produces Anonymous/Anonymous2/Impls3 for inline schemas.

Publishing that as package API would be worse than publishing none, and once shipped the names are a breaking change to fix. So the typed layer is out of scope here and left to downstream consumers. Worth a follow-up issue: it needs either a generator that handles undiscriminated unions or hand-written models, and that is arguably a decision for the binding repo rather than the packaging repo.

Release machinery

csharp_{protobuf,antlr,extensions}.yml follow the same prechecks → generate → tag → pack → publish shape as rust_prost.yml, gathered by csharp_publish.yml and wired into publish_artifacts.yml. Tags are csharp/<PackageId>/vx.y.z, which satisfies tag_exists.sh's existing format check unchanged.

Two differences from the other languages, both because of how NuGet works:

• No committed version. dotnet pack -p:Version= takes it at pack time, so there is no counterpart to uv version or scripts/rust/set_version.sh. NuGet SemVer has no v prefix, so the tag's v is stripped.
• Trusted publishing. NuGet/login@v1 exchanges the workflow's OIDC token for a short-lived API key, which is the direct counterpart to the PyPI id-token: write flow and crates-io-auth-action. No long-lived secret.

New scripts:

• scripts/csharp/nuget_package_exists.sh — idempotency, peer of pypi_package_exists.sh and crate_exists.sh.
• scripts/csharp/smoke_test.sh — restores a packed .nupkg into a throwaway consumer project from a <clear />ed local feed and runs a statement against it. This is the counterpart to the Python wheel/sdist smoke tests, and it is what catches mistakes a project-reference build cannot see: wrong TFM asset folders, undeclared dependencies, or spec data left out of the nupkg.

ci_csharp.yml runs on PRs touching csharp/ and additionally packs and smoke tests all three packages.

Validation

Everything below was run locally against spec v0.99.0 using pixi's own dotnet-sdk (10.0.302, the same build conda-forge resolves for all four platforms in pixi.toml):

• All three generate tasks (pixi run csharp-generate-{protobuf,antlr,extensions}).
• pixi run csharp-build — all three libraries build both netstandard2.0 and net8.0; 15 tests pass (5 protobuf, 4 ANTLR, 6 extensions).
• dotnet pack on the solution — 6 artifacts (.nupkg + .snupkg each).
• smoke_test.sh against every packed package, at both a release version and the 0.0.0-ci prerelease CI uses.
• All 28 workflow files parse, and publish_artifacts.yml now fans out to 5 languages.

The ANTLR tests assert the exact same parse trees as python/substrait-antlr, so a grammar change that alters a tree shows up identically across targets.

As with the other languages, main carries only the machinery — the generated parsers and vendored spec data are .gitkeep placeholders here and land on release tags.

Publishing prerequisites — done

Both org-side settings are now in place:

1. A nuget GitHub environment, matching the existing pypi and crates-io environments.
2. A NUGET_USER repository variable, plus trusted-publishing policies on nuget.org.

One correction to an earlier version of this description, which said the policies are registered "for the three package ids": NuGet policies are not scoped to package ids. They are scoped to (Repository Owner, Repository, Workflow File, optional Environment), and apply to every package owned by the policy's package owner. The workflow filename is what matters, and since three separate workflows push, three policies are required:

Repository Owner	Repository	Workflow File	Environment
substrait-io	substrait-packaging	csharp_protobuf.yml	nuget
substrait-io	substrait-packaging	csharp_antlr.yml	nuget
substrait-io	substrait-packaging	csharp_extensions.yml	nuget

These name the reusable workflows rather than publish_artifacts.yml, because the match is against the job's own workflow file. That is confirmed empirically rather than assumed: all three Python packages are on PyPI at 0.99.0, published through the identical nesting (python_protobuf.yml <- python_publish.yml <- publish_artifacts.yml).

Policies are registered with package owner substrait (the nuget.org organization), created by an individual account. NUGET_USER is set to that creator's username, not the organization's -- which is what NuGet/login requires: it posts {"username": ...} to the token endpoint and its own failure message reads "Make sure you are using the username of the policy creator, not the policy owner".

Notes for review

• Both published assets are executed. Test projects target net10.0 only, so they cover that asset. netstandard2.0 gets its runtime coverage from smoke_test.sh, which consumes each packed package twice — from a net8.0 project (which resolves netstandard2.0, since net10.0 is not compatible) and from a net10.0 project. Verified by reading TargetFrameworkAttribute off the bound assembly: .NETStandard,Version=v2.0 and .NETCoreApp,Version=v10.0 respectively. No net472 test project needed.
• The smoke test could previously pass against a stale package. NuGet keys its cache on id + version and will not re-extract a version it has already seen, and the version is constant across runs here (the spec version, or 0.0.0-ci). smoke_test.sh now restores into a scratch NUGET_PACKAGES folder so the nupkg under test is always the one exercised. Caught while verifying asset selection — a rebuilt package resolved the previously cached copy.
• ANTLR runtime version. Antlr4.Runtime.Standard trails the tool: 4.13.1 is the newest published C# runtime while pixi.toml pins antlr >=4.13.2. Safe today, since ANTLR's RuntimeMetaData.CheckVersion compares major.minor only, but a bump to 4.14 needs a matching C# runtime release first. I left the shared pin alone (it is also Java's and C++'s) and noted the constraint in pixi.toml and the package README.
• Tests as sibling projects. Substrait.Protobuf.Tests/ sits next to Substrait.Protobuf/ rather than inside it, unlike the Python and C++ layouts. Nesting a project inside another makes the library's default source globbing swallow the test sources; working around that needs Directory.Build.targets ordering tricks, and sibling test projects are the .NET norm.

Downstream

This is what lets substrait-csharp drop its substrait submodule and its own Grpc.Tools codegen (see substrait-csharp#5, which is currently bumping that submodule by hand) in favour of a PackageReference to Substrait.Protobuf.

🤖 Generated with AI