Skip to content

Split native packaging into a package family + Cake Frosting build #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sergey-v9 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Split native packaging into a package family + Cake Frosting build #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d6ab14d
feat: split native packaging into a package family + Cake build
sergey-v9 May 30, 2026
0d10831
Delete .agents/notes/PR.md
sergey-v9 May 30, 2026
b9802bd
ci: bump GitHub Actions to Node 24 majors
sergey-v9 May 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 60 additions & 2 deletions .agents/notes/DECISIONS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,8 +131,66 @@ Keep this updated whenever a decision is made.
The submodule `origin` and the monorepo `.gitmodules` URL were repointed to it; the trusted-publishing
policy owner is now `LadybugDB` and `RepositoryUrl` points at the org repo.
- `ci.yml` gained a `native-test` matrix (linux-x64 + win-x64) that downloads the prebuilt `liblbug-*`
for `ENGINE_VERSION` (now pinned to `v0.17.0`) and runs the full suite with `LADYBUG_REQUIRE_NATIVE=1`,
so native round-trips run on every push - not only in the release gate.
for the pinned engine release and runs the full suite with `LADYBUG_REQUIRE_NATIVE=1`, so native
round-trips run on every push - not only in the release gate.

## D18 - Split package family + Cake Frosting build (supersedes D16's single fat package)
- Packaging changed from one fat `LadybugDB` package (managed + all natives) to a **family**: managed-only
`LadybugDB`, one `LadybugDB.Native.<rid>` per RID, and a `LadybugDB.Native` meta-package depending on all
five. Consumers now reference TWO packages (`LadybugDB` + a native package). This is a deliberate, breaking
change to the consumption model: it lets an app pull only the platform(s) it needs (reference a single
`LadybugDB.Native.<rid>`) instead of carrying every platform's binary. The managed package has NO
dependency on the native packages - that decoupling is exactly what enables the slim, single-RID install.
- Naming: `LadybugDB.Native` (meta) + `LadybugDB.Native.{win-x64, linux-x64, linux-arm64, osx-x64, osx-arm64}`.
Each native package carries only `runtimes/<rid>/native/*` plus an empty `lib/netstandard2.0/_._` marker
(so NuGet adds no compile reference) and uses `SuppressDependenciesWhenPacking` (no framework dependency).
- `nuget/nuget-package.props` no longer globs `lib/runtimes/**` into the managed package; `LadybugDB` is now
purely `lib/net10.0` + `lib/netstandard2.0` + README + symbols.
- Build tool: **Cake Frosting** (not Nuke), as a normal .NET console app under `cake/`
(`LadybugDB.Build.csproj`, `BuildContext.cs` which also hosts `Main`, `Tasks/`). The folder is named `cake/` (not
the conventional `build/`, which the monorepo root `.gitignore` reserves for build output) so the build
tool is obvious. Layout: `cake/` + `cake/common.props` + `cake/native/`. Tasks: Clean, Restore, BuildManaged, Test,
FetchNatives, PackManaged, PackRuntimes, PackNativeMeta, VerifyPackages, Pack, Default.
- Per-RID packaging is ONE parameterized template (`cake/native/LadybugDB.Native.Runtime.csproj`, packed
once per RID via `-p:NativeRid/-p:PackageId`) rather than five hand-authored nuspecs. The meta-package IS a
nuspec (`cake/native/LadybugDB.Native.nuspec`, a template) because a csproj can't cleanly declare NuGet
dependencies on packages that don't exist on a feed yet; version/commit tokens are substituted in C# at
pack time (avoids passing semicolon-laden `NuspecProperties` through MSBuild). All packing is `dotnet`-only
(no nuget.exe / mono).
- `BuildContext` is the single source of truth for the RID set and the RID->(release asset, library file)
mapping. `EnsureNativeStaged` skips when a native is already staged (e.g. built locally from source),
otherwise downloads the pinned engine asset via `gh` and extracts the canonical library (taking the real
shared object out of the `.so`/`.dylib` symlink chain, since NuGet doesn't preserve symlinks and the binding
loads by path). `VerifyPackages` asserts the managed assemblies, each per-RID payload, and the meta deps.
- Versioning: all packages take ONE release version from the `v*` tag (`--package-version`); `ENGINE_VERSION`
remains the internal pin for WHICH prebuilt natives to fetch, decoupled from the package version. (Alternative
- version native packages by engine version - rejected for a simpler single-version family.)
- Cake gotcha: the host reserves `--version` (prints Cake's own version), so the package version argument is
`--package-version`. The orchestrator project must exclude `native/**` from its compile items, or it picks up
the native projects' generated `AssemblyInfo.cs` and fails with duplicate-attribute errors.
- CI/release now invoke the pipeline (`dotnet run --project cake/... -- --target Test|Pack`) instead of inline
bash. The release `publish` job pushes all 7 packages via OIDC; the trusted-publishing policy / package-id
ownership on nuget.org must now cover every id (`LadybugDB`, `LadybugDB.Native`, and the five
`LadybugDB.Native.<rid>`), not just `LadybugDB`.

## D19 - Package version tracks the upstream engine version with an -alpha.N dev suffix
- All 7 packages share ONE version (per D18). That version's BASE now equals the upstream engine version
(`ENGINE_VERSION` without the leading `v`, e.g. `v0.17.0` -> `0.17.0`), so the binding's released
version lines up with the engine it wraps. This refines D18's versioning bullet: the family is still
uniformly versioned, we just pick the engine version as the base instead of an arbitrary number.
- While the binding is in development the version carries a prerelease suffix (`alpha.1`, `alpha.2`, ...),
so the current default is `0.17.0-alpha.1`. `--prerelease <x>` overrides the suffix, `--prerelease ""`
cuts a stable build equal to the engine version, and `--package-version <v>` overrides the whole thing
(the release workflow passes the exact version from the `v*` git tag).
- Single source of truth: the version lives in ONE data file, `version.txt` at the binding root
(currently `0.17.0-alpha.1`) - not hardcoded in code or workflows. `BuildContext` reads it (engine
release = its base prefixed with `v`, e.g. `v0.17.0`; package version = base + suffix); the managed
`nuget-package.props` reads it (via an MSBuild `File.ReadAllText`) for a direct `dotnet pack`; and the
CI/release workflows derive from it (the `v*` tag overrides on release). Bumping the alpha (or moving
to a new engine) is a one-line edit to `version.txt` - no code or workflow change. This replaced the
earlier `DevelopmentPrerelease` C# constant and the `ENGINE_VERSION` workflow env (both removed);
`--engine-version` / `ENGINE_VERSION` survive only as optional per-run overrides. The old `0.0.0-dev`
/ `0.0.1-alpha` placeholders are gone.

## Open (decide later)
- Timestamp representation: `DateTime` (UTC) for non-tz precisions vs `DateTimeOffset` for `TIMESTAMP_TZ` (Phase 2).
Expand Down
114 changes: 71 additions & 43 deletions .agents/notes/HANDOFF.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,19 +49,31 @@ Copy-Item build/release/src/lbug_shared.dll tools/csharp_api/lib/runtimes/win-x6
dotnet test tools/csharp_api/test/LadybugDB.Tests/LadybugDB.Tests.csproj
```

## Managed-only build / pack (no native needed)
## Build / test / pack via the Cake Frosting pipeline (`cake/`)
All packaging is driven by the Cake Frosting build project under `cake/` (don't hand-run `dotnet pack`).
Use the `build.ps1` / `build.sh` bootstrap from `tools/csharp_api`:
```powershell
cd tools/csharp_api
dotnet build LadybugDB.slnx -c Release # both TFMs, 0 warnings
dotnet pack src/LadybugDB/LadybugDB.csproj -c Release -p:Version=0.1.0-local
./build.ps1 --target Test # build both TFMs + stage host native + run suite
./build.ps1 --target Pack # full package family -> ./artifacts, verified
```
- Whatever native libs sit under `lib/runtimes/<rid>/native/` at pack time are bundled into the
package at `runtimes/<rid>/native/`. With only win-x64 staged, the package is win-x64-only.
- Verified package contents: `lib/net10.0` + `lib/netstandard2.0` (each with XML docs), `README.md`,
and `runtimes/win-x64/native/lbug_shared.dll`. Zero external dependencies in the nuspec.
- Verified by consuming `LadybugDB.0.1.0-local.nupkg` from a local feed: queries run end-to-end and
the native DLL flows to the consumer's `bin/.../runtimes/win-x64/native/` automatically.
- Output goes to `tools/csharp_api/packages/` (gitignored).
- Versioning: the package version tracks the upstream engine version (`v0.17.0` -> `0.17.0`) with an
`-alpha.N` prerelease suffix while in development, so the default is `0.17.0-alpha.1`. It lives in ONE
place - `version.txt` at the binding root - which `BuildContext`, `nuget-package.props`, and both
workflows all read; bump the alpha (or the engine) there with no code change. Overrides: `--prerelease
alpha.2` (suffix), `--package-version <v>` (exact; the release workflow passes the git tag), and
`--prerelease ""` (stable build equal to the engine version).
- The binding now ships a FAMILY (see DECISIONS D18): managed-only `LadybugDB`, one
`LadybugDB.Native.<rid>` per RID, and the `LadybugDB.Native` meta-package. `Pack` stages every RID's
native (downloading from the pinned engine release when not already present), packs all 7, and
`VerifyPackages` asserts contents.
- VERIFIED locally end-to-end (2026-05-30, win-x64 host, engine `v0.17.0`): `--target Pack` produced
and verified all 7 packages; `--target Test` passed 28/28 with the native loaded. Confirmed layouts:
`LadybugDB` = `lib/net10.0` + `lib/netstandard2.0` (with XML docs) + `README.md`, no `runtimes/` and
zero dependencies; `LadybugDB.Native.<rid>` = `runtimes/<rid>/native/<lib>` + `lib/netstandard2.0/_._`
and no dependencies; `LadybugDB.Native` = `_._` + dependencies on all five per-RID packages.
- Cake arg notes: the package version is `--package-version` (the host reserves `--version`);
`--engine-version` overrides the pinned engine; `--commit` (or `GITHUB_SHA`) stamps the repository
metadata. Packages land in `tools/csharp_api/artifacts/` (gitignored).

## Gotchas
- PowerShell mangles `-DCMAKE_POLICY_VERSION_MINIMUM=3.5` into `=3` when args aren't quoted; pass cmake
Expand All @@ -75,14 +87,28 @@ dotnet pack src/LadybugDB/LadybugDB.csproj -c Release -p:Version=0.1.0-local
```
tools/csharp_api/
Directory.Build.props # shared build props (LangVersion, OS/arch detection, paths)
nuget/nuget-package.props # package metadata + native runtime packing
nuget/nuget-package.props # managed package metadata (managed-only; natives ship separately)
LadybugDB.slnx
build.ps1 / build.sh # bootstrap for the Cake Frosting pipeline
cake/ # Cake Frosting packaging pipeline (NOT in LadybugDB.slnx)
LadybugDB.Build.csproj # the build console app (Cake.Frosting)
BuildContext.cs # Main entry point + RID set, paths, EnsureNativeStaged()
Tasks/ # Clean, Restore, BuildManaged, Test, FetchNatives,
# PackManaged, PackRuntimes, PackNativeMeta, VerifyPackages, Pack, Default
common.props # shared NuGet metadata for the native packages
native/ # native packaging assets
LadybugDB.Native.Runtime.csproj # one template, packed once per RID
LadybugDB.Native.Meta.csproj # thin host to pack the meta nuspec via dotnet
LadybugDB.Native.nuspec # meta-package template ($version$/$commit$ tokens)
_._ # empty lib marker
src/LadybugDB/ # the binding (multi-target net10.0 + netstandard2.0)
Interop/ # Native (P/Invoke), per-TFM marshaling, structs, resolver
*.cs # Database, Connection, QueryResult, FlatTuple, Value, ...
test/LadybugDB.Tests/ # xUnit tests (net10.0)
lib/runtimes/<rid>/native/ # native libs are dropped here for packaging/tests (gitignored)
.agents/notes/ # DECISIONS / HANDOFF / ROADMAP
lib/runtimes/<rid>/native/ # native libs are staged here for packaging/tests (gitignored)
artifacts/ # produced .nupkg/.snupkg (gitignored)
download/ # cached engine release assets (gitignored)
.agents/notes/ # DECISIONS / HANDOFF / ROADMAP
```

## Native library requirement
Expand All @@ -93,47 +119,49 @@ tools/csharp_api/
- Tests SKIP (not fail) when the native lib is absent (`TestEnvironment.NativeAvailable`).

## CI / CD (GitHub Actions, in the standalone repo)
Two workflows at the repo root. They do NOT build the engine - natives come from upstream releases.
- `.github/workflows/ci.yml` - PR/push validation. The `build-test` job builds both TFMs, runs the
managed + ABI suite (native round-trips skip without a native lib), and a `dotnet pack` smoke check.
A `native-test` matrix (linux-x64 + win-x64) then downloads the prebuilt `liblbug-*` for `ENGINE_VERSION`
from `LadybugDB/ladybug`, stages it, and re-runs the suite with `LADYBUG_REQUIRE_NATIVE=1` (no skips).
Path-filtered to `src/**`, `test/**`, `**/*.props`, `LadybugDB.slnx`.
Both workflows invoke the Cake pipeline via `dotnet run --project cake/LadybugDB.Build.csproj -- ...`
(natives come from upstream releases; the engine is never built here). `GH_TOKEN` is set so
`FetchNatives` can download the prebuilt assets.
- `.github/workflows/ci.yml` - PR/push validation, path-filtered to `src/**`, `test/**`, `cake/**`,
`nuget/**`, `**/*.props`, `LadybugDB.slnx`. A `test` matrix (linux-x64 + win-x64) runs `--target Test`
(stages the host native, runs the suite with no skips), and a `pack` job runs `--target Pack`
(`--package-version 0.0.0-ci`) to build + verify the whole family without publishing.
- `.github/workflows/release.yml` - the release pipeline. Two jobs:
1. `pack`: `gh release download "$ENGINE_VERSION" --repo LadybugDB/ladybug` pulls the prebuilt
`liblbug-*` assets for all 5 RIDs, `cp -L`s each into `lib/runtimes/<rid>/native/`
(win-x64=`lbug_shared.dll`, linux-x64/arm64=`liblbug.so`, osx-x64/arm64=`liblbug.dylib`),
runs the FULL suite on linux-x64 against the real engine as a gate (`LADYBUG_REQUIRE_NATIVE=1`),
packs with `-p:Version` from the tag, then ASSERTS all 5 natives + both managed TFMs are in the `.nupkg`.
1. `pack`: resolves the version (tag `v1.2.3` -> `1.2.3`) and engine version, runs `--target Test`
as the linux-x64 gate against the real engine, then `--target Pack` (which stages all 5 RIDs, packs
the 7 packages, and `VerifyPackages` asserts every package's contents). Uploads the artifacts.
2. `publish` (only on `v*` tags, `environment: release`): trusted publishing via `NuGet/login@v1`
(`id-token: write`) + `dotnet nuget push --skip-duplicate`.
- `ENGINE_VERSION` (env in `release.yml`) pins the upstream engine release the natives are taken from;
override per-run with the `engine_version` dispatch input. Keep it in sync with the managed ABI.
(`id-token: write`) + `dotnet nuget push "artifacts/*.nupkg" --skip-duplicate` - now pushes ALL 7
packages (the `.snupkg` symbols ride along with the managed package).
- The upstream engine release the natives are taken from defaults to the version's base (from
`version.txt`, or the `v*` tag on release); override per-run with `--engine-version` / `ENGINE_VERSION`
or the `engine_version` dispatch input. Keep it in sync with the managed ABI.

### Releasing a version
```bash
git tag v0.1.0 # tag drives the package version (v1.2.3 -> 1.2.3)
git push origin v0.1.0
```
`workflow_dispatch` (with a `version` input) builds + packs + uploads the artifact WITHOUT publishing -
use it to dry-run the multi-RID build before tagging.
`workflow_dispatch` (with a `version` input) builds + packs + uploads the artifacts WITHOUT publishing -
use it to dry-run the full family build before tagging.

### One-time setup before the first publish (MAINTAINER ACTION)
- nuget.org -> Account -> Trusted Publishing -> Add policy:
owner=`LadybugDB`, repo=`ladybug-dotnet`, workflow file=`release.yml`, environment=`release`.
- nuget.org -> Account -> Trusted Publishing -> Add a policy for EACH package id (the publish job pushes
all 7): `LadybugDB`, `LadybugDB.Native`, and `LadybugDB.Native.{win-x64, linux-x64, linux-arm64,
osx-x64, osx-arm64}`. owner=`LadybugDB`, repo=`ladybug-dotnet`, workflow file=`release.yml`,
environment=`release` for each.
- Repo Settings -> Environments -> `release`: add secret `NUGET_USER` = the nuget.org PROFILE name
(not email) that owns/co-owns the package id. Optionally add required reviewers as an approval gate.
- The `LadybugDB` package id must be owned (or reserved) by that nuget.org account before the first real
push; until then, use `workflow_dispatch` / local `dotnet pack` as a dry run.
- Before the first real publish, dry-run: `dotnet pack -c Release -o ./artifacts` (or the dispatch run).
(not email) that owns/co-owns the package ids. Optionally add required reviewers as an approval gate.
- All 7 package ids must be owned (or reserved) by that nuget.org account before the first real push;
until then, use `workflow_dispatch` / local `./build.ps1 --target Pack` as a dry run.

## Next steps
1. Confirm the pinned `ENGINE_VERSION` (`v0.17.0`) release of `LadybugDB/ladybug` actually carries the
`liblbug-*` assets for all 5 RIDs, then run `release.yml` via `workflow_dispatch` once to confirm the
multi-RID natives download, the package assembles, and the linux-x64 gate passes (proven on win-x64
locally; the new `ci.yml` native matrix now also exercises linux-x64 + win-x64 on every push).
2. Complete the nuget.org trusted-publishing policy + `release` environment, then tag `v*`.
3. Reserve/own the `LadybugDB` package id on nuget.org under the publishing account (the repo already
lives in the LadybugDB org).
1. Local end-to-end is DONE (2026-05-30): `--target Pack` built + verified all 7 packages and
`--target Test` passed 28/28 against the engine `v0.17.0` natives on win-x64. Next, run `release.yml`
via `workflow_dispatch` once to confirm the same on CI (the linux-x64 gate + all-RID download/pack).
2. Complete the nuget.org trusted-publishing policies (one per package id) + `release` environment, then
tag `v*`.
3. Reserve/own all 7 package ids on nuget.org under the publishing account (the repo already lives in the
LadybugDB org).
4. Phase 3 (remaining): expand the suite to mirror the Java + C API tests over `dataset/tinysnb`.
5. Phase 5 (optional): Native AOT validation, Arrow C Data interface, observability.
Loading
Loading