From d6ab14d50c9f53fc05df4e439554170de09ab784 Mon Sep 17 00:00:00 2001 From: Sergey Vershinin Date: Sat, 30 May 2026 23:09:33 +0200 Subject: [PATCH 1/3] feat: split native packaging into a package family + Cake build Restructure the .NET binding's packaging so the native engine no longer ships inside the managed package: - LadybugDB - managed-only (no native payload) - LadybugDB.Native. - one package per platform (win-x64, linux-x64, linux-arm64, osx-x64, osx-arm64), carrying just that RID's native library - LadybugDB.Native - meta-package depending on all per-RID packages Consumers reference LadybugDB plus a native package (the meta for "runs anywhere", or a single RID for a slim app). The managed package has no dependency on the natives, which is what enables the slim, single-RID install. Packaging is driven by a new Cake Frosting build under cake/: one parameterized per-RID template plus a meta nuspec. Natives are staged from a locally-built library when present, otherwise downloaded from the pinned engine release via gh and extracted out of the .so/.dylib symlink chain. VerifyPackages asserts the managed assemblies, every per-RID payload, and the meta's deps. The version is single-sourced from version.txt at the binding root (currently 0.17.0-alpha.1): it tracks the upstream engine version with an -alpha.N dev suffix and is read by BuildContext, nuget-package.props, and both workflows. ci.yml and release.yml now invoke the pipeline (dotnet run -- --target ...) instead of inline shell; release publishes all 7 packages to nuget.org via OIDC trusted publishing. --- .agents/notes/DECISIONS.md | 62 ++++- .agents/notes/HANDOFF.md | 114 +++++---- .agents/notes/PR.md | 110 +++++++++ .agents/notes/ROADMAP.md | 23 +- .github/workflows/ci.yml | 99 +++----- .github/workflows/release.yml | 139 ++++------- .gitignore | 2 + AGENTS.md | 42 +++- README.md | 59 ++++- build.ps1 | 16 ++ build.sh | 11 + cake/BuildContext.cs | 250 ++++++++++++++++++++ cake/LadybugDB.Build.csproj | 25 ++ cake/LadybugDB.Build.slnx | 7 + cake/Tasks/Pipeline.cs | 190 +++++++++++++++ cake/Tasks/VerifyPackagesTask.cs | 124 ++++++++++ cake/common.props | 17 ++ cake/native/LadybugDB.Native.Meta.csproj | 17 ++ cake/native/LadybugDB.Native.Runtime.csproj | 39 +++ cake/native/LadybugDB.Native.nuspec | 40 ++++ cake/native/_._ | 0 nuget/nuget-package.props | 12 +- version.txt | 1 + 23 files changed, 1158 insertions(+), 241 deletions(-) create mode 100644 .agents/notes/PR.md create mode 100644 build.ps1 create mode 100644 build.sh create mode 100644 cake/BuildContext.cs create mode 100644 cake/LadybugDB.Build.csproj create mode 100644 cake/LadybugDB.Build.slnx create mode 100644 cake/Tasks/Pipeline.cs create mode 100644 cake/Tasks/VerifyPackagesTask.cs create mode 100644 cake/common.props create mode 100644 cake/native/LadybugDB.Native.Meta.csproj create mode 100644 cake/native/LadybugDB.Native.Runtime.csproj create mode 100644 cake/native/LadybugDB.Native.nuspec create mode 100644 cake/native/_._ create mode 100644 version.txt diff --git a/.agents/notes/DECISIONS.md b/.agents/notes/DECISIONS.md index c07570d..abac802 100644 --- a/.agents/notes/DECISIONS.md +++ b/.agents/notes/DECISIONS.md @@ -131,8 +131,66 @@ Keep this updated whenever a decision is made. The submodule `origin` and the monorepo `.gitmodules` URL were repointed to it; the trusted-publishing policy owner is now `LadybugDB` and `RepositoryUrl` points at the org repo. - `ci.yml` gained a `native-test` matrix (linux-x64 + win-x64) that downloads the prebuilt `liblbug-*` - for `ENGINE_VERSION` (now pinned to `v0.17.0`) and runs the full suite with `LADYBUG_REQUIRE_NATIVE=1`, - so native round-trips run on every push - not only in the release gate. + for the pinned engine release and runs the full suite with `LADYBUG_REQUIRE_NATIVE=1`, so native + round-trips run on every push - not only in the release gate. + +## D18 - Split package family + Cake Frosting build (supersedes D16's single fat package) +- Packaging changed from one fat `LadybugDB` package (managed + all natives) to a **family**: managed-only + `LadybugDB`, one `LadybugDB.Native.` per RID, and a `LadybugDB.Native` meta-package depending on all + five. Consumers now reference TWO packages (`LadybugDB` + a native package). This is a deliberate, breaking + change to the consumption model: it lets an app pull only the platform(s) it needs (reference a single + `LadybugDB.Native.`) instead of carrying every platform's binary. The managed package has NO + dependency on the native packages - that decoupling is exactly what enables the slim, single-RID install. +- Naming: `LadybugDB.Native` (meta) + `LadybugDB.Native.{win-x64, linux-x64, linux-arm64, osx-x64, osx-arm64}`. + Each native package carries only `runtimes//native/*` plus an empty `lib/netstandard2.0/_._` marker + (so NuGet adds no compile reference) and uses `SuppressDependenciesWhenPacking` (no framework dependency). +- `nuget/nuget-package.props` no longer globs `lib/runtimes/**` into the managed package; `LadybugDB` is now + purely `lib/net10.0` + `lib/netstandard2.0` + README + symbols. +- Build tool: **Cake Frosting** (not Nuke), as a normal .NET console app under `cake/` + (`LadybugDB.Build.csproj`, `BuildContext.cs` which also hosts `Main`, `Tasks/`). The folder is named `cake/` (not + the conventional `build/`, which the monorepo root `.gitignore` reserves for build output) so the build + tool is obvious. Layout: `cake/` + `cake/common.props` + `cake/native/`. Tasks: Clean, Restore, BuildManaged, Test, + FetchNatives, PackManaged, PackRuntimes, PackNativeMeta, VerifyPackages, Pack, Default. +- Per-RID packaging is ONE parameterized template (`cake/native/LadybugDB.Native.Runtime.csproj`, packed + once per RID via `-p:NativeRid/-p:PackageId`) rather than five hand-authored nuspecs. The meta-package IS a + nuspec (`cake/native/LadybugDB.Native.nuspec`, a template) because a csproj can't cleanly declare NuGet + dependencies on packages that don't exist on a feed yet; version/commit tokens are substituted in C# at + pack time (avoids passing semicolon-laden `NuspecProperties` through MSBuild). All packing is `dotnet`-only + (no nuget.exe / mono). +- `BuildContext` is the single source of truth for the RID set and the RID->(release asset, library file) + mapping. `EnsureNativeStaged` skips when a native is already staged (e.g. built locally from source), + otherwise downloads the pinned engine asset via `gh` and extracts the canonical library (taking the real + shared object out of the `.so`/`.dylib` symlink chain, since NuGet doesn't preserve symlinks and the binding + loads by path). `VerifyPackages` asserts the managed assemblies, each per-RID payload, and the meta deps. +- Versioning: all packages take ONE release version from the `v*` tag (`--package-version`); `ENGINE_VERSION` + remains the internal pin for WHICH prebuilt natives to fetch, decoupled from the package version. (Alternative + - version native packages by engine version - rejected for a simpler single-version family.) +- Cake gotcha: the host reserves `--version` (prints Cake's own version), so the package version argument is + `--package-version`. The orchestrator project must exclude `native/**` from its compile items, or it picks up + the native projects' generated `AssemblyInfo.cs` and fails with duplicate-attribute errors. +- CI/release now invoke the pipeline (`dotnet run --project cake/... -- --target Test|Pack`) instead of inline + bash. The release `publish` job pushes all 7 packages via OIDC; the trusted-publishing policy / package-id + ownership on nuget.org must now cover every id (`LadybugDB`, `LadybugDB.Native`, and the five + `LadybugDB.Native.`), not just `LadybugDB`. + +## D19 - Package version tracks the upstream engine version with an -alpha.N dev suffix +- All 7 packages share ONE version (per D18). That version's BASE now equals the upstream engine version + (`ENGINE_VERSION` without the leading `v`, e.g. `v0.17.0` -> `0.17.0`), so the binding's released + version lines up with the engine it wraps. This refines D18's versioning bullet: the family is still + uniformly versioned, we just pick the engine version as the base instead of an arbitrary number. +- While the binding is in development the version carries a prerelease suffix (`alpha.1`, `alpha.2`, ...), + so the current default is `0.17.0-alpha.1`. `--prerelease ` overrides the suffix, `--prerelease ""` + cuts a stable build equal to the engine version, and `--package-version ` overrides the whole thing + (the release workflow passes the exact version from the `v*` git tag). +- Single source of truth: the version lives in ONE data file, `version.txt` at the binding root + (currently `0.17.0-alpha.1`) - not hardcoded in code or workflows. `BuildContext` reads it (engine + release = its base prefixed with `v`, e.g. `v0.17.0`; package version = base + suffix); the managed + `nuget-package.props` reads it (via an MSBuild `File.ReadAllText`) for a direct `dotnet pack`; and the + CI/release workflows derive from it (the `v*` tag overrides on release). Bumping the alpha (or moving + to a new engine) is a one-line edit to `version.txt` - no code or workflow change. This replaced the + earlier `DevelopmentPrerelease` C# constant and the `ENGINE_VERSION` workflow env (both removed); + `--engine-version` / `ENGINE_VERSION` survive only as optional per-run overrides. The old `0.0.0-dev` + / `0.0.1-alpha` placeholders are gone. ## Open (decide later) - Timestamp representation: `DateTime` (UTC) for non-tz precisions vs `DateTimeOffset` for `TIMESTAMP_TZ` (Phase 2). diff --git a/.agents/notes/HANDOFF.md b/.agents/notes/HANDOFF.md index d9a8243..d40e305 100644 --- a/.agents/notes/HANDOFF.md +++ b/.agents/notes/HANDOFF.md @@ -49,19 +49,31 @@ Copy-Item build/release/src/lbug_shared.dll tools/csharp_api/lib/runtimes/win-x6 dotnet test tools/csharp_api/test/LadybugDB.Tests/LadybugDB.Tests.csproj ``` -## Managed-only build / pack (no native needed) +## Build / test / pack via the Cake Frosting pipeline (`cake/`) +All packaging is driven by the Cake Frosting build project under `cake/` (don't hand-run `dotnet pack`). +Use the `build.ps1` / `build.sh` bootstrap from `tools/csharp_api`: ```powershell -cd tools/csharp_api -dotnet build LadybugDB.slnx -c Release # both TFMs, 0 warnings -dotnet pack src/LadybugDB/LadybugDB.csproj -c Release -p:Version=0.1.0-local +./build.ps1 --target Test # build both TFMs + stage host native + run suite +./build.ps1 --target Pack # full package family -> ./artifacts, verified ``` -- Whatever native libs sit under `lib/runtimes//native/` at pack time are bundled into the - package at `runtimes//native/`. With only win-x64 staged, the package is win-x64-only. -- Verified package contents: `lib/net10.0` + `lib/netstandard2.0` (each with XML docs), `README.md`, - and `runtimes/win-x64/native/lbug_shared.dll`. Zero external dependencies in the nuspec. -- Verified by consuming `LadybugDB.0.1.0-local.nupkg` from a local feed: queries run end-to-end and - the native DLL flows to the consumer's `bin/.../runtimes/win-x64/native/` automatically. -- Output goes to `tools/csharp_api/packages/` (gitignored). +- Versioning: the package version tracks the upstream engine version (`v0.17.0` -> `0.17.0`) with an + `-alpha.N` prerelease suffix while in development, so the default is `0.17.0-alpha.1`. It lives in ONE + place - `version.txt` at the binding root - which `BuildContext`, `nuget-package.props`, and both + workflows all read; bump the alpha (or the engine) there with no code change. Overrides: `--prerelease + alpha.2` (suffix), `--package-version ` (exact; the release workflow passes the git tag), and + `--prerelease ""` (stable build equal to the engine version). +- The binding now ships a FAMILY (see DECISIONS D18): managed-only `LadybugDB`, one + `LadybugDB.Native.` per RID, and the `LadybugDB.Native` meta-package. `Pack` stages every RID's + native (downloading from the pinned engine release when not already present), packs all 7, and + `VerifyPackages` asserts contents. +- VERIFIED locally end-to-end (2026-05-30, win-x64 host, engine `v0.17.0`): `--target Pack` produced + and verified all 7 packages; `--target Test` passed 28/28 with the native loaded. Confirmed layouts: + `LadybugDB` = `lib/net10.0` + `lib/netstandard2.0` (with XML docs) + `README.md`, no `runtimes/` and + zero dependencies; `LadybugDB.Native.` = `runtimes//native/` + `lib/netstandard2.0/_._` + and no dependencies; `LadybugDB.Native` = `_._` + dependencies on all five per-RID packages. +- Cake arg notes: the package version is `--package-version` (the host reserves `--version`); + `--engine-version` overrides the pinned engine; `--commit` (or `GITHUB_SHA`) stamps the repository + metadata. Packages land in `tools/csharp_api/artifacts/` (gitignored). ## Gotchas - PowerShell mangles `-DCMAKE_POLICY_VERSION_MINIMUM=3.5` into `=3` when args aren't quoted; pass cmake @@ -75,14 +87,28 @@ dotnet pack src/LadybugDB/LadybugDB.csproj -c Release -p:Version=0.1.0-local ``` tools/csharp_api/ Directory.Build.props # shared build props (LangVersion, OS/arch detection, paths) - nuget/nuget-package.props # package metadata + native runtime packing + nuget/nuget-package.props # managed package metadata (managed-only; natives ship separately) LadybugDB.slnx + build.ps1 / build.sh # bootstrap for the Cake Frosting pipeline + cake/ # Cake Frosting packaging pipeline (NOT in LadybugDB.slnx) + LadybugDB.Build.csproj # the build console app (Cake.Frosting) + BuildContext.cs # Main entry point + RID set, paths, EnsureNativeStaged() + Tasks/ # Clean, Restore, BuildManaged, Test, FetchNatives, + # PackManaged, PackRuntimes, PackNativeMeta, VerifyPackages, Pack, Default + common.props # shared NuGet metadata for the native packages + native/ # native packaging assets + LadybugDB.Native.Runtime.csproj # one template, packed once per RID + LadybugDB.Native.Meta.csproj # thin host to pack the meta nuspec via dotnet + LadybugDB.Native.nuspec # meta-package template ($version$/$commit$ tokens) + _._ # empty lib marker src/LadybugDB/ # the binding (multi-target net10.0 + netstandard2.0) Interop/ # Native (P/Invoke), per-TFM marshaling, structs, resolver *.cs # Database, Connection, QueryResult, FlatTuple, Value, ... test/LadybugDB.Tests/ # xUnit tests (net10.0) - lib/runtimes//native/ # native libs are dropped here for packaging/tests (gitignored) - .agents/notes/ # DECISIONS / HANDOFF / ROADMAP + lib/runtimes//native/ # native libs are staged here for packaging/tests (gitignored) + artifacts/ # produced .nupkg/.snupkg (gitignored) + download/ # cached engine release assets (gitignored) + .agents/notes/ # DECISIONS / HANDOFF / ROADMAP ``` ## Native library requirement @@ -93,47 +119,49 @@ tools/csharp_api/ - Tests SKIP (not fail) when the native lib is absent (`TestEnvironment.NativeAvailable`). ## CI / CD (GitHub Actions, in the standalone repo) -Two workflows at the repo root. They do NOT build the engine - natives come from upstream releases. -- `.github/workflows/ci.yml` - PR/push validation. The `build-test` job builds both TFMs, runs the - managed + ABI suite (native round-trips skip without a native lib), and a `dotnet pack` smoke check. - A `native-test` matrix (linux-x64 + win-x64) then downloads the prebuilt `liblbug-*` for `ENGINE_VERSION` - from `LadybugDB/ladybug`, stages it, and re-runs the suite with `LADYBUG_REQUIRE_NATIVE=1` (no skips). - Path-filtered to `src/**`, `test/**`, `**/*.props`, `LadybugDB.slnx`. +Both workflows invoke the Cake pipeline via `dotnet run --project cake/LadybugDB.Build.csproj -- ...` +(natives come from upstream releases; the engine is never built here). `GH_TOKEN` is set so +`FetchNatives` can download the prebuilt assets. +- `.github/workflows/ci.yml` - PR/push validation, path-filtered to `src/**`, `test/**`, `cake/**`, + `nuget/**`, `**/*.props`, `LadybugDB.slnx`. A `test` matrix (linux-x64 + win-x64) runs `--target Test` + (stages the host native, runs the suite with no skips), and a `pack` job runs `--target Pack` + (`--package-version 0.0.0-ci`) to build + verify the whole family without publishing. - `.github/workflows/release.yml` - the release pipeline. Two jobs: - 1. `pack`: `gh release download "$ENGINE_VERSION" --repo LadybugDB/ladybug` pulls the prebuilt - `liblbug-*` assets for all 5 RIDs, `cp -L`s each into `lib/runtimes//native/` - (win-x64=`lbug_shared.dll`, linux-x64/arm64=`liblbug.so`, osx-x64/arm64=`liblbug.dylib`), - runs the FULL suite on linux-x64 against the real engine as a gate (`LADYBUG_REQUIRE_NATIVE=1`), - packs with `-p:Version` from the tag, then ASSERTS all 5 natives + both managed TFMs are in the `.nupkg`. + 1. `pack`: resolves the version (tag `v1.2.3` -> `1.2.3`) and engine version, runs `--target Test` + as the linux-x64 gate against the real engine, then `--target Pack` (which stages all 5 RIDs, packs + the 7 packages, and `VerifyPackages` asserts every package's contents). Uploads the artifacts. 2. `publish` (only on `v*` tags, `environment: release`): trusted publishing via `NuGet/login@v1` - (`id-token: write`) + `dotnet nuget push --skip-duplicate`. -- `ENGINE_VERSION` (env in `release.yml`) pins the upstream engine release the natives are taken from; - override per-run with the `engine_version` dispatch input. Keep it in sync with the managed ABI. + (`id-token: write`) + `dotnet nuget push "artifacts/*.nupkg" --skip-duplicate` - now pushes ALL 7 + packages (the `.snupkg` symbols ride along with the managed package). +- The upstream engine release the natives are taken from defaults to the version's base (from + `version.txt`, or the `v*` tag on release); override per-run with `--engine-version` / `ENGINE_VERSION` + or the `engine_version` dispatch input. Keep it in sync with the managed ABI. ### Releasing a version ```bash git tag v0.1.0 # tag drives the package version (v1.2.3 -> 1.2.3) git push origin v0.1.0 ``` -`workflow_dispatch` (with a `version` input) builds + packs + uploads the artifact WITHOUT publishing - -use it to dry-run the multi-RID build before tagging. +`workflow_dispatch` (with a `version` input) builds + packs + uploads the artifacts WITHOUT publishing - +use it to dry-run the full family build before tagging. ### One-time setup before the first publish (MAINTAINER ACTION) -- nuget.org -> Account -> Trusted Publishing -> Add policy: - owner=`LadybugDB`, repo=`ladybug-dotnet`, workflow file=`release.yml`, environment=`release`. +- nuget.org -> Account -> Trusted Publishing -> Add a policy for EACH package id (the publish job pushes + all 7): `LadybugDB`, `LadybugDB.Native`, and `LadybugDB.Native.{win-x64, linux-x64, linux-arm64, + osx-x64, osx-arm64}`. owner=`LadybugDB`, repo=`ladybug-dotnet`, workflow file=`release.yml`, + environment=`release` for each. - Repo Settings -> Environments -> `release`: add secret `NUGET_USER` = the nuget.org PROFILE name - (not email) that owns/co-owns the package id. Optionally add required reviewers as an approval gate. -- The `LadybugDB` package id must be owned (or reserved) by that nuget.org account before the first real - push; until then, use `workflow_dispatch` / local `dotnet pack` as a dry run. -- Before the first real publish, dry-run: `dotnet pack -c Release -o ./artifacts` (or the dispatch run). + (not email) that owns/co-owns the package ids. Optionally add required reviewers as an approval gate. +- All 7 package ids must be owned (or reserved) by that nuget.org account before the first real push; + until then, use `workflow_dispatch` / local `./build.ps1 --target Pack` as a dry run. ## Next steps -1. Confirm the pinned `ENGINE_VERSION` (`v0.17.0`) release of `LadybugDB/ladybug` actually carries the - `liblbug-*` assets for all 5 RIDs, then run `release.yml` via `workflow_dispatch` once to confirm the - multi-RID natives download, the package assembles, and the linux-x64 gate passes (proven on win-x64 - locally; the new `ci.yml` native matrix now also exercises linux-x64 + win-x64 on every push). -2. Complete the nuget.org trusted-publishing policy + `release` environment, then tag `v*`. -3. Reserve/own the `LadybugDB` package id on nuget.org under the publishing account (the repo already - lives in the LadybugDB org). +1. Local end-to-end is DONE (2026-05-30): `--target Pack` built + verified all 7 packages and + `--target Test` passed 28/28 against the engine `v0.17.0` natives on win-x64. Next, run `release.yml` + via `workflow_dispatch` once to confirm the same on CI (the linux-x64 gate + all-RID download/pack). +2. Complete the nuget.org trusted-publishing policies (one per package id) + `release` environment, then + tag `v*`. +3. Reserve/own all 7 package ids on nuget.org under the publishing account (the repo already lives in the + LadybugDB org). 4. Phase 3 (remaining): expand the suite to mirror the Java + C API tests over `dataset/tinysnb`. 5. Phase 5 (optional): Native AOT validation, Arrow C Data interface, observability. diff --git a/.agents/notes/PR.md b/.agents/notes/PR.md new file mode 100644 index 0000000..1bfbdbc --- /dev/null +++ b/.agents/notes/PR.md @@ -0,0 +1,110 @@ +# Split native packaging into a package family + Cake Frosting build + +> Living PR description — keep this current as the branch evolves. The checklist at the bottom is the +> source of truth for what's left before merge / first publish. + +## Summary + +Restructures the LadybugDB .NET binding's packaging so the native engine no longer ships inside the +managed package. Instead it's a **package family**: + +- `LadybugDB` — managed-only (no native payload). +- `LadybugDB.Native.` — one package per platform, carrying just that RID's native library. +- `LadybugDB.Native` — meta-package that depends on all per-RID packages. + +Consumers reference `LadybugDB` **plus** a native package: the meta-package for "runs anywhere", or a +single `LadybugDB.Native.` for a slim, single-platform app. All packaging is driven by a new +**Cake Frosting** build project under `cake/`. + +## Motivation + +The old single package bundled every platform's native library into one `.nupkg`, so every consumer +carried all of them regardless of target. Splitting the natives into per-RID packages (with a meta +"all platforms" package) lets an app pull only what it needs, while keeping a one-line install for the +common case. + +## What changed + +### Package family + +- Managed `LadybugDB` is now managed-only: removed the `runtimes/`** native glob from +`nuget/nuget-package.props` (it now ships `lib/net10.0` + `lib/netstandard2.0` + README + symbols only). +- Each `LadybugDB.Native.` carries `runtimes//native/`* plus an empty `lib/netstandard2.0/_._` +marker (so NuGet adds no compile reference) and declares no dependencies. +- `LadybugDB.Native` meta-package depends on all five per-RID packages. +- Shipped RIDs: `win-x64`, `linux-x64`, `linux-arm64`, `osx-x64`, `osx-arm64`. +- The managed package intentionally has **no** dependency on any native package — that decoupling is +what makes the slim, single-RID install possible. + +### Build pipeline (`cake/`) + +- A Cake Frosting console app: `LadybugDB.Build.csproj` + `BuildContext.cs` (which also hosts the `Main` +entry point), and the pipeline tasks under `Tasks/` (`Pipeline.cs` for the small tasks + +`VerifyPackagesTask.cs`). Bootstrap with `build.ps1` / `build.sh`. +- Tasks: `Clean`, `Restore`, `BuildManaged`, `Test`, `FetchNatives`, `PackManaged`, `PackRuntimes`, +`PackNativeMeta`, `VerifyPackages`, `Pack`, `Default`. +- `BuildContext` is the single source of truth for the RID set and the RID -> (release asset, library) +mapping. `EnsureNativeStaged` reuses a locally-built native if present, otherwise downloads the pinned +engine release asset (via `gh`) and extracts the canonical library (pulling the real shared object out +of the `.so`/`.dylib` symlink chain, since NuGet doesn't preserve symlinks). +- Per-RID packages come from one parameterized template (`cake/native/LadybugDB.Native.Runtime.csproj`, +packed once per RID). The meta-package is a nuspec template (`cake/native/LadybugDB.Native.nuspec`) +with version/commit substituted at pack time. All packing is `dotnet`-only (no nuget.exe / mono). +- `VerifyPackages` asserts the managed assemblies, every per-RID payload, and the meta's dependency set, +so a silent packaging regression fails the build instead of shipping a broken package. + +### Versioning + +- The package version tracks the upstream engine version, with an `-alpha.N` prerelease suffix while in +development. Current default: **`0.17.0-alpha.1`**. +- It lives in ONE place - `version.txt` at the binding root - read by `BuildContext`, the managed +`nuget-package.props`, and both CI workflows. Bump the alpha (or the engine) there with no code change. +`--prerelease ""` cuts a stable build equal to the engine version; `--package-version ` overrides +exactly (the release workflow passes the `v*` git tag). + +### CI / release + +- `ci.yml` and `release.yml` now invoke the pipeline (`dotnet run --project cake/... -- --target ...`) +instead of inline shell. `ci.yml` runs a `test` matrix (linux-x64 + win-x64) and a `pack` smoke; +`release.yml` runs the linux-x64 test gate, packs + verifies, then publishes all packages via OIDC. + +### Docs + +- README (install + build sections), `AGENTS.md` (Packaging). + +## Consumption + +```bash +# runs on any supported platform +dotnet add package LadybugDB +dotnet add package LadybugDB.Native + +# slim, single-platform +dotnet add package LadybugDB +dotnet add package LadybugDB.Native.win-x64 +``` + +## Validation + +- Local end-to-end (win-x64 host, engine `v0.17.0`): `--target Pack` produced and verified all 7 +packages as `0.17.0-alpha.1` (correct layouts; meta pins all five per-RID deps; no stray references); +`--target Test` passed 28/28 with the native loaded (no skips). +- Build is warning- and lint-clean. + +## Next steps / TODO + +Maintainer actions before first publish: + +- **Owner — still TODO (one-time):** create the `release` GitHub environment on +`LadybugDB/ladybug-dotnet` and add secret `NUGET_USER` = the nuget.org **profile name** (NOT email). +The `publish` job declares `environment: release` and `NuGet/login@v1` reads `secrets.NUGET_USER`, so +without these the publish step fails with a 403/auth error. +- First green CI run: trigger `release.yml` via `workflow_dispatch` to confirm the linux-x64 gate + +all-RID download/pack on CI (proven locally on win-x64 so far). + +Follow-ups (can land later): + +- Bump the `-alpha.N` suffix as the binding stabilizes; drop it for the first stable `v0.17.0`. +- Phase 3 (remaining): expand the suite to mirror the Java + C API tests over `dataset/tinysnb`. +- Phase 5 (optional): Native AOT validation, Arrow C Data interface, observability. + diff --git a/.agents/notes/ROADMAP.md b/.agents/notes/ROADMAP.md index 5170d7f..b6b50ae 100644 --- a/.agents/notes/ROADMAP.md +++ b/.agents/notes/ROADMAP.md @@ -40,20 +40,23 @@ via `../../`. - [x] ABI guard tests (`StructLayoutTests`, 17) assert struct sizes/offsets without needing the native lib ## Phase 4 - Packaging + release [in progress] -- [x] `dotnet pack` produces a valid package (lib/net10.0 + lib/netstandard2.0 + README + XML docs) - [x] Local native build recipe + `scripts/build-native-and-test.ps1` (win-x64, MSVC + Ninja) - [x] win-x64 native lib wired into `lib/runtimes/win-x64/native/` and loaded by the tests -- [x] Package contains the win-x64 native lib at `runtimes/win-x64/native/lbug_shared.dll`; - consume-test from a local feed runs queries end-to-end (native asset flows to consumer output) - [x] Repo split: standalone `LadybugDB/ladybug-dotnet` (in the LadybugDB org), wired as the `tools/csharp_api` submodule (local) -- [x] Multi-RID release workflow (`.github/workflows/release.yml`, tag `v*`): downloads prebuilt - `liblbug-*` for all 5 RIDs (win-x64, linux-x64/arm64, osx-x64/arm64) from `LadybugDB/ladybug` - releases (pinned `ENGINE_VERSION`), stages -> linux-x64 gate -> packs -> asserts contents -> OIDC publish -- [x] `.github/workflows/ci.yml` on PR/push: managed `build-test` (both TFMs + ABI tests + pack smoke) plus a `native-test` matrix (linux-x64 + win-x64) running the suite against downloaded natives - [x] Repo adopted into the LadybugDB org as `LadybugDB/ladybug-dotnet` -- [ ] One-time nuget.org trusted-publishing policy + `release` environment / `NUGET_USER` secret (maintainer action) -- [ ] First green release run (`release.yml` needs a `LadybugDB/ladybug` release carrying `liblbug-*`; confirm `ENGINE_VERSION` = `v0.17.0`) -- [ ] `LadybugDB` package-id ownership/reservation on nuget.org before a real publish +- [x] **Split package family** (DECISIONS D18): managed-only `LadybugDB` + `LadybugDB.Native.` per + RID + `LadybugDB.Native` meta-package. Consumers reference `LadybugDB` plus a native package; + managed has no native dependency so single-RID slim installs work. +- [x] **Cake Frosting** build project under `cake/` (Clean/Restore/BuildManaged/Test/FetchNatives/ + PackManaged/PackRuntimes/PackNativeMeta/VerifyPackages/Pack), with `build.ps1`/`build.sh` bootstrap. +- [x] `nuget/nuget-package.props` stripped of native packing (managed-only). +- [x] Verified end-to-end locally (2026-05-30, win-x64, engine `v0.17.0`): `--target Pack` produced + + verified all 7 packages (correct layouts, meta deps, no stray references); `--target Test` 28/28. +- [x] CI/release rewired to drive the pipeline (`dotnet run --project cake/... -- --target Test|Pack`); + release `publish` pushes all 7 packages via OIDC; `ci.yml` runs the `test` matrix + a `pack` job. +- [ ] One-time nuget.org trusted-publishing policy (one per package id) + `release` environment / `NUGET_USER` secret (maintainer action) +- [ ] First green release run on CI (`workflow_dispatch` to confirm the linux-x64 gate + all-RID download/pack) +- [ ] Ownership/reservation of all 7 package ids on nuget.org before a real publish - [ ] Optional: win-arm64 / linux-musl RIDs (not produced by the upstream precompiled workflow today) ## Phase 5 (optional) - Extras [pending] diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6b96093..53fc5e0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,8 +1,13 @@ name: CI -# Fast managed validation plus native smoke coverage for the C# binding. The managed job builds -# both target frameworks and verifies packing; the native matrix stages prebuilt engine libraries -# and runs the same test suite with native skips disabled. +# Fast validation for the C# binding, driven by the Cake Frosting pipeline in cake/. +# - test: builds both target frameworks, stages the host RID's prebuilt engine library, and runs the +# full suite with native skips disabled. +# - pack: produces and validates the whole package family (managed + per-RID native + meta) without +# publishing. +# +# Native engine libraries are downloaded as prebuilt `liblbug-*` release assets from the upstream engine +# repo (LadybugDB/ladybug); the release is pinned via version.txt at the binding root. on: push: @@ -10,6 +15,8 @@ on: paths: - 'src/**' - 'test/**' + - 'cake/**' + - 'nuget/**' - '**/*.props' - 'LadybugDB.slnx' - '.github/workflows/ci.yml' @@ -17,6 +24,8 @@ on: paths: - 'src/**' - 'test/**' + - 'cake/**' + - 'nuget/**' - '**/*.props' - 'LadybugDB.slnx' - '.github/workflows/ci.yml' @@ -25,33 +34,9 @@ on: permissions: contents: read -env: - # Upstream engine release that the prebuilt native libraries are taken from. Bump in lockstep with - # the native ABI; release.yml uses the same default. - ENGINE_VERSION: v0.17.0 - jobs: - build-test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - - name: Setup .NET - uses: actions/setup-dotnet@v4 - with: - dotnet-version: '10.0.x' - - - name: Build (net10.0 + netstandard2.0) - run: dotnet build src/LadybugDB/LadybugDB.csproj -c Release - - - name: Test (managed + ABI guards; native round-trips skip) - run: dotnet test test/LadybugDB.Tests/LadybugDB.Tests.csproj -c Release -v minimal - - - name: Pack smoke check - run: dotnet pack src/LadybugDB/LadybugDB.csproj -c Release -o artifacts - - native-test: - name: native-test (${{ matrix.rid }}) + test: + name: test (${{ matrix.rid }}) runs-on: ${{ matrix.os }} strategy: fail-fast: false @@ -59,13 +44,8 @@ jobs: include: - os: ubuntu-latest rid: linux-x64 - asset: liblbug-linux-x86_64.tar.gz - library: liblbug.so - os: windows-latest rid: win-x64 - asset: liblbug-windows-x86_64.zip - library: lbug_shared.dll - steps: - uses: actions/checkout@v4 @@ -74,38 +54,31 @@ jobs: with: dotnet-version: '10.0.x' - - name: Download prebuilt native engine + - name: Build + test (stages the host native engine, no skips) + shell: bash env: GH_TOKEN: ${{ github.token }} - ASSET: ${{ matrix.asset }} - run: gh release download "$ENGINE_VERSION" --repo LadybugDB/ladybug --dir dl --pattern "$ASSET" - shell: bash + run: dotnet run --project cake/LadybugDB.Build.csproj -- --target Test - - name: Stage native library (Linux) - if: matrix.rid == 'linux-x64' - run: | - set -euo pipefail - mkdir -p lib/runtimes/${{ matrix.rid }}/native - work=$(mktemp -d) - tar -xzf dl/${{ matrix.asset }} -C "$work" - cp -L "$work/${{ matrix.library }}" lib/runtimes/${{ matrix.rid }}/native/${{ matrix.library }} - ls -l lib/runtimes/${{ matrix.rid }}/native - shell: bash + pack: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 - - name: Stage native library (Windows) - if: matrix.rid == 'win-x64' - run: | - $ErrorActionPreference = 'Stop' - $nativeDir = 'lib/runtimes/${{ matrix.rid }}/native' - New-Item -ItemType Directory -Force -Path $nativeDir | Out-Null - $work = Join-Path $env:RUNNER_TEMP 'lbug-native' - New-Item -ItemType Directory -Force -Path $work | Out-Null - Expand-Archive -Path 'dl/${{ matrix.asset }}' -DestinationPath $work -Force - Copy-Item -Path (Join-Path $work '${{ matrix.library }}') -Destination (Join-Path $nativeDir '${{ matrix.library }}') -Force - Get-ChildItem $nativeDir - shell: pwsh + - name: Setup .NET + uses: actions/setup-dotnet@v4 + with: + dotnet-version: '10.0.x' - - name: Test against native engine + - name: Pack the full package family and verify contents (no publish) env: - LADYBUG_REQUIRE_NATIVE: '1' - run: dotnet test test/LadybugDB.Tests/LadybugDB.Tests.csproj -c Release -v minimal + GH_TOKEN: ${{ github.token }} + run: dotnet run --project cake/LadybugDB.Build.csproj -- --target Pack + + - uses: actions/upload-artifact@v4 + with: + name: nuget-packages + path: | + artifacts/*.nupkg + artifacts/*.snupkg + if-no-files-found: error diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f7a3f98..d361e3a 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,18 +1,21 @@ name: Release -# Builds the LadybugDB NuGet package (managed binding + per-RID native libs) and publishes it to -# nuget.org via trusted publishing (OIDC). +# Builds the full LadybugDB package family (managed binding + per-RID native packages + native +# meta-package) via the Cake Frosting pipeline in cake/, and publishes it to nuget.org through +# trusted publishing (OIDC). # -# Native engine libraries are NOT built here; they are downloaded as prebuilt `liblbug-*` release -# assets from the upstream engine repo (LadybugDB/ladybug), pinned to ENGINE_VERSION. +# Native engine libraries are NOT built here; the pipeline downloads prebuilt `liblbug-*` release +# assets from the upstream engine repo (LadybugDB/ladybug). The engine release defaults to the base +# of the version being built (the git tag, or version.txt for manual runs). # # Triggers: -# - push a tag like `v0.1.0` -> build, test, pack, and PUBLISH to nuget.org +# - push a tag like `v0.1.0` -> build, test, pack, and PUBLISH all packages to nuget.org # - manual `workflow_dispatch` -> build, test, and pack only (artifact uploaded; no publish) # # One-time setup required before the first publish (see notes/HANDOFF.md): -# - nuget.org trusted publishing policy: owner=LadybugDB, repo=ladybug-dotnet, -# workflow file=release.yml, environment=release +# - nuget.org trusted publishing policy covering every package id: LadybugDB, LadybugDB.Native, and +# LadybugDB.Native.{win-x64, linux-x64, linux-arm64, osx-x64, osx-arm64} +# (owner=LadybugDB, repo=ladybug-dotnet, workflow file=release.yml, environment=release) # - GitHub environment `release` with secret NUGET_USER = your nuget.org profile name (not email) on: @@ -22,25 +25,20 @@ on: workflow_dispatch: inputs: version: - description: 'Package version to build (e.g. 0.1.0-preview.1). Not published on manual runs.' + description: 'Package version to build (defaults to version.txt). Not published on manual runs.' required: false - default: '0.0.0-dev' + default: '' engine_version: - description: 'LadybugDB/ladybug release tag to pull prebuilt native libs from (overrides ENGINE_VERSION).' + description: "LadybugDB/ladybug release tag to pull prebuilt native libs from (defaults to the version's base)." required: false default: '' permissions: contents: read -env: - # Upstream engine release that the prebuilt native libraries are taken from. Bump in lockstep with - # the native ABI; override per-run via the workflow_dispatch engine_version input. - ENGINE_VERSION: v0.17.0 - jobs: - # Download prebuilt natives for every RID, stage them, validate end-to-end on linux-x64 against the - # real engine, then pack the multi-RID package. + # Stage prebuilt natives for every RID, gate on the linux-x64 suite against the real engine, then + # pack and validate the whole package family. pack: runs-on: ubuntu-latest steps: @@ -57,109 +55,52 @@ jobs: run: | set -euo pipefail if [[ "${GITHUB_REF:-}" == refs/tags/v* ]]; then - VERSION="${GITHUB_REF_NAME#v}" - else + VERSION="${GITHUB_REF_NAME#v}" # the tag is the authoritative release version + elif [[ -n "${{ inputs.version }}" ]]; then VERSION="${{ inputs.version }}" + else + VERSION="$(tr -d '[:space:]' < version.txt)" # single source of truth at the binding root fi echo "version=$VERSION" >> "$GITHUB_OUTPUT" echo "Resolved package version: $VERSION" - - name: Download prebuilt native engine + - name: Resolve engine version + id: engine shell: bash - env: - GH_TOKEN: ${{ github.token }} run: | set -euo pipefail + VERSION="${{ steps.ver.outputs.version }}" ENGINE="${{ inputs.engine_version }}" - ENGINE="${ENGINE:-$ENGINE_VERSION}" + ENGINE="${ENGINE:-v${VERSION%%-*}}" # default: engine release matching the version's base + echo "engine=$ENGINE" >> "$GITHUB_OUTPUT" echo "Pulling native libs from LadybugDB/ladybug release: $ENGINE" - gh release download "$ENGINE" --repo LadybugDB/ladybug --dir dl \ - --pattern 'liblbug-windows-x86_64.zip' \ - --pattern 'liblbug-linux-x86_64.tar.gz' \ - --pattern 'liblbug-linux-aarch64.tar.gz' \ - --pattern 'liblbug-osx-x86_64.tar.gz' \ - --pattern 'liblbug-osx-arm64.tar.gz' - ls -l dl - - - name: Stage native libraries into runtimes//native - shell: bash - run: | - set -euo pipefail - ROOT=lib/runtimes - mkdir -p "$ROOT"/{win-x64,linux-x64,linux-arm64,osx-x64,osx-arm64}/native - work=$(mktemp -d) - - # Windows x64: zip with lbug_shared.dll - unzip -o dl/liblbug-windows-x86_64.zip -d "$work/win" - cp "$work/win/lbug_shared.dll" "$ROOT/win-x64/native/lbug_shared.dll" - # Linux: tarballs contain liblbug.so -> .so.0 -> .so.0.x.y symlink chain; -L copies the - # real object into a single regular file (we load it by path, so the SONAME is irrelevant). - mkdir -p "$work/lx64" "$work/la64" - tar -xzf dl/liblbug-linux-x86_64.tar.gz -C "$work/lx64" - cp -L "$work/lx64/liblbug.so" "$ROOT/linux-x64/native/liblbug.so" - tar -xzf dl/liblbug-linux-aarch64.tar.gz -C "$work/la64" - cp -L "$work/la64/liblbug.so" "$ROOT/linux-arm64/native/liblbug.so" - - # macOS: tarballs contain liblbug.dylib symlink chain; same -L treatment. - mkdir -p "$work/mx64" "$work/ma64" - tar -xzf dl/liblbug-osx-x86_64.tar.gz -C "$work/mx64" - cp -L "$work/mx64/liblbug.dylib" "$ROOT/osx-x64/native/liblbug.dylib" - tar -xzf dl/liblbug-osx-arm64.tar.gz -C "$work/ma64" - cp -L "$work/ma64/liblbug.dylib" "$ROOT/osx-arm64/native/liblbug.dylib" - - echo "==== staged native libraries ====" - find "$ROOT" -type f -printf '%p (%s bytes)\n' - - - name: Test end-to-end (linux-x64 against the real engine) + - name: Test gate (linux-x64 against the real engine) env: - LADYBUG_REQUIRE_NATIVE: '1' # turn a "native didn't load" skip into a hard failure - run: dotnet test test/LadybugDB.Tests/LadybugDB.Tests.csproj -c Release -v minimal - - - name: Pack + GH_TOKEN: ${{ github.token }} run: > - dotnet pack src/LadybugDB/LadybugDB.csproj - -c Release - -p:Version=${{ steps.ver.outputs.version }} - -p:ContinuousIntegrationBuild=true - -o artifacts + dotnet run --project cake/LadybugDB.Build.csproj -- + --target Test + --engine-version "${{ steps.engine.outputs.engine }}" - - name: Verify package contents - shell: bash - run: | - set -euo pipefail - pkg=$(ls artifacts/*.nupkg) - echo "==== $pkg ====" - unzip -l "$pkg" - listing=$(unzip -Z1 "$pkg") - expected=( - "lib/net10.0/LadybugDB.dll" - "lib/netstandard2.0/LadybugDB.dll" - "runtimes/win-x64/native/lbug_shared.dll" - "runtimes/linux-x64/native/liblbug.so" - "runtimes/linux-arm64/native/liblbug.so" - "runtimes/osx-x64/native/liblbug.dylib" - "runtimes/osx-arm64/native/liblbug.dylib" - ) - missing=0 - for entry in "${expected[@]}"; do - if ! grep -Fxq "$entry" <<<"$listing"; then - echo "::error::package is missing expected entry: $entry" - missing=1 - fi - done - [ "$missing" -eq 0 ] || { echo "::error::package validation failed"; exit 1; } - echo "All expected managed assemblies and per-RID native libraries are present." + - name: Pack + verify the full package family + env: + GH_TOKEN: ${{ github.token }} + run: > + dotnet run --project cake/LadybugDB.Build.csproj -- + --target Pack + --package-version "${{ steps.ver.outputs.version }}" + --engine-version "${{ steps.engine.outputs.engine }}" - uses: actions/upload-artifact@v4 with: - name: nuget-package + name: nuget-packages path: | artifacts/*.nupkg artifacts/*.snupkg if-no-files-found: error - # Publish to nuget.org with trusted publishing. Only runs for v* tags. + # Publish every package to nuget.org with trusted publishing. Only runs for v* tags. publish: needs: pack if: startsWith(github.ref, 'refs/tags/v') @@ -177,7 +118,7 @@ jobs: - name: Download packed artifacts uses: actions/download-artifact@v4 with: - name: nuget-package + name: nuget-packages path: artifacts - name: NuGet login (OIDC trusted publishing) diff --git a/.gitignore b/.gitignore index c2ed35c..9d52094 100644 --- a/.gitignore +++ b/.gitignore @@ -2,5 +2,7 @@ bin/ obj/ packages/ lib/ +artifacts/ +download/ *.user .vs/ diff --git a/AGENTS.md b/AGENTS.md index 09b918f..bf642cc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,7 +19,7 @@ descriptions under [Deeper docs](#deeper-docs)). `lbug.h`; stage a native lib and run the FULL test suite before proposing any interop change (ABI bugs never surface at compile time). - **Ask first:** changing marshalling, calling convention, or struct layout; bumping the pinned engine -version (`ENGINE_VERSION`); adding/dropping a RID or changing `runtimes/{rid}/native/` packaging. +version (the base of `version.txt`); adding/dropping a RID or changing `runtimes/{rid}/native/` packaging. - **Never:** alter a struct's field order/width, the `byte`-for-C-`bool` rule, or the who-frees-what string/blob ownership without sign-off. Never add a finalizer or `SafeHandle` to the disposables (deliberate — see D10). Never commit native binaries: everything under `lib/` is a gitignored artifact. @@ -83,19 +83,39 @@ There is **no** binding generator (no ClangSharp). "Source-generated" refers onl ## Upstream coupling The binding targets the Ladybug engine C API in the separate `LadybugDB/ladybug` repo, pinned to one -release: `ENGINE_VERSION` in `.github/workflows/release.yml` is the upstream tag the -prebuilt natives are pulled from. The two repos are no longer a single commit: when bumping -`ENGINE_VERSION`, re-sync the managed signatures/structs/enums against that release's -`src/include/c_api/lbug.h` and update the ABI tests in the same change. +release: the engine tag is the base of `version.txt` at the repo root (e.g. `0.17.0-alpha.1` -> `v0.17.0`), +overridable per build via `--engine-version` / `ENGINE_VERSION`. The two repos are no longer a single +commit: when moving to a new engine release, re-sync the managed signatures/structs/enums against that +release's `src/include/c_api/lbug.h` and update the ABI tests in the same change. ## Packaging -`dotnet pack src/LadybugDB/LadybugDB.csproj -c Release -p:Version=` bundles whatever sits under -`lib/runtimes//native/` into `runtimes//native/` in the `.nupkg`. Shipped RIDs: win-x64, -linux-x64, linux-arm64, osx-x64, osx-arm64. The release pipeline (`.github/workflows/release.yml`, -tag `v`*) downloads the prebuilt `liblbug-`* assets for the pinned `ENGINE_VERSION` from -`LadybugDB/ladybug` releases, stages them, gates on the linux-x64 suite, asserts package contents, -then publishes via OIDC. Details + one-time nuget.org setup: `.agents/notes/HANDOFF.md`. +The binding ships as a **family** of packages, not one fat package: the managed-only `LadybugDB`, one +`LadybugDB.Native.` per RID (carrying just `runtimes//native/*` + an empty `_._` lib marker), +and the `LadybugDB.Native` meta-package that depends on all of them. Consumers reference `LadybugDB` +plus a native package (the meta for all platforms, or a single RID for a slim app). Shipped RIDs: +win-x64, linux-x64, linux-arm64, osx-x64, osx-arm64. + +A **Cake Frosting** build project under `cake/` drives everything (don't hand-run `dotnet pack`): + +```bash +./build.sh --target Test # build + stage host native + run the suite +./build.sh --target Pack --package-version # full family into ./artifacts, then verify contents +``` + +- `cake/BuildContext.cs` is the single source of truth for the RID set and the RID->(asset, library) + mapping; `EnsureNativeStaged` downloads the pinned engine asset (`gh`) and extracts the canonical + library into `lib/runtimes//native/` (skips when one is already staged from a local source build). +- `cake/native/LadybugDB.Native.Runtime.csproj` is one template packed once per RID; the meta-package + is `cake/native/LadybugDB.Native.nuspec` (version/commit tokens substituted at pack time). `_._` and + `SuppressDependenciesWhenPacking` keep the native packages free of compile references and framework deps. +- `VerifyPackages` asserts the managed assemblies, every per-RID payload, and the meta's dependency set. + +The release pipeline (`.github/workflows/release.yml`, tag `v*`) runs `--target Test` (linux-x64 gate +against the real engine) then `--target Pack`, and publishes all 7 packages via OIDC. The engine release +the natives come from defaults to the version's base (`version.txt`, or the `v*` tag on release). Details ++ one-time nuget.org setup (the trusted publishing policy must now cover every package id): +`.agents/notes/HANDOFF.md`. ## Deeper docs diff --git a/README.md b/README.md index bb8311a..0f2b4db 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,31 @@ platforms, so you can run Cypher queries against an embedded graph database dire - `net10.0` (primary, AOT/trim friendly, source-generated `LibraryImport`) - `netstandard2.0` (broad reach, including .NET Framework) +## Installation + +The binding is split into a managed package and separate native packages, so an app only carries the +platform binaries it needs. Reference the managed package **plus** a native package. + +For an app that should run on any supported platform, add the native meta-package (it pulls in every +per-platform native package): + +```bash +dotnet add package LadybugDB +dotnet add package LadybugDB.Native +``` + +For a slim, single-platform app, reference just the native package for that platform instead of the +meta-package: + +```bash +dotnet add package LadybugDB +dotnet add package LadybugDB.Native.win-x64 +``` + +Available native packages: `LadybugDB.Native.win-x64`, `LadybugDB.Native.linux-x64`, +`LadybugDB.Native.linux-arm64`, `LadybugDB.Native.osx-x64`, `LadybugDB.Native.osx-arm64`. The +`LadybugDB.Native` meta-package depends on all of them. + ## Quick start ```csharp @@ -40,16 +65,30 @@ The binding needs the native Ladybug shared library at runtime (`lbug_shared.dll `liblbug.so` / `liblbug.dylib`). When the native library is not available, the native round-trip tests skip (the ABI/struct-layout guards still run). -## How the native library is obtained +## How the packages are built -This repo does not contain the engine source; the native library comes from the upstream -[Ladybug](https://github.com/ladybugdb/ladybug) engine: +This repo does not contain the engine source; the native libraries come from the upstream +[Ladybug](https://github.com/ladybugdb/ladybug) engine. Packaging is driven by a Cake Frosting build +project under [`cake/`](cake) (run it with the `build.ps1` / `build.sh` bootstrap): -- **CI / release** (`.github/workflows/release.yml`) downloads the prebuilt `liblbug-*` assets from - an upstream `LadybugDB/ladybug` GitHub Release (pinned via `ENGINE_VERSION`) and stages them into - `lib/runtimes//native/` before packing the multi-RID NuGet package. -- **Local development** is easiest when this repo is checked out as the `tools/csharp_api` submodule - inside the Ladybug monorepo: `scripts/build-native-and-test.ps1` then builds `lbug_shared` from the - parent engine tree, stages it into `lib/runtimes/win-x64/native/`, and runs the suite. +```bash +./build.sh --target Test # build + stage the host native + run the suite +./build.sh --target Pack # build the full package family into ./artifacts +``` -See `.agents/notes/HANDOFF.md` for details. +The package version tracks the upstream engine version (e.g. `0.17.0`), with an `-alpha.N` prerelease +suffix while the binding is in development. It is defined once in `version.txt` at the repo root; override +it with `--package-version ` (the release workflow uses the git tag), or `--prerelease ""` for a stable +build that matches the engine version. + +- **`Pack`** stages the prebuilt `liblbug-*` assets for every shipped RID (downloaded from an upstream + `LadybugDB/ladybug` GitHub Release, pinned to the engine version from `version.txt`; override with + `--engine-version`), + packs the managed `LadybugDB` package, one `LadybugDB.Native.` package per RID, and the + `LadybugDB.Native` meta-package, then verifies every package's contents. +- **CI / release** (`.github/workflows/`) invoke the same pipeline; the release workflow gates on the + linux-x64 suite against the real engine and publishes all packages to nuget.org via OIDC. +- **Local development** is easiest when this repo is checked out as the `tools/csharp_api` submodule + inside the Ladybug monorepo: `scripts/build-native-and-test.ps1` builds `lbug_shared` from the parent + engine tree, stages it into `lib/runtimes/win-x64/native/`, and runs the suite. With a native already + staged there, `--target Test` reuses it instead of downloading. diff --git a/build.ps1 b/build.ps1 new file mode 100644 index 0000000..ca55f47 --- /dev/null +++ b/build.ps1 @@ -0,0 +1,16 @@ +#!/usr/bin/env pwsh +# Bootstrap for the Cake Frosting packaging pipeline. Forwards all arguments to the build project, e.g. +# .\build.ps1 --target Test +# .\build.ps1 --target Pack # version + engine release come from version.txt; override with --package-version / --engine-version +[CmdletBinding()] +param( + [Parameter(ValueFromRemainingArguments = $true)] + [string[]]$Arguments +) + +$ErrorActionPreference = 'Stop' +$env:DOTNET_CLI_TELEMETRY_OPTOUT = '1' +$env:DOTNET_NOLOGO = '1' + +dotnet run --project "$PSScriptRoot/cake/LadybugDB.Build.csproj" -- $Arguments +exit $LASTEXITCODE diff --git a/build.sh b/build.sh new file mode 100644 index 0000000..cee50a8 --- /dev/null +++ b/build.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# Bootstrap for the Cake Frosting packaging pipeline. Forwards all arguments to the build project, e.g. +# ./build.sh --target Test +# ./build.sh --target Pack # version + engine release come from version.txt; override with --package-version / --engine-version +set -euo pipefail + +export DOTNET_CLI_TELEMETRY_OPTOUT=1 +export DOTNET_NOLOGO=1 + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +dotnet run --project "$script_dir/cake/LadybugDB.Build.csproj" -- "$@" diff --git a/cake/BuildContext.cs b/cake/BuildContext.cs new file mode 100644 index 0000000..5a03c28 --- /dev/null +++ b/cake/BuildContext.cs @@ -0,0 +1,250 @@ +using System.Formats.Tar; +using System.IO.Compression; +using System.Runtime.InteropServices; +using Cake.Common; +using Cake.Core; +using Cake.Core.Diagnostics; +using Cake.Core.IO; +using Cake.Frosting; +using Path = System.IO.Path; + +namespace LadybugDB.Build; + +/// +/// Shared state and paths for the packaging pipeline. Resolves the binding root from the running +/// assembly (independent of the caller's working directory) and stages prebuilt engine libraries. +/// +public sealed class BuildContext : FrostingContext +{ + /// RID -> (release asset name, canonical native library file name). + public static readonly IReadOnlyDictionary NativeAssets = + new Dictionary(StringComparer.Ordinal) + { + ["win-x64"] = ("liblbug-windows-x86_64.zip", "lbug_shared.dll"), + ["linux-x64"] = ("liblbug-linux-x86_64.tar.gz", "liblbug.so"), + ["linux-arm64"] = ("liblbug-linux-aarch64.tar.gz", "liblbug.so"), + ["osx-x64"] = ("liblbug-osx-x86_64.tar.gz", "liblbug.dylib"), + ["osx-arm64"] = ("liblbug-osx-arm64.tar.gz", "liblbug.dylib"), + }; + + public BuildContext(ICakeContext context) : base(context) + { + BuildConfiguration = context.Argument("configuration", "Release"); + Commit = context.Argument("commit", Environment.GetEnvironmentVariable("GITHUB_SHA") ?? string.Empty); + + Root = FindBindingRoot(); + + // version.txt at the binding root is the single source of truth for the version (e.g. + // "0.17.0-alpha.1"): its base drives both the engine release we pull natives from and the + // package version's base, and its prerelease suffix (alpha.1, alpha.2, ...) is the dev suffix. + // Bump it there - no code change. Overrides: --engine-version / ENGINE_VERSION pick a different + // engine release; --prerelease "" cuts a stable build; --package-version sets the version + // verbatim (the release workflow passes the git tag). 'version' is reserved by the Cake host. + (string baseVersion, string filePrerelease) = ReadVersion(Root); + EngineVersion = context.Argument("engine-version", + Environment.GetEnvironmentVariable("ENGINE_VERSION") ?? $"v{baseVersion}"); + string prerelease = context.Argument("prerelease", filePrerelease); + string engineBase = EngineVersion.TrimStart('v', 'V'); + Version = context.HasArgument("package-version") + ? context.Argument("package-version") + : prerelease.Length == 0 ? engineBase : $"{engineBase}-{prerelease}"; + + ManagedProject = Path.Combine(Root, "src", "LadybugDB", "LadybugDB.csproj"); + TestProject = Path.Combine(Root, "test", "LadybugDB.Tests", "LadybugDB.Tests.csproj"); + Solution = Path.Combine(Root, "LadybugDB.slnx"); + NativeDir = Path.Combine(Root, "cake", "native"); + RuntimeProject = Path.Combine(NativeDir, "LadybugDB.Native.Runtime.csproj"); + MetaProject = Path.Combine(NativeDir, "LadybugDB.Native.Meta.csproj"); + MetaNuspecTemplate = Path.Combine(NativeDir, "LadybugDB.Native.nuspec"); + RuntimesStageDir = Path.Combine(Root, "lib", "runtimes"); + ArtifactsDir = Path.Combine(Root, "artifacts"); + DownloadDir = Path.Combine(Root, "download"); + } + + public string BuildConfiguration { get; } + public string Version { get; } + public string Commit { get; } + public string EngineVersion { get; } + + public string Root { get; } + public string ManagedProject { get; } + public string TestProject { get; } + public string Solution { get; } + public string NativeDir { get; } + public string RuntimeProject { get; } + public string MetaProject { get; } + public string MetaNuspecTemplate { get; } + public string RuntimesStageDir { get; } + public string ArtifactsDir { get; } + public string DownloadDir { get; } + + public IReadOnlyList AllRids { get; } = [.. NativeAssets.Keys]; + + public string HostRid { get; } = + (OperatingSystem.IsWindows() ? "win" : OperatingSystem.IsMacOS() ? "osx" : "linux") + + "-" + + (RuntimeInformation.OSArchitecture == Architecture.Arm64 ? "arm64" : "x64"); + + /// Absolute path the native library for is staged to. + public string StagedLibraryPath(string rid) => + Path.Combine(RuntimesStageDir, rid, "native", NativeAssets[rid].Library); + + /// + /// Ensure the prebuilt engine library for sits under + /// lib/runtimes/{rid}/native. No-op when it is already present (e.g. built from source locally); + /// otherwise downloads the pinned engine release asset and extracts the canonical library out of it. + /// + public void EnsureNativeStaged(string rid) + { + if (!NativeAssets.TryGetValue(rid, out (string Asset, string Library) info)) + { + throw new CakeException($"Unknown RID '{rid}'. Known: {string.Join(", ", AllRids)}."); + } + + string dest = StagedLibraryPath(rid); + if (File.Exists(dest)) + { + Log.Information($"native already staged for {rid}: {dest}"); + return; + } + + string asset = Path.Combine(DownloadDir, info.Asset); + if (!File.Exists(asset)) + { + DownloadEngineAsset(info.Asset); + } + + Directory.CreateDirectory(Path.GetDirectoryName(dest)!); + if (info.Asset.EndsWith(".zip", StringComparison.OrdinalIgnoreCase)) + { + ExtractFromZip(asset, info.Library, dest); + } + else + { + ExtractFromTarGz(asset, dest); + } + + Log.Information($"staged {rid} -> {dest}"); + } + + private void DownloadEngineAsset(string asset) + { + Directory.CreateDirectory(DownloadDir); + Log.Information($"downloading {asset} from LadybugDB/ladybug@{EngineVersion}"); + + var args = new ProcessArgumentBuilder() + .Append("release").Append("download").AppendQuoted(EngineVersion) + .Append("--repo").Append("LadybugDB/ladybug") + .Append("--dir").AppendQuoted(DownloadDir) + .Append("--pattern").AppendQuoted(asset); + + int exit; + try + { + exit = this.StartProcess("gh", new ProcessSettings { Arguments = args }); + } + catch (Exception ex) + { + throw new CakeException( + $"Could not run 'gh' to download '{asset}'. Install the GitHub CLI and authenticate, " + + $"or stage the native library under lib/runtimes//native/ manually. ({ex.Message})"); + } + + if (exit != 0 || !File.Exists(Path.Combine(DownloadDir, asset))) + { + throw new CakeException($"gh release download failed for '{asset}' (exit {exit})."); + } + } + + private static void ExtractFromZip(string archive, string library, string dest) + { + using ZipArchive zip = ZipFile.OpenRead(archive); + ZipArchiveEntry entry = zip.Entries.FirstOrDefault(e => + string.Equals(Path.GetFileName(e.FullName), library, + StringComparison.OrdinalIgnoreCase)) + ?? throw new CakeException($"'{library}' not found in '{Path.GetFileName(archive)}'."); + entry.ExtractToFile(dest, overwrite: true); + } + + // The Linux/macOS tarballs ship a symlink chain (liblbug.so -> .so.0 -> .so.0.x.y). NuGet does not + // preserve symlinks and the binding loads by path, so copy the real shared object out under the + // canonical name: the largest regular file in the "liblbug" family with the platform extension. + private static void ExtractFromTarGz(string archive, string dest) + { + string ext = dest.EndsWith(".dylib", StringComparison.OrdinalIgnoreCase) ? ".dylib" : ".so"; + long bestLength = -1; + byte[]? best = null; + + using FileStream fs = File.OpenRead(archive); + using GZipStream gz = new(fs, CompressionMode.Decompress); + using TarReader tar = new(gz); + while (tar.GetNextEntry() is { } entry) + { + if (entry.EntryType is not (TarEntryType.RegularFile or TarEntryType.V7RegularFile) || + entry.DataStream is null) + { + continue; + } + + string name = Path.GetFileName(entry.Name); + if (!name.StartsWith("liblbug", StringComparison.OrdinalIgnoreCase) || + !name.Contains(ext, StringComparison.OrdinalIgnoreCase)) + { + continue; + } + + using MemoryStream ms = new(); + entry.DataStream.CopyTo(ms); + if (ms.Length > bestLength) + { + bestLength = ms.Length; + best = ms.ToArray(); + } + } + + if (best is null) + { + throw new CakeException($"No 'liblbug*{ext}' regular file found in '{Path.GetFileName(archive)}'."); + } + + File.WriteAllBytes(dest, best); + } + + /// + /// Reads the binding's version from version.txt at the root - the single source of truth for + /// the package version (e.g. "0.17.0-alpha.1"). Returns the base ("0.17.0") and the prerelease + /// suffix ("alpha.1", or "" when stable). Bump it there to advance the alpha or the engine; no code change. + /// + private static (string Base, string Prerelease) ReadVersion(string root) + { + string path = Path.Combine(root, "version.txt"); + if (!File.Exists(path)) + { + throw new CakeException( + $"Version file not found at '{path}'. " + + $"It is the single source of truth for the package version (e.g. 0.17.0-alpha.1)."); + } + + string raw = File.ReadAllText(path).Trim(); + int dash = raw.IndexOf('-'); + return dash < 0 ? (raw, string.Empty) : (raw[..dash], raw[(dash + 1)..]); + } + + private static string FindBindingRoot() + { + for (DirectoryInfo? dir = new(AppContext.BaseDirectory); dir is not null; dir = dir.Parent) + { + if (File.Exists(Path.Combine(dir.FullName, "LadybugDB.slnx"))) + { + return dir.FullName; + } + } + + throw new CakeException("Could not locate the binding root (no LadybugDB.slnx found above the build output)."); + } + + /// Console entry point. Cake resolves the task graph and constructs this context via DI. + public static int Main(string[] args) => new CakeHost() + .UseContext() + .Run(args); +} diff --git a/cake/LadybugDB.Build.csproj b/cake/LadybugDB.Build.csproj new file mode 100644 index 0000000..d1dbcec --- /dev/null +++ b/cake/LadybugDB.Build.csproj @@ -0,0 +1,25 @@ + + + + Exe + net10.0 + LadybugDB.Build + false + + false + enable + enable + + + + + + + + + + + + + diff --git a/cake/LadybugDB.Build.slnx b/cake/LadybugDB.Build.slnx new file mode 100644 index 0000000..c9237ed --- /dev/null +++ b/cake/LadybugDB.Build.slnx @@ -0,0 +1,7 @@ + + + + + + + diff --git a/cake/Tasks/Pipeline.cs b/cake/Tasks/Pipeline.cs new file mode 100644 index 0000000..197d085 --- /dev/null +++ b/cake/Tasks/Pipeline.cs @@ -0,0 +1,190 @@ +using Cake.Common.IO; +using Cake.Common.Tools.DotNet; +using Cake.Common.Tools.DotNet.Build; +using Cake.Common.Tools.DotNet.MSBuild; +using Cake.Common.Tools.DotNet.Pack; +using Cake.Common.Tools.DotNet.Test; +using Cake.Core.Diagnostics; +using Cake.Frosting; + +namespace LadybugDB.Build.Tasks; + +// The packaging pipeline, in run order. Cake Frosting discovers tasks by attribute, so these small +// tasks live together here; the only one big enough to warrant its own file is VerifyPackages. + +[TaskName("Clean")] +public sealed class CleanTask : FrostingTask +{ + public override void Run(BuildContext context) + { + // The build project's own bin/obj are intentionally left alone (it is running from there). + string[] areas = ["src", "test", Path.Combine("cake", "native")]; + foreach (string area in areas) + { + context.CleanDirectories($"{context.Root}/{area.Replace('\\', '/')}/**/bin"); + context.CleanDirectories($"{context.Root}/{area.Replace('\\', '/')}/**/obj"); + } + + context.EnsureDirectoryExists(context.ArtifactsDir); + context.CleanDirectory(context.ArtifactsDir); + } +} + +[TaskName("Restore")] +public sealed class RestoreTask : FrostingTask +{ + public override void Run(BuildContext context) => context.DotNetRestore(context.Solution); +} + +[TaskName("BuildManaged")] +[IsDependentOn(typeof(RestoreTask))] +public sealed class BuildManagedTask : FrostingTask +{ + public override void Run(BuildContext context) => + context.DotNetBuild(context.ManagedProject, new DotNetBuildSettings + { + Configuration = context.BuildConfiguration, + NoRestore = true, + }); +} + +/// +/// Stages the prebuilt engine library for the host RID so the suite can exercise the native +/// round-trips. Skips quietly when the host platform has no shipped RID. +/// +[TaskName("FetchNatives")] +public sealed class FetchNativesTask : FrostingTask +{ + public override void Run(BuildContext context) + { + if (BuildContext.NativeAssets.ContainsKey(context.HostRid)) + { + context.EnsureNativeStaged(context.HostRid); + } + else + { + context.Log.Warning($"no prebuilt native shipped for host RID '{context.HostRid}'; native round-trips will skip"); + } + } +} + +[TaskName("Test")] +[IsDependentOn(typeof(BuildManagedTask))] +[IsDependentOn(typeof(FetchNativesTask))] +public sealed class TestTask : FrostingTask +{ + public override void Run(BuildContext context) + { + DotNetTestSettings settings = new() { Configuration = context.BuildConfiguration }; + + // When the host native is present, fail (don't skip) if it doesn't actually load. + if (BuildContext.NativeAssets.ContainsKey(context.HostRid) && + File.Exists(context.StagedLibraryPath(context.HostRid))) + { + settings.EnvironmentVariables = new Dictionary { ["LADYBUG_REQUIRE_NATIVE"] = "1" }; + } + + context.DotNetTest(context.TestProject, settings); + } +} + +/// Default target: build the binding and run the suite. +[TaskName("Default")] +[IsDependentOn(typeof(TestTask))] +public sealed class DefaultTask : FrostingTask; + +/// Packs the managed-only LadybugDB package (no native payload). +[TaskName("PackManaged")] +[IsDependentOn(typeof(BuildManagedTask))] +public sealed class PackManagedTask : FrostingTask +{ + public override void Run(BuildContext context) + { + var msbuild = new DotNetMSBuildSettings() + .WithProperty("Version", context.Version) + .WithProperty("ContinuousIntegrationBuild", "true"); + + if (!string.IsNullOrEmpty(context.Commit)) + { + msbuild.WithProperty("RepositoryCommit", context.Commit); + } + + context.DotNetPack(context.ManagedProject, new DotNetPackSettings + { + Configuration = context.BuildConfiguration, + OutputDirectory = context.ArtifactsDir, + MSBuildSettings = msbuild, + }); + } +} + +/// +/// Stages and packs one native package per shipped RID (LadybugDB.Native.<rid>) from the single +/// runtime packaging template. +/// +[TaskName("PackRuntimes")] +public sealed class PackRuntimesTask : FrostingTask +{ + public override void Run(BuildContext context) + { + foreach (string rid in context.AllRids) + { + context.EnsureNativeStaged(rid); + + var msbuild = new DotNetMSBuildSettings() + .WithProperty("Version", context.Version) + .WithProperty("NativeRid", rid) + .WithProperty("PackageId", $"LadybugDB.Native.{rid}"); + + if (!string.IsNullOrEmpty(context.Commit)) + { + msbuild.WithProperty("RepositoryCommit", context.Commit); + } + + context.DotNetPack(context.RuntimeProject, new DotNetPackSettings + { + Configuration = context.BuildConfiguration, + OutputDirectory = context.ArtifactsDir, + MSBuildSettings = msbuild, + }); + } + } +} + +/// +/// Packs the LadybugDB.Native meta-package: no payload, depends on every per-RID native package. +/// The committed nuspec is a template; version/commit tokens are substituted here so we never pass +/// semicolon-laden NuspecProperties through MSBuild. +/// +[TaskName("PackNativeMeta")] +[IsDependentOn(typeof(PackRuntimesTask))] +public sealed class PackNativeMetaTask : FrostingTask +{ + public override void Run(BuildContext context) + { + string nuspec = File.ReadAllText(context.MetaNuspecTemplate) + .Replace("$version$", context.Version) + .Replace("$commit$", context.Commit); + + string generatedDir = Path.Combine(context.NativeDir, "obj"); + Directory.CreateDirectory(generatedDir); + string generated = Path.Combine(generatedDir, "LadybugDB.Native.generated.nuspec"); + File.WriteAllText(generated, nuspec); + + var msbuild = new DotNetMSBuildSettings() + .WithProperty("NuspecFile", generated) + .WithProperty("NuspecBasePath", context.NativeDir); + + context.DotNetPack(context.MetaProject, new DotNetPackSettings + { + Configuration = context.BuildConfiguration, + OutputDirectory = context.ArtifactsDir, + MSBuildSettings = msbuild, + }); + } +} + +/// Aggregate: produce and validate the full package family (managed + per-RID native + meta). +[TaskName("Pack")] +[IsDependentOn(typeof(VerifyPackagesTask))] +public sealed class PackTask : FrostingTask; diff --git a/cake/Tasks/VerifyPackagesTask.cs b/cake/Tasks/VerifyPackagesTask.cs new file mode 100644 index 0000000..f3915a8 --- /dev/null +++ b/cake/Tasks/VerifyPackagesTask.cs @@ -0,0 +1,124 @@ +using System.IO.Compression; +using System.Xml.Linq; +using Cake.Core; +using Cake.Core.Diagnostics; +using Cake.Frosting; + +namespace LadybugDB.Build.Tasks; + +/// +/// Guards against a silent packaging regression: asserts the managed assemblies, every per-RID native +/// payload, and the meta-package's dependency set are all present in the produced .nupkg files. +/// +[TaskName("VerifyPackages")] +[IsDependentOn(typeof(PackManagedTask))] +[IsDependentOn(typeof(PackRuntimesTask))] +[IsDependentOn(typeof(PackNativeMetaTask))] +public sealed class VerifyPackagesTask : FrostingTask +{ + public override void Run(BuildContext context) + { + IReadOnlyDictionary packages = ReadPackages(context.ArtifactsDir); + List errors = []; + + Require(packages, "LadybugDB", errors, p => + { + RequireFile(p, "lib/net10.0/LadybugDB.dll", errors); + RequireFile(p, "lib/netstandard2.0/LadybugDB.dll", errors); + }); + + foreach (string rid in context.AllRids) + { + string library = BuildContext.NativeAssets[rid].Library; + Require(packages, $"LadybugDB.Native.{rid}", errors, + p => RequireFile(p, $"runtimes/{rid}/native/{library}", errors)); + } + + Require(packages, "LadybugDB.Native", errors, p => + { + foreach (string rid in context.AllRids) + { + var dep = $"LadybugDB.Native.{rid}"; + if (!p.Dependencies.Contains(dep, StringComparer.OrdinalIgnoreCase)) + { + errors.Add($"meta package 'LadybugDB.Native' is missing dependency '{dep}'"); + } + } + }); + + if (errors.Count > 0) + { + throw new CakeException("Package validation failed:" + Environment.NewLine + + string.Join(Environment.NewLine, errors.Select(e => " - " + e))); + } + + context.Log.Information($"verified {packages.Count} package(s) in {context.ArtifactsDir}"); + } + + private static void Require( + IReadOnlyDictionary packages, string id, List errors, Action check) + { + if (packages.TryGetValue(id, out Package? package)) + { + check(package); + } + else + { + errors.Add($"expected package '{id}' was not produced"); + } + } + + private static void RequireFile(Package package, string path, List errors) + { + if (!package.Files.Contains(path, StringComparer.OrdinalIgnoreCase)) + { + errors.Add($"package '{package.Id}' is missing entry '{path}'"); + } + } + + private static IReadOnlyDictionary ReadPackages(string artifactsDir) + { + var result = new Dictionary(StringComparer.OrdinalIgnoreCase); + if (!Directory.Exists(artifactsDir)) + { + return result; + } + + foreach (string path in Directory.GetFiles(artifactsDir, "*.nupkg")) + { + if (path.EndsWith(".snupkg", StringComparison.OrdinalIgnoreCase)) + { + continue; + } + + using ZipArchive archive = ZipFile.OpenRead(path); + HashSet files = archive.Entries.Select(e => e.FullName).ToHashSet(StringComparer.OrdinalIgnoreCase); + + ZipArchiveEntry? nuspecEntry = archive.Entries.FirstOrDefault( + e => e.FullName.EndsWith(".nuspec", StringComparison.OrdinalIgnoreCase) && !e.FullName.Contains('/')); + if (nuspecEntry is null) + { + continue; + } + + using Stream stream = nuspecEntry.Open(); + var doc = XDocument.Load(stream); + string id = doc.Descendants().FirstOrDefault(e => e.Name.LocalName == "id")?.Value?.Trim() ?? string.Empty; + HashSet deps = doc.Descendants() + .Where(e => e.Name.LocalName == "dependency") + .Select(e => (string?)e.Attribute("id")) + .Where(v => !string.IsNullOrEmpty(v)) + .Select(v => v!) + .ToHashSet(StringComparer.OrdinalIgnoreCase); + + if (id.Length > 0) + { + result[id] = new Package(id, files, deps); + } + } + + return result; + } + + private sealed record Package(string Id, HashSet Files, HashSet Dependencies); +} diff --git a/cake/common.props b/cake/common.props new file mode 100644 index 0000000..8953da0 --- /dev/null +++ b/cake/common.props @@ -0,0 +1,17 @@ + + + + + LadybugDB + LadybugDB + Ladybug + Copyright (c) LadybugDB + MIT + https://github.com/ladybugdb/ladybug + https://github.com/LadybugDB/ladybug-dotnet + git + true + graph;graph-database;database;cypher;embedded;ladybug;native + + + diff --git a/cake/native/LadybugDB.Native.Meta.csproj b/cake/native/LadybugDB.Native.Meta.csproj new file mode 100644 index 0000000..9c0dc3c --- /dev/null +++ b/cake/native/LadybugDB.Native.Meta.csproj @@ -0,0 +1,17 @@ + + + + + netstandard2.0 + false + false + false + true + $(NoWarn);NU5128 + + + diff --git a/cake/native/LadybugDB.Native.Runtime.csproj b/cake/native/LadybugDB.Native.Runtime.csproj new file mode 100644 index 0000000..6a1346d --- /dev/null +++ b/cake/native/LadybugDB.Native.Runtime.csproj @@ -0,0 +1,39 @@ + + + + + + + netstandard2.0 + false + false + true + false + true + + win-x64 + LadybugDB.Native.$(NativeRid) + LadybugDB native engine ($(NativeRid)) + Native Ladybug engine library for $(NativeRid). Companion runtime payload for the managed LadybugDB package; reference it (or the LadybugDB.Native meta-package) alongside LadybugDB. + + + $(NoWarn);NU5128 + + + + + true + runtimes\$(NativeRid)\native\%(RecursiveDir)%(Filename)%(Extension) + + + true + lib\netstandard2.0\_._ + + + + diff --git a/cake/native/LadybugDB.Native.nuspec b/cake/native/LadybugDB.Native.nuspec new file mode 100644 index 0000000..f5e8da4 --- /dev/null +++ b/cake/native/LadybugDB.Native.nuspec @@ -0,0 +1,40 @@ + + + + + + LadybugDB.Native + LadybugDB native engine (all platforms) + $version$ + Native Ladybug engine libraries for all supported platforms. Companion to the managed LadybugDB package: reference both LadybugDB and LadybugDB.Native to run on any supported platform, or reference a single LadybugDB.Native.<rid> package for a slim, platform-specific app. + Native Ladybug engine binaries for all supported platforms. + https://github.com/ladybugdb/ladybug + + graph graph-database database cypher embedded ladybug native + + + MIT + LadybugDB + LadybugDB + false + Copyright (c) LadybugDB + + + + + + + + + + + + + + + + + + + + diff --git a/cake/native/_._ b/cake/native/_._ new file mode 100644 index 0000000..e69de29 diff --git a/nuget/nuget-package.props b/nuget/nuget-package.props index 977c761..19c6f22 100644 --- a/nuget/nuget-package.props +++ b/nuget/nuget-package.props @@ -2,7 +2,9 @@ - 0.0.1-alpha + + $([System.IO.File]::ReadAllText('$(CSharpDir)version.txt').Trim()) LadybugDB LadybugDB Ladybug @@ -22,8 +24,12 @@ - - + + diff --git a/version.txt b/version.txt new file mode 100644 index 0000000..22a956e --- /dev/null +++ b/version.txt @@ -0,0 +1 @@ +0.17.0-alpha.1 From 0d108313b50f27d0aa5a026b84eb6134bc6fa3b1 Mon Sep 17 00:00:00 2001 From: Sergey Vershinin Date: Sat, 30 May 2026 23:13:56 +0200 Subject: [PATCH 2/3] Delete .agents/notes/PR.md --- .agents/notes/PR.md | 110 -------------------------------------------- 1 file changed, 110 deletions(-) delete mode 100644 .agents/notes/PR.md diff --git a/.agents/notes/PR.md b/.agents/notes/PR.md deleted file mode 100644 index 1bfbdbc..0000000 --- a/.agents/notes/PR.md +++ /dev/null @@ -1,110 +0,0 @@ -# Split native packaging into a package family + Cake Frosting build - -> Living PR description — keep this current as the branch evolves. The checklist at the bottom is the -> source of truth for what's left before merge / first publish. - -## Summary - -Restructures the LadybugDB .NET binding's packaging so the native engine no longer ships inside the -managed package. Instead it's a **package family**: - -- `LadybugDB` — managed-only (no native payload). -- `LadybugDB.Native.` — one package per platform, carrying just that RID's native library. -- `LadybugDB.Native` — meta-package that depends on all per-RID packages. - -Consumers reference `LadybugDB` **plus** a native package: the meta-package for "runs anywhere", or a -single `LadybugDB.Native.` for a slim, single-platform app. All packaging is driven by a new -**Cake Frosting** build project under `cake/`. - -## Motivation - -The old single package bundled every platform's native library into one `.nupkg`, so every consumer -carried all of them regardless of target. Splitting the natives into per-RID packages (with a meta -"all platforms" package) lets an app pull only what it needs, while keeping a one-line install for the -common case. - -## What changed - -### Package family - -- Managed `LadybugDB` is now managed-only: removed the `runtimes/`** native glob from -`nuget/nuget-package.props` (it now ships `lib/net10.0` + `lib/netstandard2.0` + README + symbols only). -- Each `LadybugDB.Native.` carries `runtimes//native/`* plus an empty `lib/netstandard2.0/_._` -marker (so NuGet adds no compile reference) and declares no dependencies. -- `LadybugDB.Native` meta-package depends on all five per-RID packages. -- Shipped RIDs: `win-x64`, `linux-x64`, `linux-arm64`, `osx-x64`, `osx-arm64`. -- The managed package intentionally has **no** dependency on any native package — that decoupling is -what makes the slim, single-RID install possible. - -### Build pipeline (`cake/`) - -- A Cake Frosting console app: `LadybugDB.Build.csproj` + `BuildContext.cs` (which also hosts the `Main` -entry point), and the pipeline tasks under `Tasks/` (`Pipeline.cs` for the small tasks + -`VerifyPackagesTask.cs`). Bootstrap with `build.ps1` / `build.sh`. -- Tasks: `Clean`, `Restore`, `BuildManaged`, `Test`, `FetchNatives`, `PackManaged`, `PackRuntimes`, -`PackNativeMeta`, `VerifyPackages`, `Pack`, `Default`. -- `BuildContext` is the single source of truth for the RID set and the RID -> (release asset, library) -mapping. `EnsureNativeStaged` reuses a locally-built native if present, otherwise downloads the pinned -engine release asset (via `gh`) and extracts the canonical library (pulling the real shared object out -of the `.so`/`.dylib` symlink chain, since NuGet doesn't preserve symlinks). -- Per-RID packages come from one parameterized template (`cake/native/LadybugDB.Native.Runtime.csproj`, -packed once per RID). The meta-package is a nuspec template (`cake/native/LadybugDB.Native.nuspec`) -with version/commit substituted at pack time. All packing is `dotnet`-only (no nuget.exe / mono). -- `VerifyPackages` asserts the managed assemblies, every per-RID payload, and the meta's dependency set, -so a silent packaging regression fails the build instead of shipping a broken package. - -### Versioning - -- The package version tracks the upstream engine version, with an `-alpha.N` prerelease suffix while in -development. Current default: **`0.17.0-alpha.1`**. -- It lives in ONE place - `version.txt` at the binding root - read by `BuildContext`, the managed -`nuget-package.props`, and both CI workflows. Bump the alpha (or the engine) there with no code change. -`--prerelease ""` cuts a stable build equal to the engine version; `--package-version ` overrides -exactly (the release workflow passes the `v*` git tag). - -### CI / release - -- `ci.yml` and `release.yml` now invoke the pipeline (`dotnet run --project cake/... -- --target ...`) -instead of inline shell. `ci.yml` runs a `test` matrix (linux-x64 + win-x64) and a `pack` smoke; -`release.yml` runs the linux-x64 test gate, packs + verifies, then publishes all packages via OIDC. - -### Docs - -- README (install + build sections), `AGENTS.md` (Packaging). - -## Consumption - -```bash -# runs on any supported platform -dotnet add package LadybugDB -dotnet add package LadybugDB.Native - -# slim, single-platform -dotnet add package LadybugDB -dotnet add package LadybugDB.Native.win-x64 -``` - -## Validation - -- Local end-to-end (win-x64 host, engine `v0.17.0`): `--target Pack` produced and verified all 7 -packages as `0.17.0-alpha.1` (correct layouts; meta pins all five per-RID deps; no stray references); -`--target Test` passed 28/28 with the native loaded (no skips). -- Build is warning- and lint-clean. - -## Next steps / TODO - -Maintainer actions before first publish: - -- **Owner — still TODO (one-time):** create the `release` GitHub environment on -`LadybugDB/ladybug-dotnet` and add secret `NUGET_USER` = the nuget.org **profile name** (NOT email). -The `publish` job declares `environment: release` and `NuGet/login@v1` reads `secrets.NUGET_USER`, so -without these the publish step fails with a 403/auth error. -- First green CI run: trigger `release.yml` via `workflow_dispatch` to confirm the linux-x64 gate + -all-RID download/pack on CI (proven locally on win-x64 so far). - -Follow-ups (can land later): - -- Bump the `-alpha.N` suffix as the binding stabilizes; drop it for the first stable `v0.17.0`. -- Phase 3 (remaining): expand the suite to mirror the Java + C API tests over `dataset/tinysnb`. -- Phase 5 (optional): Native AOT validation, Arrow C Data interface, observability. - From b9802bda2626a1ccb1e602828ddc742f7724e65b Mon Sep 17 00:00:00 2001 From: Sergey Vershinin Date: Sat, 30 May 2026 23:15:42 +0200 Subject: [PATCH 3/3] ci: bump GitHub Actions to Node 24 majors Node 20 JS actions are being force-upgraded on GitHub runners (June 16, 2026). Bump to current Node 24 majors instead of the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 escape hatch: checkout v4->v6, setup-dotnet v4->v5, upload-artifact v4->v7, download-artifact v4->v8. NuGet/login@v1 already runs on Node 24 (v1.2.0), so it is unchanged. --- .github/workflows/ci.yml | 10 +++++----- .github/workflows/release.yml | 10 +++++----- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 53fc5e0..d768daf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,10 +47,10 @@ jobs: - os: windows-latest rid: win-x64 steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Setup .NET - uses: actions/setup-dotnet@v4 + uses: actions/setup-dotnet@v5 with: dotnet-version: '10.0.x' @@ -63,10 +63,10 @@ jobs: pack: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Setup .NET - uses: actions/setup-dotnet@v4 + uses: actions/setup-dotnet@v5 with: dotnet-version: '10.0.x' @@ -75,7 +75,7 @@ jobs: GH_TOKEN: ${{ github.token }} run: dotnet run --project cake/LadybugDB.Build.csproj -- --target Pack - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@v7 with: name: nuget-packages path: | diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d361e3a..9e1757c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -42,10 +42,10 @@ jobs: pack: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Setup .NET - uses: actions/setup-dotnet@v4 + uses: actions/setup-dotnet@v5 with: dotnet-version: '10.0.x' @@ -92,7 +92,7 @@ jobs: --package-version "${{ steps.ver.outputs.version }}" --engine-version "${{ steps.engine.outputs.engine }}" - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@v7 with: name: nuget-packages path: | @@ -111,12 +111,12 @@ jobs: contents: read steps: - name: Setup .NET - uses: actions/setup-dotnet@v4 + uses: actions/setup-dotnet@v5 with: dotnet-version: '10.0.x' - name: Download packed artifacts - uses: actions/download-artifact@v4 + uses: actions/download-artifact@v8 with: name: nuget-packages path: artifacts