Sync upstream v1.5.54 - #354
Closed
camphillips22 wants to merge 4 commits into
Closed
Conversation
Fix compilation and linking on macOS (Apple Silicon, macOS 15+) so that all 784 Bazel targets build successfully. The upstream project only supports Linux; these changes address platform differences in vendored dependencies, the Bazel 6.x toolchain, and GNU ld vs Apple ld. - Add tools/bazel wrapper to fix Bazel 6.x wrapped_clang/libtool_check_unique missing LC_UUID on macOS 15+ - Suppress Clang 17 -Wmissing-template-arg-list-after-template-kw - Patch zlib TARGET_OS_MAC misidentification and riegeli missing <unistd.h>; move zlib http_archive before protobuf_deps() for precedence - Guard --version-script / --undefined-version behind select() in 26 spanner_pg shared library BUILD files - Replace -Wl,-E with -Wl,-export_dynamic, -rpath=$ORIGIN with -rpath,@loader_path, and guard -static-libgcc/-lrt/-ldl with select() - Add macOS platform config to pg_config.h (strlcpy/strlcat, xlocale, epoll, sync_file_range, strerror_r, libedit) - Fix duplicate GetRunfilesDir ODR violation, long-to-Value ambiguity on ARM, and stale google-internal gtest include --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
Adds CircleCI CI/CD pipeline for the Cloud Spanner Emulator with Linux
and macOS support.
### CI Pipeline (`build-and-test` workflow)
- **Build + test** on both Linux (`xlarge`) and macOS (`m4pro.medium`,
Xcode 26.4.0)
- Uses `fastbuild` mode (no optimization) for faster compile times —
tests are mostly I/O and sleep-bound, not CPU-bound
- Installs `gcloud` CLI in all jobs and sets `GCLOUD_DIR` for the
Python-based gcloud integration tests
- Bazel disk cache saved/restored between runs per platform
- Test results collected and stored as CircleCI artifacts
### Release Pipeline (`release` workflow)
- Triggered by `v*.*.*-fs*` tags
- Builds optimized (`-c opt`) binaries for both platforms
- Packages `emulator_main` + `gateway_main` into platform-specific
tarballs
- Uploads to GCS via authenticated `gcloud` (using `OPS_GCLOUD_CREDS`)
### Test fixes
- **`instance_admin_test.py`**: Relaxed timestamp regex from `[0-9]{9}`
to `[0-9]+` — the emulator returns microsecond precision (6 digits) via
`absl::Time`, not nanosecond (9 digits)
- **`pg_functions_test.cc`**: Skipped `to_jsonb('-15e1500')` assertion
on macOS — nlohmann JSON's `long double` overflows at different exponent
limits on macOS vs Linux
- **`change_streams_test`** and **`schema_updater_test`**: Increased
test size to `large` (900s timeout) since fastbuild makes them slower
- Excluded `third_party/...` from CI build/test — several spanner_pg
tests have platform-specific failures (JSON `long double` overflow,
PostgreSQL enum/nodetag mismatches on macOS)
## Test plan
- [x] All gcloud integration tests pass locally with `GCLOUD_DIR` set
- [x] Conformance tests pass on macOS with `__APPLE__` guard
- [x] `change_streams_test` passes within `large` timeout
- [x] macOS CI pipeline passes (pipeline GoogleCloudPlatform#6 job#22, job#23)
- [x] Linux CI pipeline — gcloud and conformance tests pass
- [ ] Verify full green on next pipeline run (pipeline GoogleCloudPlatform#7)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary - Adds a persistence layer that allows the Cloud Spanner emulator to save and restore state across restarts via a `--data_dir` flag - Implements WAL (write-ahead log) for recording mutations, periodic snapshots for full state serialization, and a snapshot loader for restoring state on startup - Includes a `PersistenceManager` that coordinates background snapshot writing and WAL rotation - Adds protobuf schema (`persistence.proto`) for serializing all emulator state: instances, databases, schemas, storage, and keys - Adds `--help` usage message documenting emulator flags ## Components - **`persistence.proto`** — Protobuf definitions for serialized emulator state - **`value_serializer`** — Serializes/deserializes ZetaSQL values and keys to/from proto - **`wal_writer`** — Append-only write-ahead log with CRC32C integrity checks - **`persistent_storage`** — Storage implementation that wraps in-memory storage with WAL recording - **`snapshot_writer`** — Full state snapshot serialization (instances, databases, schemas, data) - **`snapshot_loader`** — Restores emulator state from snapshots + WAL replay - **`persistence_manager`** — Orchestrates background snapshots, WAL rotation, and shutdown ## Test plan - [x] Unit tests for value serializer (`value_serializer_test`) - [x] Unit tests for WAL writer (`wal_writer_test`) - [x] Unit tests for persistent storage (`persistent_storage_test`) - [ ] Integration testing with emulator binary using `--data_dir` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merge tag v1.5.54 from GoogleCloudPlatform/cloud-spanner-emulator into
fork.
Conflict resolutions:
- .bazelrc: take upstream rewrite (C++20, perf tuning, warning suppression),
preserve fork's macOS platform-specific config flag and
-Wno-missing-template-arg-list-after-template-kw.
- WORKSPACE: accept upstream delete (migrated to bzlmod via MODULE.bazel).
Fork's macOS zlib/riegeli WORKSPACE patches dropped; revisit if macOS
build breaks.
- binaries/emulator_main.cc: take upstream zetasql -> googlesql rename,
keep absl/flags/usage.h.
- third_party/spanner_pg/src/backend/utils/cache/lsyscache_test.cc:
take upstream rename + extra gtest include.
Mechanical rename in fork-only persistence files (backend/storage/*,
frontend/server/persistence_manager*): zetasql:: -> googlesql::,
zetasql/{base,public} -> googlesql/{base,public}, @com_google_zetasql ->
@com_google_googlesql to match the upstream-wide rename.
Build not yet validated on the merged tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v1.5.54from GoogleCloudPlatform/cloud-spanner-emulator.zetasql→googlesqlto match upstream-wide rename.Conflict resolutions
.bazelrc: upstream rewrite + preserved macOS-specific config.WORKSPACE: deleted (bzlmod migration).binaries/emulator_main.cc: rename + keptabsl/flags/usage.h.third_party/spanner_pg/src/backend/utils/cache/lsyscache_test.cc: upstream version.Test plan
bazel build //...on macOSbazel build //...on Linux (CI)bazel test //backend/storage/...bazel test //frontend/server/...persistence_manager_test--data_dir=/tmp/spanner-emuround-trip (create instance/db, write, restart, read)MODULE.bazel🤖 Generated with Claude Code