Reject primary-key conflicts on insert (BigQuery)

[sagargg]

What

When a resource declares a primaryKey, datastore_upsert with method=insert (and datastore_create with records on an existing table) now rejects the batch with a 400 Validation Error if any incoming row would duplicate a key — either against an existing table row or against another row in the same batch. Nothing is written.

PK-less resources are unchanged: a plain INSERT, no check.

Why

BigQuery does not enforce primary-key / unique constraints — they're informational only, and this codebase doesn't even emit them in DDL (the primaryKey lives in the table's Frictionless metadata). So a plain INSERT silently created duplicate rows. This mirrors CKAN's Postgres datastore, where method=insert on a duplicate key errors.

How

insert_guarded_sql wraps the conflict probe and the INSERT in a single BigQuery script:

BEGIN
  DECLARE _conflicts INT64 DEFAULT 0;
  SET _conflicts = (<conflict-count probe>);
  IF _conflicts > 0 THEN RAISE USING MESSAGE = FORMAT('DATASTORE_PK_CONFLICT %d', _conflicts); END IF;
  INSERT INTO … SELECT … FROM UNNEST(JSON_QUERY_ARRAY(@rows));
END

• One job: the probe and the INSERT share a single submission, so the @rows batch is serialised + uploaded only once (vs. two jobs / two uploads).
• No partial inserts: the RAISE fires before the INSERT, and the INSERT is the script's only write and a single atomic DML statement — all rows land or none.
• The RAISE emits a sentinel + count; _translate_bigquery_error rebuilds a user-facing ValidationError from BigQuery's wrapped error text.

Tests

New unit tests cover the guarded-script shape, the single-write / guard-ordering invariant, the missing-PK guard, the reject path (writes nothing), the clean single-job path, and the no-PK skip. Full suite green.

Note for reviewers

origin/main was behind by 2 earlier datastore_search_sql commits when this branch was cut, so this PR may list them until main is synced. Only the latest commit (Reject primary-key conflicts on insert) is the substance of this PR.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Insert operations now validate and reject duplicate primary key values, raising a validation error with guidance to use upsert instead of silently allowing duplicates.

• Documentation

  • Updated documentation for insert behavior to clarify that duplicate primary key rows are now rejected.

• Performance

  • Optimized row counting queries for improved efficiency.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 77b8ce45-8b64-4289-99ef-da2eb0c738b8

📥 Commits

Reviewing files that changed from the base of the PR and between 12e124d and 3c46b18.

📒 Files selected for processing (3)

• datastore/infrastructure/engines/bigquery/backend.py
• datastore/infrastructure/engines/bigquery/lib.py
• tests/engines/bigquery/test_tables.py

---

📝 Walkthrough

Walkthrough

BigQueryBackend gains PK-conflict detection on method="insert" writes: new insert_conflict_count_sql and insert_guarded_sql helpers in lib.py build an atomic BEGIN…END script that counts duplicate keys and RAISEs a sentinel before inserting. _translate_bigquery_error maps that sentinel to a ValidationError. On the read path, search_sql gains stable ordering via a new default_order_by utility, and the unfiltered row-count query switches from INFORMATION_SCHEMA.TABLE_STORAGE to a direct COUNT(*).

Changes

BigQuery Write Guard and Read Path

Layer / File(s)	Summary
PK conflict-count SQL builders datastore/infrastructure/engines/bigquery/lib.py	Adds insert_conflict_count_sql (counts in-batch and table-collision duplicates), PK_CONFLICT_SENTINEL constant, and insert_guarded_sql (wraps the probe and INSERT into a single BEGIN…END script that RAISEs with the sentinel on conflict).
Backend wiring: _insert_records and error translation datastore/infrastructure/engines/bigquery/backend.py	Imports the new helpers; _insert_records routes to insert_guarded_sql when normalize_pk(schema) is set; _translate_bigquery_error detects the sentinel and emits a ValidationError with conflicting row count; upsert docstring updated.
Tests for PK-guarded INSERT tests/engines/bigquery/test_tables.py	Unit tests assert PK conflicts raise ValidationError, successful PK inserts submit one BEGIN…END job with shared @rows, no primaryKey falls back to plain INSERT, and insert_conflict_count_sql/insert_guarded_sql SQL structure is validated. Existing upsert(method='insert') tests tightened.
Read path: default_order_by + COUNT(*) swap datastore/infrastructure/engines/bigquery/lib.py, datastore/infrastructure/engines/bigquery/backend.py	Adds default_order_by using sqlglot to inject ORDER BY only for safe single-table SELECTs. search_sql applies it before the data query. _search_sql_count_query replaces the INFORMATION_SCHEMA.TABLE_STORAGE lookup with a direct SELECT COUNT(*) AS n FROM <table>.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• datopian/datastore#3: Refactors normalize_pk in lib.py and extracts COUNT-query logic in backend.py, both of which are directly consumed by the guarded INSERT and COUNT(*) changes in this PR.

Suggested reviewers

• luccasmmg

🐇 A rabbit hops with glee today,
No duplicate keys shall sneak away!
BEGIN … RAISE … END — the guard is set,
Conflicting rows? A ValidationError you'll get.
And ORDER BY keeps pages neat in a row —
This warren of data now has stable flow! 🥕

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/insert-pk-conflict-rejection

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}