Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/domain.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,13 +197,13 @@ renames. The two exceptions are listed in §5.4.
| `wb__` | Wildberries seller API | FBS orders, commissions, returns |
| `excel__` | Logistics spreadsheets | Container manifests, cross-dock — the inbound-container storyline |
| `pg_ops__` | Postgres OLTP (hot tier) via CDC | Operational order/customer rows promoted into the vault |
| `mp__` (currently `x5__`) | Consolidated marketplace order feed | High-volume retail order history (see §5.4) |
| `mp__` | Consolidated marketplace order feed | High-volume retail order history |

### 5.4 Planned renames / repins (the only code changes the legend requires)

| Change | Scope | Status |
| ------ | ----- | ------ |
| `x5__*` → `mp__*` record_source (+ governance SQL, officer probes, admission tests) | The prefix currently carries the name of the Kaggle seed dataset (X5 Retail Hero) that the demo loader replays as transaction history. Under the legend it is the **consolidated marketplace feed**, and the prefix should say so. Dataset attribution stays in the loader README | Planned (data phase) |
| `x5__*` → `mp__*` record_source (+ governance SQL, officer probes, admission tests) | The prefix carried the name of the Kaggle seed dataset (X5 Retail Hero) that the demo loader replays as transaction history. Under the legend it is the **consolidated marketplace feed**, and the prefix says so. Dataset attribution stays in the loader README | **Done** (B2) |
| Demo value repin: currencies to `RUB` (primary), `AED`/`KZT` in branch stories; demo revenue/counts consistent with §1–2 | `contracts/entities/order.yaml` currency examples, NL demo answers, seeded `ORD-*` rows | Planned (data phase, after the generator spec) |

Vocabulary guardrails for all public docs: the company is an **own-brand /
Expand Down
26 changes: 19 additions & 7 deletions tests/unit/test_dv2_business_vault_ddl.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
this file. It pins that every view parses under sqlglot's ClickHouse dialect and
that the customer MDM views admit hub rows by branch via
``splitByString('__', record_source)[2]`` — NOT by a hard-coded
``record_source = '1c__<branch>'`` filter that silently dropped OLTP/X5-promoted
customers (``record_source`` ``pg_ops__`` / ``x5__``).
``record_source = '1c__<branch>'`` filter that silently dropped OLTP/marketplace-
promoted customers (``record_source`` ``pg_ops__`` / ``mp__``).

This is the ClickHouse half of audit_28_06_26 #12. The PostgreSQL port was fixed
in ``test_dv2_postgres_ddl.py::test_customer_mdm_views_admit_all_source_conventions``;
Expand Down Expand Up @@ -59,15 +59,17 @@ def test_customer_mdm_views_admit_all_source_conventions():
"""audit_28_06_26 #12 (ClickHouse half): the customer MDM views must select
hub rows by branch via ``splitByString('__', record_source)[2]``, NOT by a
hard-coded ``record_source = '1c__<branch>'`` filter that silently drops
OLTP/X5-promoted customers (record_source ``pg_ops__`` / ``x5__``). Proven
live on PostgreSQL in #99: the buggy filter returns 1 of 2 seeded customers,
the source-agnostic filter returns both. This keeps the ClickHouse views in
lock-step with the PostgreSQL port so the engines cannot diverge again."""
OLTP/marketplace-promoted customers (record_source ``pg_ops__`` / ``mp__``).
Proven live on PostgreSQL in #99: the buggy filter returns 1 of 2 seeded
customers, the source-agnostic filter returns both. This keeps the
ClickHouse views in lock-step with the PostgreSQL port so the engines
cannot diverge again."""
for branch in MDM_BRANCHES:
path = BV_DIR / f"bv_customer_mdm__{branch}.sql"
# Strip block/line comments: the headers deliberately quote the old,
# buggy ``record_source = '1c__<branch>'`` filter to explain the fix.
body = _strip_comments(path.read_text(encoding="utf-8"))
raw = path.read_text(encoding="utf-8")
body = _strip_comments(raw)
assert f"CREATE OR REPLACE VIEW rv.bv_customer_mdm__{branch}" in body, (
f"bv_customer_mdm__{branch}.sql must define rv.bv_customer_mdm__{branch}"
)
Expand All @@ -80,3 +82,13 @@ def test_customer_mdm_views_admit_all_source_conventions():
f"hard-coded record_source = '1c__<branch>' filter in "
f"bv_customer_mdm__{branch} reintroduces audit #12"
)
# B2 (domain.md §5.4): the legend's marketplace-feed vocabulary replaces
# the Kaggle dataset name in the third-source-convention example (checked
# on the RAW text — the example lives in the header comment, which body
# strips).
assert "x5__" not in raw, (
f"stale Kaggle-dataset record_source prefix x5__ leaked back into bv_customer_mdm__{branch}.sql"
)
assert "mp__" in raw, (
f"bv_customer_mdm__{branch}.sql should document the mp__ marketplace-feed convention (domain.md §5.3)"
)
19 changes: 15 additions & 4 deletions tests/unit/test_dv2_postgres_ddl.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,10 +147,12 @@ def test_business_vault_uses_postgres_collapse():
def test_customer_mdm_views_admit_all_source_conventions():
"""audit_28_06_26 #12: the customer MDM views must select hub rows by
branch via split_part(record_source, '__', 2), NOT by a hard-coded
record_source = '1c__<branch>' filter that silently drops OLTP/X5-promoted
customers (record_source pg_ops__/x5__). Proven live on PG: the buggy filter
returns 1 of 2 seeded customers, the split_part filter returns both."""
body = _strip_comments((PG_DIR / "03_business_vault.sql").read_text(encoding="utf-8")).lower()
record_source = '1c__<branch>' filter that silently drops OLTP/marketplace-
promoted customers (record_source pg_ops__/mp__). Proven live on PG: the
buggy filter returns 1 of 2 seeded customers, the split_part filter returns
both."""
raw = (PG_DIR / "03_business_vault.sql").read_text(encoding="utf-8").lower()
body = _strip_comments(raw)
branches = ("msk", "spb", "ekb", "dxb", "ala")
for branch in branches:
assert f"view rv.bv_customer_mdm__{branch}" in body, f"missing PG view for {branch}"
Expand All @@ -161,6 +163,15 @@ def test_customer_mdm_views_admit_all_source_conventions():
assert "record_source = '1c__" not in body, (
"hard-coded record_source = '1c__<branch>' filter reintroduces audit #12"
)
# B2 (domain.md §5.4): the legend's marketplace-feed vocabulary replaces the
# Kaggle dataset name in the third-source-convention example (checked on the
# RAW text — the example lives in the header comment, which body strips).
assert "x5__" not in raw, (
"stale Kaggle-dataset record_source prefix x5__ leaked back into 03_business_vault.sql"
)
assert "mp__" in raw, (
"03_business_vault.sql should document the mp__ marketplace-feed convention (domain.md §5.3)"
)


def test_hubs_and_links_have_bytea_primary_keys():
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Purpose: Canonical customer record for the ALA branch.
Layer: Business Vault.
Branch: ala (KZ jurisdiction; Bitrix loyalty not wired in ALA).
Hub admission: splitByString('__', record_source)[2] = 'ala' (source-agnostic:
1c__/pg_ops__/x5__ all integrated, not only 1C; audit_28_06_26 #12;
1c__/pg_ops__/mp__ all integrated, not only 1C; audit_28_06_26 #12;
mirrors the PostgreSQL port's split_part(record_source,'__',2)).
Security: SQL SECURITY DEFINER (ADR 0006 Phase 2) — readers query this view
under the definer's rights, so the column-limited grants in
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Conflict policy:
keeps the loyalty columns for schema parity with bv_customer_mdm__msk so
downstream marts can UNION ALL the two branches without renaming.
Hub admission: splitByString('__', record_source)[2] = 'dxb' (source-agnostic:
1c__/pg_ops__/x5__ all integrated, not only 1C; audit_28_06_26 #12;
1c__/pg_ops__/mp__ all integrated, not only 1C; audit_28_06_26 #12;
mirrors the PostgreSQL port's split_part(record_source,'__',2)).
Security: SQL SECURITY DEFINER (ADR 0006 Phase 2) — readers query this view
under the definer's rights, so the column-limited grants in
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Purpose: Canonical customer record for the EKB branch.
Layer: Business Vault.
Branch: ekb (RU jurisdiction; same conflict policy as MSK).
Hub admission: splitByString('__', record_source)[2] = 'ekb' (source-agnostic:
1c__/pg_ops__/x5__ all integrated, not only 1C; audit_28_06_26 #12;
1c__/pg_ops__/mp__ all integrated, not only 1C; audit_28_06_26 #12;
mirrors the PostgreSQL port's split_part(record_source,'__',2)).
Security: SQL SECURITY DEFINER (ADR 0006 Phase 2) — readers query this view
under the definer's rights, so the column-limited grants in
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ Conflict policy:
- If a customer exists only in Bitrix, PII columns are NULL but the row
is still returned so loyalty-only customers stay visible.
Hub admission: splitByString('__', record_source)[2] = 'msk', so a customer
promoted under ANY source convention (1c__msk, pg_ops__msk, x5__msk,
promoted under ANY source convention (1c__msk, pg_ops__msk, mp__msk,
...) is integrated, not only 1C. The old record_source = '1c__msk'
filter silently dropped OLTP/X5-promoted customers (audit_28_06_26 #12);
filter silently dropped OLTP/marketplace-promoted customers (audit_28_06_26 #12);
this mirrors the PostgreSQL port's split_part(record_source,'__',2).
Security: SQL SECURITY DEFINER (ADR 0006 Phase 2) — readers query this view
under the definer's rights, so the column-limited grants in
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Purpose: Canonical customer record for the SPB branch.
Layer: Business Vault.
Branch: spb (RU jurisdiction; same conflict policy as MSK).
Hub admission: splitByString('__', record_source)[2] = 'spb' (source-agnostic:
1c__/pg_ops__/x5__ all integrated, not only 1C; audit_28_06_26 #12;
1c__/pg_ops__/mp__ all integrated, not only 1C; audit_28_06_26 #12;
mirrors the PostgreSQL port's split_part(record_source,'__',2)).
Security: SQL SECURITY DEFINER (ADR 0006 Phase 2) — readers query this view
under the definer's rights, so the column-limited grants in
Expand Down
4 changes: 2 additions & 2 deletions warehouse/agentflow/dv2/postgres/03_business_vault.sql
Original file line number Diff line number Diff line change
Expand Up @@ -144,10 +144,10 @@ Dialect: postgresql.
Branch: one view per jurisdiction by design — PII stays in branch.
Hub admission: split_part(record_source,'__',2) = '<branch>', so a customer
promoted under ANY source convention (1c__<branch>, pg_ops__<branch>,
x5__<branch>, ...) is integrated. This mirrors bv_order_canonical's
mp__<branch>, ...) is integrated. This mirrors bv_order_canonical's
order_branch derivation above. The ClickHouse views hard-code
record_source = '1c__<branch>', which silently dropped every OLTP- and
X5-promoted customer (record_source pg_ops__/x5__) from the MDM result
marketplace-promoted customer (record_source pg_ops__/mp__) from the MDM result
(audit_28_06_26 #12). The hash keys were never incompatible —
customer_hk = md5(business_key) is identical across loaders; only the
hub record_source filter excluded them. split_part is the source-
Expand Down
2 changes: 1 addition & 1 deletion warehouse/agentflow/dv2/postgres/governance/verify_live.sh
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ INSERT INTO rv.hub_customer (customer_hk, customer_bk, record_source) VALUES
(decode(md5('CUST-MSK-5'),'hex'),'CUST-MSK-5','1c__msk'),
(decode(md5('CUST-MSK-6'),'hex'),'CUST-MSK-6','1c__msk'),
(decode(md5('CUST-MSK-7'),'hex'),'CUST-MSK-7','pg_ops__msk'),
(decode(md5('CUST-MSK-8'),'hex'),'CUST-MSK-8','x5__msk'),
(decode(md5('CUST-MSK-8'),'hex'),'CUST-MSK-8','mp__msk'),
(decode(md5('CUST-DXB-1'),'hex'),'CUST-DXB-1','1c__dxb'),
(decode(md5('CUST-DXB-2'),'hex'),'CUST-DXB-2','1c__dxb')
ON CONFLICT (customer_hk) DO NOTHING;
Expand Down