Skip to content

.dbml have problems with apices #73

@FaediMichele

Description

@FaediMichele

I need to analyze a dbml that come from dbterd.
dbterd is a utility that create dbml given a catalogs.json and manifest.json (dbt compilation results).

Then I need to store the dbml in a database. I'm saving it as a string serializing it using .dbml and to retrieve I serialize with PyDBML(value).
The problem is that the dbterd correctly put apices on the types while pydbml removes them on the serializer, so the type "timestamp without time zone" is wrongly parsed.

To test this I used this script:

import tempfile, urllib.request
from pathlib import Path
from dbterd.api import DbtErd
from pydbml import PyDBML

manifest = urllib.request.urlopen('http://localhost:8080/manifest.json').read().decode()
catalog = urllib.request.urlopen('http://localhost:8080/catalog.json').read().decode()

with tempfile.TemporaryDirectory() as d:
    with open(Path(d)/'manifest.json','w') as f: f.write(manifest)
    with open(Path(d)/'catalog.json','w') as f: f.write(catalog)
    dbml = DbtErd(artifacts_dir=d, output=Path(d)/'output.dbml').get_erd()

db = PyDBML(dbml)
print('=== original dbml (first 500 chars) ===')
print(dbml[:500])
print()
print('=== db.dbml (first 500 chars) ===')
print(db.dbml[:500])
print()
# try round-trip
try:
    db2 = PyDBML(db.dbml)
    print('Round-trip OK')
except Exception as e:
    print(f'Round-trip FAILED: {e}')
    # find the problematic area
    lines = db.dbml.split('\n')
    for i, line in enumerate(lines[5:9], start=6):
        print(f'  line {i}: {repr(line)}')

This is the result:

=== original dbml (first 500 chars) ===
//Tables (based on the selection criteria)
//--configured at schema: jaffle_shop.public
Table "model.jaffle_shop.customers" {
  "customer_id" "text" [note: "The unique key of the orders mart."]
  "customer_name" "text" [note: "Customers' full name."]
  "count_lifetime_orders" "bigint" [note: "Total number of orders a customer has ever placed."]
  "first_ordered_at" "timestamp without time zone" [note: "The timestamp when a customer placed their first order."]
  "last_ordered_at" "timestamp witho

=== db.dbml (first 500 chars) ===
// Tables (based on the selection criteria)
// --configured at schema: jaffle_shop.public
Table "model.jaffle_shop.customers" {
    "customer_id" text [note: 'The unique key of the orders mart.']
    "customer_name" text [note: 'Customers\' full name.']
    "count_lifetime_orders" bigint [note: 'Total number of orders a customer has ever placed.']
    "first_ordered_at" timestamp without time zone [note: 'The timestamp when a customer placed their first order.']
    "last_ordered_at" timestamp w

Round-trip FAILED: Expected '}', found '"'  (at char 354), (line:7, col:5)
  line 6: '    "count_lifetime_orders" bigint [note: \'Total number of orders a customer has ever placed.\']'
  line 7: '    "first_ordered_at" timestamp without time zone [note: \'The timestamp when a customer placed their first order.\']'
  line 8: '    "last_ordered_at" timestamp without time zone [note: \'The timestamp of a customer\\\'s most recent order.\']'
  line 9: '    "lifetime_spend_pretax" numeric [note: \'The sum of all the pre-tax subtotals of every order a customer has placed.\']'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions