Skip to content

Add DynamoDB connector support (defineDynamoDBConnection + dynamodb datasource block) #164

Description

@gnzjgo

Summary

The Tinybird DynamoDB connector is generally available through the Tinybird CLI (tb connection create dynamodb, TYPE dynamodb in .connection files, IMPORT_TABLE_ARN / IMPORT_EXPORT_BUCKET in .datasource files), but the TypeScript SDK currently only exposes helpers for Kafka, S3, and GCS.

This forces SDK users to keep a mixed project (TypeScript definitions + raw .connection / .datasource files) just for DynamoDB.

Docs page: https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb

Proposal

Add a defineDynamoDBConnection helper and a dynamodb block on defineDatasource, mirroring the Kafka / S3 / GCS shape.

Connection

import { defineDynamoDBConnection, secret } from "@tinybirdco/sdk";

export const myDdb = defineDynamoDBConnection("my_ddb", {
  arn: secret("dynamodb_role_arn_my_ddb"),  // DYNAMODB_ARN
  region: "us-east-1",                       // DYNAMODB_REGION
});

Datasource

import { defineDatasource, engine, t } from "@tinybirdco/sdk";
import { myDdb } from "./connections";

export const orders = defineDatasource("orders", {
  schema: {
    pk: t.string(),
    sk: t.string(),
    _record: t.string(),
    _old_record: t.string().nullable(),
    _timestamp: t.dateTime64(3),
    _event_name: t.string().lowCardinality(),
    _is_deleted: t.uint8(),
  },
  engine: engine.replacingMergeTree({
    sortingKey: ["pk", "sk"],
    ver: "_timestamp",
    isDeleted: "_is_deleted",
  }),
  dynamodb: {
    connection: myDdb,
    tableArn: "arn:aws:dynamodb:us-east-1:123456789012:table/orders",  // IMPORT_TABLE_ARN
    exportBucket: "my-orders-exports",                                  // IMPORT_EXPORT_BUCKET
  },
});

Notes:

  • The DynamoDB datasource must use ReplacingMergeTree; consider validating this at the type level if possible.
  • The columns _record, _old_record, _timestamp, _event_name, _is_deleted are mandatory system columns. Consider exposing them as a helper / preset so users don't have to redeclare them every time.
  • Once engine.replacingMergeTree and the dynamodb block exist, the docs at https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb can add TypeScript SDK tabs alongside the Tinybird CLI ones.

Why

Today docs need an explicit callout that the DynamoDB connector is CLI-datafile-only, which breaks the "define everything in TypeScript" workflow the SDK is built around.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions