feat: add Amazon Redshift connector by samuelayanshina · Pull Request #304 · Kaelio/ktx

samuelayanshina · 2026-06-17T10:42:04Z

Closes #161

Adds a redshift connector for read-only schema introspection, table/column sampling, column statistics, and read-only SQL execution against Amazon Redshift.

Approach

Redshift speaks the PostgreSQL wire protocol, so the connector is modeled on connectors/postgres and reuses the pg driver for connection plumbing. Metadata introspection is swapped to Redshift system views for accuracy:

SVV_TABLES for the table/view list (including EXTERNAL TABLE / Spectrum)
SVV_TABLE_INFO.tbl_rows for accurate row counts (Redshift's pg_class.reltuples is unreliable)
SVV_COLUMNS for column metadata
Primary/foreign keys read from information_schema (Redshift stores declared constraints there; it does not enforce them, and there is no better SVV_* source)

The driver is wired into the connection driver union, dialect factory, driver registry, and warehouse connection schema. Default port is 5439.

Tests

New test/connectors/redshift/connector.test.ts with mocked pool factories covering introspection, sampling, column stats, read-only SQL, schema/table listing, and pool error handling
Added redshift fixtures to the existing dialect and driver conformance tests
Full suite passes; pre-commit (ruff, biome, knip, detect-secrets) clean

Docs

Added a Redshift section to primary-sources.mdx with connection-config examples and a feature/auth matrix.

Scope (honest about what's not here)

This is a focused v1. Two acceptance-criteria items from the issue are intentionally deferred so this PR stays reviewable, and I've marked them as such in the docs:

IAM authentication only password / URL auth is implemented so far. IAM (temporary credentials via GetClusterCredentials) is a planned follow-up.
Query history (STL_QUERY) hasHistoricSqlReader is false; query-history ingestion is a planned follow-up.

Happy to fast-follow with either (or fold them into this PR) based on what you'd prefer to see first.

(Following up from the MySQL columnStats work in #233.)

vercel · 2026-06-17T10:42:09Z

@samuelayanshina is attempting to deploy a commit to the Kaelio Team on Vercel.

A member of the Team first needs to authorize it.

samuelayanshina · 2026-06-22T21:33:35Z

Friendly ping on this one. CI is green and it merges cleanly. The only red check is the Vercel preview deploy, which needs a maintainer to authorize it (external-contributor gate) rather than anything in the diff.

No rush on your end, just flagging it's ready for review whenever someone has a cycle. And as noted above, happy to fold in IAM auth or STL_QUERY query history if you'd rather see those land in the same PR.

Add a redshift connector modeled on the Postgres connector (Redshift speaks the Postgres wire protocol), using Redshift system views for introspection: SVV_TABLES and SVV_TABLE_INFO for tables/row counts, SVV_COLUMNS for columns. Wires the driver into the type union, dialect factory, driver registry, and connection schema.

Mirror the Postgres connector test suite with mocked pool factories, adapting SQL mock keys to the Redshift SVV_* introspection queries (svv_tables, svv_columns, svv_table_info) and the 5439 default port.

…spection Add a Redshift section to primary-sources.mdx with connection-config examples and honest feature/auth scoping (password auth + SVV_* metadata introspection; IAM auth and STL_QUERY history noted as follow-ups). Remove the copied live-database-introspection module and its tests since this connector does not wire the live-database ingest path.

andreybavt

Left few inline review comments below.

andreybavt · 2026-07-03T08:25:02Z

@@ -0,0 +1,834 @@
+import { resolveStringReference } from '../shared/string-reference.js';
+import { getDialectForDriver } from '../../context/connections/dialects.js';


This should use getSqlDialectForDriver('redshift'), not getDialectForDriver. The latter returns only KtxDialect, but this connector calls SQL-only methods such as generateSampleQuery, quoteIdentifier, and formatTableName, so the PR currently fails TypeScript with TS2339 errors.

andreybavt · 2026-07-03T08:25:02Z

+        CASE WHEN table_type = 'VIEW' THEN 'v' ELSE 'r' END AS table_kind
+      FROM svv_tables
+      WHERE table_schema = ANY($1)
+        AND table_type IN ('TABLE', 'VIEW', 'EXTERNAL TABLE')


This filter drops normal Redshift tables. SVV_TABLES.table_type reports regular tables as BASE TABLE, not TABLE, so both listTables() and the similar introspection filter below can come back empty or view-only against a normal schema.

andreybavt · 2026-07-03T08:25:02Z

+      SELECT table_schema AS schema_name, table_name AS table_name,
+        CASE WHEN table_type = 'VIEW' THEN 'v' ELSE 'r' END AS table_kind
+      FROM svv_tables
+      WHERE table_schema = ANY($1)


This copies the Postgres array-binding pattern, but Redshift does not accept a bound Postgres array for = ANY($1) here. listTables(['public']) and scoped scans can fail against a real cluster even though the mocks pass. Please generate a Redshift-valid scoped predicate, for example safe positional IN ($1, $2, ...) placeholders.

andreybavt · 2026-07-03T08:25:02Z

+  }
+
+  getSampleValueAggregation(innerSql: string): string {
+    return `(SELECT STRING_AGG(CAST(value AS TEXT), CHR(31)) FROM (${innerSql}) AS relationship_profile_values)`;


This emits Postgres STRING_AGG, but Redshift's string aggregation function is LISTAGG. This path is used by relationship profiling, so enriched/relationship scans can fail or lose profile data once they hit this dialect. CHR(31) is fine on Redshift; the aggregate function is the issue.

andreybavt · 2026-07-03T08:25:02Z

  clickhouse: () => new KtxClickHouseDialect(),
  mysql: () => new KtxMysqlDialect(),
  postgres: () => new KtxPostgresDialect(),
+  redshift: () => new KtxRedshiftDialect(),


Adding the SQL dialect factory here is only half of the dialect wiring. packages/cli/src/context/sql-analysis/dialect.ts still does not map the redshift driver, so SQL analysis and local semantic-layer query compilation silently fall back to Postgres. That means Redshift-native SQL can be parsed/generated under the wrong dialect.

andreybavt · 2026-07-03T08:25:02Z

      };
    },
  },
+  redshift: {


This registers Redshift for scanning, but it is still missing from the local warehouse identity path: no REDSHIFT ConnectionType, no DRIVER_TO_CONNECTION_TYPE entry in local-warehouse-descriptor.ts, and no Metabase/Looker mapping support. A hand-written Redshift connection can scan, but local ingest warehouse discovery and BI mapping flows do not treat it as a first-class warehouse target.

andreybavt · 2026-07-03T08:25:02Z

 | Field | Required | Applies to | Description |
 |-------|----------|------------|-------------|
-| `driver` | Yes | all connections | Connector driver such as `postgres`, `snowflake`, `bigquery`, `mysql`, `clickhouse`, `sqlserver`, `sqlite`, or `mongodb` |
+| `driver` | Yes | all connections | Connector driver such as `postgres`, `redshift`, `snowflake`, `bigquery`, `mysql`, `clickhouse`, `sqlserver`, `sqlite`, or `mongodb` |


This advertises redshift as a normal primary-source driver, but ktx setup still does not know about it: the setup driver union, interactive options, default connection id, scope discovery, and URL/field setup path exclude Redshift. Either setup should support creating/managing Redshift connections in this PR, or the docs should clearly say manual config only.

andreybavt · 2026-07-03T08:25:02Z

+  };
+  const searchPathSchemas = searchPathSchemasFromConnection(merged);
+  if (searchPathSchemas.length > 0) {
+    config.options = `-c search_path=${searchPathSchemas.join(',')}`;


The Postgres connector now includes query_timeout_ms in the pool options as server-side statement_timeout and normalizes 57014 cancellations to the shared query exceeded Ns error. This fork dropped that behavior, which is user-visible because the docs describe query_timeout_ms as applying to all warehouses. The broader shared pg-wire base can be a follow-up, but the timeout behavior should not regress in the new connector.

# Conflicts: # docs-site/content/docs/integrations/primary-sources.mdx # packages/cli/src/context/scan/types.ts # packages/cli/test/context/connections/dialects.test.ts

Upstream split KtxDialect into base and SQL variants; quoteIdentifier now lives on KtxSqlDialect, so the connector must resolve its dialect via getSqlDialectForDriver like the Postgres connector does.

Extract the pg-protocol plumbing that Postgres and Redshift both need into connectors/shared/pg-wire.ts: pool factory and pool types, pool config resolution (parameterised by default port and connector label), URL parsing, read-only query prep, and query/row helpers. Postgres and Redshift now import this module instead of each carrying their own copy. Each connector keeps its own metadata introspection (Postgres pg_catalog, Redshift SVV_*), driver name, dialect, and default port (5432 / 5439). Postgres behaviour is unchanged: its test suite passes untouched, and its public exports are preserved as thin aliases over the shared types. Redshift picks up the statement_timeout / query-deadline handling it was previously missing, matching Postgres.

samuelayanshina · 2026-07-09T14:33:40Z

Thanks for the thorough review @andreybavt, and apologies for the slow reply, I missed the inline comments until now.

Two are already addressed in the commits I pushed since:

getSqlDialectForDriver — fixed in d207a8e.
The dropped statement_timeout / 57014 normalization, fixed in 9d8e51f, which also extracts the shared connectors/shared/pg-wire.ts base you mentioned, so Postgres and Redshift now share the pool/config/query plumbing. Postgres behaviour is unchanged (its test file passes untouched).

On the rest — the Redshift-correctness ones are fair hits, and the mocks hid them:

BASE TABLE vs TABLE in SVV_TABLES.table_type: agreed, the current filter would drop regular tables. Fixing.
= ANY($1) array binding: agreed, switching to positional IN ($1, $2, ...) placeholders.
STRING_AGG → LISTAGG in the dialect: agreed.
sql-analysis/dialect.ts mapping: will add.
Warehouse identity (REDSHIFT ConnectionType, DRIVER_TO_CONNECTION_TYPE, Metabase/Looker mapping): will add.

On ktx setup: I'd rather not balloon this PR, my preference is to scope setup out and make the docs say Redshift is manual ktx.yaml config for now, then add setup support in a follow-up. Happy to do it here instead if you'd rather it land together.

I'll push the fixes and ping you.

…ng aggregate Three Redshift-correctness fixes from review: - SVV_TABLES.table_type reports regular tables as BASE TABLE, not TABLE. The old filter dropped every ordinary table from listTables() and introspection. - Redshift does not accept a bound array for '= ANY($n)'. Scoped predicates now expand to positional IN ($n, $n+1, ...) placeholders, derived from the value count only. - Redshift has no STRING_AGG; the dialect now emits LISTAGG for sample value aggregation used by relationship profiling. Tests assert the generated SQL for each, so the previous behaviour would now fail.

sqlglot has a first-class redshift dialect that differs from postgres in identifier normalization (case-insensitive) and array index offset, so falling through to the postgres default parsed Redshift SQL under the wrong dialect.

Register REDSHIFT as a ConnectionType, map the redshift driver to it in DRIVER_TO_CONNECTION_TYPE, and give it its sqlglot dialect in the connection-type dialect map, so a Redshift connection is treated as a first-class warehouse target by local ingest and semantic-layer compute rather than only being scannable.

ktx setup has no Redshift path yet, so the integration docs should not imply it can create one. Redshift is manual ktx.yaml config for now.

samuelayanshina · 2026-07-09T15:22:17Z

Pushed fixes for the review. Rundown:
Fixed

getSqlDialectForDriver instead of getDialectForDriver (d207a8e).
statement_timeout / 57014 normalization restored. Rather than re-copy it, I extracted connectors/shared/pg-wire.ts, the pool factory, pool-config resolution (parameterised by default port and connector label), URL parsing, read-only query prep, and query/row helpers, and both Postgres and Redshift now import it (9d8e51f). Postgres behaviour is unchanged: test/connectors/postgres/ passes with its test file untouched, and its public exports are preserved as aliases over the shared types.
SVV_TABLES.table_type reports regular tables as BASE TABLE, not TABLE. Fixed in both the listTables() filter and the introspection filter (0a82c2f).
= ANY($n) array binding replaced with positional IN ($n, $n+1, ...) placeholders, generated from the value count only (0a82c2f).
STRING_AGG → LISTAGG in the dialect (0a82c2f).
sql-analysis/dialect.ts now maps redshift to sqlglot's redshift dialect. This was a real bug rather than imprecision: sqlglot's Redshift dialect differs from Postgres in identifier normalization (case-insensitive) and array index offset (bd01aa2).
REDSHIFT ConnectionType, DRIVER_TO_CONNECTION_TYPE, and the connection-type sqlglot mapping are now wired (a0cf489).
Docs now state that Redshift is manual ktx.yaml config and ktx setup doesn't manage it (d24dbaa).

The BASE TABLE and ANY($n) bugs were mock-hidden: the fake pool returned table_kind already CASE-transformed, so neither the table_type filter nor the scoped predicate was ever exercised. Tests now assert the generated SQL for each, so the old behaviour fails.
Proposed follow-ups

Metabase/Looker mapping. I'd rather not guess the keys, Looker exposes several Redshift dialects (Amazon Redshift, 2.1+, Serverless 2.1+) and its identifiers aren't guessable from outside (awsathena, bigquery_standard_sql); same for Metabase's engine string. A wrong key silently never matches, which is worse than omitting it. This also mirrors how Athena landed: connector in #309, warehouse identity + BI mapping in #332. Happy to do it here if you can confirm the exact identifiers, or as a follow-up.
ktx setup. Kept out of this PR, with the docs updated to say so.

Also rebased onto latest main. Full suite green (only the pre-existing repo-fetch failure), pre-commit clean including biome and knip.

samuelayanshina · 2026-07-22T08:31:40Z

@andreybavt whenever you have a cycle, all review comments are addressed and CI is green. The two open questions are whether you want the Metabase/Looker mapping in this PR (I'd need the exact Looker dialect and Metabase engine identifiers for Redshift) or as a follow-up like Athena's #332, and same for ktx setup. Happy either way.

samuelayanshina mentioned this pull request Jun 27, 2026

Add Amazon Redshift connector #161

Open

5 tasks

luca-martial requested a review from andreybavt June 28, 2026 22:36

samuelayanshina added 3 commits June 29, 2026 21:31

test(redshift): add connector unit tests

e7e7b78

Mirror the Postgres connector test suite with mocked pool factories, adapting SQL mock keys to the Redshift SVV_* introspection queries (svv_tables, svv_columns, svv_table_info) and the 5439 default port.

samuelayanshina force-pushed the feat/redshift-connector branch from 696a607 to de464cc Compare June 29, 2026 20:57

andreybavt reviewed Jul 3, 2026

View reviewed changes

andreybavt and others added 3 commits July 3, 2026 10:35

Merge remote-tracking branch 'origin/main' into feat/redshift-connector

a8bc2ae

# Conflicts: # docs-site/content/docs/integrations/primary-sources.mdx # packages/cli/src/context/scan/types.ts # packages/cli/test/context/connections/dialects.test.ts

fix(redshift): use getSqlDialectForDriver after upstream dialect split

d207a8e

Upstream split KtxDialect into base and SQL variants; quoteIdentifier now lives on KtxSqlDialect, so the connector must resolve its dialect via getSqlDialectForDriver like the Postgres connector does.

samuelayanshina added 4 commits July 9, 2026 15:57

docs(redshift): note that setup does not manage Redshift connections

d24dbaa

ktx setup has no Redshift path yet, so the integration docs should not imply it can create one. Redshift is manual ktx.yaml config for now.

Merge branch 'main' into feat/redshift-connector

b952dca

		@@ -0,0 +1,834 @@
		import { resolveStringReference } from '../shared/string-reference.js';
		import { getDialectForDriver } from '../../context/connections/dialects.js';

Uh oh!

Conversation

samuelayanshina commented Jun 17, 2026

Approach

Tests

Docs

Scope (honest about what's not here)

Uh oh!

vercel Bot commented Jun 17, 2026

Uh oh!

samuelayanshina commented Jun 22, 2026

Uh oh!

andreybavt left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelayanshina commented Jul 9, 2026

Uh oh!

samuelayanshina commented Jul 9, 2026

Uh oh!

samuelayanshina commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andreybavt left a comment •

edited

Loading