feat: add Hologres connector and OpenAI-compatible LLM/embedding backend#356
Open
TimothyHologres wants to merge 16 commits into
Open
feat: add Hologres connector and OpenAI-compatible LLM/embedding backend#356TimothyHologres wants to merge 16 commits into
TimothyHologres wants to merge 16 commits into
Conversation
Add an openai-compatible LLM backend that talks to any OpenAI-compatible endpoint (base URL + API key) via @ai-sdk/openai-compatible. Wire the backend through the config schema, provider factory, local-config resolution, rate-limit classification, and health-check redaction. Prompt caching stays disabled for non-Anthropic models via the existing isAnthropicProtocolModel gate.
Add the openai-compatible backend to the interactive setup wizard and the scriptable CLI flags (--llm-base-url, --llm-model, --llm-api-key-env, --llm-api-key-file). The wizard prompts for base URL, an optional API key (paste / env var / none), and a single model id applied to every ktx role. ktx status reports the new backend, and the flags validate that they only apply to --llm-backend openai-compatible.
The openai embedding provider already accepts a custom base URL and dimensions, but the setup wizard hardcoded text-embedding-3-small at 1536 dimensions. Prompt for (and accept flags for) the base URL, model, and dimensions so users can point embeddings at any OpenAI-compatible endpoint, including ones that require a specific vector size.
Document the new openai-compatible LLM backend (provider block, setup flags, single-model preset) and the openai embedding base_url/model/ dimensions options across the LLM configuration guide, the ktx setup CLI reference, and the ktx.yaml reference.
OpenAI-compatible embedding endpoints can cap the number of inputs per request (Alibaba Bailian text-embedding-v4 rejects batches larger than 10), but ktx defaulted to 100, so schema-scan enrichment failed with an HTTP 400 during ingest. Expose the embedding batch size in the setup wizard and as a CLI flag, persisting it to ingest.embeddings.batchSize (mirrored into scan.enrichment.embeddings) so scan and ingest chunk embedding requests within the endpoint's limit.
- Exclude .qoder folder from .gitignore - Remove trailing empty lines at the end of the file
Register hologres as a first-class PostgreSQL-wire-compatible driver. KtxHologresScanConnector extends the postgres connector, reusing its transaction-free connection, introspection, sampling, and pg_stats-based statistics, and only overrides listSchemas to drop Hologres engine-internal schemas (matched exactly so user schemas like hologres_dataset_* are kept). The postgres connector now derives its driver from the connection config so snapshots stamp the correct driver. Hologres reuses the postgres SQL dialect and sqlglot dialect; CLI live-database dispatch routes it to the native Node path so it never hits the daemon (which uses transactions Hologres forbids).
Add Hologres to the ktx setup driver picker, the --database flag parser, scope discovery (schema-scoped like postgres, default port 80), default connection id, and connection-test support. Hologres reuses the shared URL/host connection flow. Query-history wizard wiring is deferred until the hg_query_log reader lands.
Add a windowed HologresQueryLogReader that aggregates hologres.hg_query_log by its digest fingerprint (executions, distinct users, p50/p95 duration, error rate from status, rows, top users) over a query_start window, plus a readiness probe runner. Missing pg_read_all_stats degrades to a warning rather than failing, since Hologres still exposes the current user's own log. Wire the hologres dialect through the historic-SQL config union, dialect maps, probe factory registry, and CLI reader selection (explicit branch so it never falls through to Snowflake), and flip the driver's hasHistoricSqlReader to true. The reader runs a single read-only statement, honoring Hologres's no-transaction constraint.
Add a Hologres section to the primary-sources integration page covering the PostgreSQL-compatible connection config (default port 80), the auto-excluded Hologres system schemas, hg_query_log query history (digest aggregation, slow-log and 30-day retention caveats, pg_read_all_stats soft-degrade), and dialect notes, and reference Hologres in the description and field table.
Detect the Hologres version with select hg_version() (which returns e.g. "Hologres 4.2.10 ...", distinct from the compatible PostgreSQL 11.3 version) and refuse connections older than 4.0 — the first release that supports ktx — in both testConnection and introspect with a clear expected error. Parses the three-part Hologres version after the "Hologres " prefix. connectionId is now protected on the shared Postgres connector so the subclass can run the check via executeReadOnly.
Document that ktx requires Hologres 4.0 or newer and checks it via hg_version() when testing a connection or scanning.
Title: chore(git): add .qoder to .gitignore 此次代码评审主要增加了OpenAI兼容端点的配置支持，包括LLM和嵌入模型的相关参数设置及验证逻辑。 Link: https://code.alibaba-inc.com/kunwu.dy/ktx/codereview/28596518
Title: docs(hologres): note the Hologres 4.0+ requirement 此次代码评审主要添加了Hologres数据库的支持，包括连接配置、查询日志读取、版本检查等功能，并更新了相关文档和测试用例。 Link: https://code.alibaba-inc.com/kunwu.dy/ktx/codereview/28596493
Standalone scan's normalizeDriver kept a hardcoded driver allowlist that drifted from the canonical driverRegistrations registry, so a hologres connection passed the connection test but failed the scan phase. Derive the allowlist (and error message) from the registry so every registered driver is accepted and future connectors cannot regress.
The daemon's /sql/analyze-batch and read-only validation passed the connection dialect straight to sqlglot, which has no "hologres" dialect, so Hologres query-history ingest failed with 500 "Unknown dialect 'hologres'". Hologres is Postgres wire/SQL compatible, so normalize hologres -> postgres at both entry points before any sqlglot call. Adds regression tests for the analyze and validate paths.
|
@TimothyDing is attempting to deploy a commit to the Kaelio Team on Vercel. A member of the Team first needs to authorize it. |
Author
|
@andreybavt Hi~~~ Could you help me to check it? |
Contributor
|
Hey @TimothyHologres , thanks for the PR! I'm AFK during this week, will do my best to have a look ASAP |
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upstreams 16 commits (62 files) from the
aliyun/ktxfork'smain, adding two independent features:hg_query_log, Hologres 4.0+ requirement viahg_version(), and postgres dialect normalization for daemon SQL analysis (plus docs).base_url/model/dimensions, and--embedding-batch-sizefor batch-limited endpoints (plus docs).Also:
chore(git): add .qoder to .gitignore.Commits
OpenAI-compatible
Hologres
Test plan
pnpm run build,pnpm run type-check,pnpm run test