feat: semantic provider plugin framework (Cube / LookML / MetricFlow → Doris SQL) - #16
Open
Jmmt-mingrui wants to merge 17 commits into
Open
feat: semantic provider plugin framework (Cube / LookML / MetricFlow → Doris SQL)#16Jmmt-mingrui wants to merge 17 commits into
Jmmt-mingrui wants to merge 17 commits into
Conversation
Introduce the plugin SPI for semantic model formats, split into a build-time half (validate/parse/compile -> CompiledArtifact) and a runtime half (get_metrics/get_dimensions/generate_sql). The compiled artifact uses a standardized envelope (provider, version, source digest) with a deliberately provider-opaque payload — this is a Semantic Model -> Doris SQL compiler framework, not a semantic exchange standard.
Cube-schema YAML provider modeled after cube-js/cube data models: - parser: cubes with measures (count/count_distinct/sum/avg/min/max/number), dimensions (string/number/time/boolean) and many_to_one joins - compiler: pre-resolved artifact with metric/dimension indexes and a source digest — queries never re-parse YAML - runtime: injection-safe Doris SQL generation (validated names, escaped literals, structured filters, join resolution, GROUP BY/ORDER BY/LIMIT)
Uploads are routed by explicit provider name or by confidence-based content sniffing (detect() >= 0.5). Optional providers (LookML/lkml) degrade gracefully when their dependencies are missing.
Bridges the existing vendored MetricFlow engine into the provider SPI: - build time: lightweight dbt semantic_models/metrics YAML parsing for validation and metadata preview - runtime: bind() attaches a live workspace MetricFlowCompiler+manifest so generate_sql delegates to the real engine; unbound use fails loudly
Looker .view.lkml support built on the MIT-licensed lkml parser:
- views with sql_table_name, dimensions, dimension_groups (expanded to
Doris date_trunc expressions) and measures (count/sum/average/...)
- ${TABLE}.x / ${x} references normalized at parse time; cross-view
references and derived tables rejected with clear errors
- compiler translates views into the Cube artifact shape so the shared
Cube SQL runtime is reused instead of reimplemented
Adds lkml>=1.3 as a dependency (MIT).
File-backed store under <workspace>/.artifacts/ with a provenance envelope (provider, source filename, digest, created_at), path-traversal safe ids, and corruption-tolerant listing.
Transport-agnostic implementations of the model lifecycle pipeline (upload/validate/compile/store artifact, metadata discovery, SQL generation, and execute-through-query) over the provider SPI, reusing the existing read-only SQL validator and response envelope.
list_semantic_providers, compile_semantic_model, list_semantic_artifacts, delete_semantic_artifact, get_semantic_metadata, generate_semantic_sql (dry-run) and query_semantic_model (generate + execute), with the same auth/audit/workspace handling as the existing metric-layer tools.
Cube provider (28): detection, validation errors, compilation, SQL generation incl. join resolution, filter escaping/injection safety. LookML provider (10): dimension_group expansion, reference rules, shared-runtime SQL generation. Registry + artifact store (14): routing, unknown providers, store CRUD, path traversal and corruption handling. Provider tools (12): lifecycle, metadata, dry-run SQL, fake-pool query.
Architecture, the no-unified-artifact-format decision, built-in provider matrix, structured-filter injection safety, the 7 new MCP tools, and an end-to-end agent example. Also updates the tool count to 17.
Mutating the registry singleton would race across concurrent workspaces; bind() now returns a new bound adapter and the registered instance stays stateless (found in review).
CI and build.sh install from requirements.txt, not pyproject — without this the LookML provider tests would fail in CI (found in review).
Offline unit tests plus a linux-x64 build check on push/PR to main. Build outputs are discarded — no artifacts uploaded, no releases. (same workflow as apache/doris-mcp-server#174, comments in English)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
本 PR 为 Doris MCP Server 引入 Semantic Provider 插件框架——一个
Semantic Model → Doris SQL的编译器运行时(定位类似 dbt compile + query engine + MCP interface),将语义模型支持从单一内置 MetricFlow 扩展为可插拔的多格式体系。架构
关键设计决策
payload各家私有(类比 Java.class/ dbtmanifest.json)——这是编译器框架,不是语义交换标准。{dimension, operator, value}而非 SQL 片段;名称对 artifact 校验、字面量转义,生成 SQL 构造上免疫注入。内置 Provider(尽量借用现有开源项目)
cubelookml.view.lkmlmetricflowsemantic_modelsYAMLbind()挂接 workspace compiler)格式自动嗅探(
detect()置信度 ≥0.5 路由)或显式指定 provider。新增 MCP Tools(7 个,总计 17 个)
list_semantic_providers/compile_semantic_model/list_semantic_artifacts/delete_semantic_artifact/get_semantic_metadata/generate_semantic_sql(干跑)/query_semantic_model(生成+只读校验+执行)测试与文档
test/run_all_tests.sh --offline;本地 20 项离线套件全绿。DESIGN.md新增 §6A 完整设计文档;双语 README 更新;examples/semantic-models/提供 Cube/LookML 示例(均通过 provider 校验)。Review 过程发现并已修复
bind()原设计会修改注册表单例 → 跨工作区并发竞态,改为返回新绑定副本(3c52dd8)requirements.txt漏加lkml(CI/build.sh 不走 pyproject)→ 补齐(28a155d)3beaef9)已知边界(后续 PR)
sql_table_name视图)generate_sql需要绑定 workspace 引擎,未绑定时明确报错引导至query_metric