Skip to content

feat: semantic provider plugin framework (Cube / LookML / MetricFlow → Doris SQL) - #16

Open
Jmmt-mingrui wants to merge 17 commits into
mainfrom
feat/semantic-provider-framework-oss
Open

feat: semantic provider plugin framework (Cube / LookML / MetricFlow → Doris SQL)#16
Jmmt-mingrui wants to merge 17 commits into
mainfrom
feat/semantic-provider-framework-oss

Conversation

@Jmmt-mingrui

Copy link
Copy Markdown
Contributor

概述

本 PR 移植自 apache/doris-mcp-server#182(应要求改投本仓库)。基于 main 分支 cherry-pick,另附 CI workflow commit(对应 apache 侧 #174)。

本 PR 为 Doris MCP Server 引入 Semantic Provider 插件框架——一个 Semantic Model → Doris SQL 的编译器运行时(定位类似 dbt compile + query engine + MCP interface),将语义模型支持从单一内置 MetricFlow 扩展为可插拔的多格式体系。

架构

     Build Time(上传模型)                 Runtime(Agent 查询)

ModelSource(filename, content)        CompiledArtifact + QueryRequest
        │                                     │
SemanticProvider                      SemanticRuntime
  validate() → parse() → compile()      get_metrics() / get_dimensions()
        │                                 generate_sql() → Doris SQL
        ▼                                     │
CompiledArtifact ──► ArtifactStore            ▼
(provider 私有 payload)                 validate_readonly → 连接池执行

关键设计决策

  • 编译产物不跨 Provider 标准化:envelope(provider/name/version/source_digest)统一,payload 各家私有(类比 Java .class / dbt manifest.json)——这是编译器框架,不是语义交换标准。
  • Provider = Parser + Compiler + Runtime SQL Generator + Metadata Provider(类似数据库驱动),而非简单 Parser。
  • 结构化 Filter:Agent 传 {dimension, operator, value} 而非 SQL 片段;名称对 artifact 校验、字面量转义,生成 SQL 构造上免疫注入。

内置 Provider(尽量借用现有开源项目)

Provider 格式 借用
cube cube-js YAML(cubes/measures/dimensions/many_to_one joins) 遵循 cube-js/cube 数据模型规范;MCP 交互参考 isaacwasserman/mcp_cube_server
lookml Looker .view.lkml 解析基于 joshtemple/lkml(MIT,新增依赖);编译时翻译成 Cube artifact 形状,复用 Cube 运行时
metricflow dbt semantic_models YAML 适配器桥接本仓库已 vendor 的 MetricFlow 引擎(bind() 挂接 workspace compiler)

格式自动嗅探(detect() 置信度 ≥0.5 路由)或显式指定 provider。

新增 MCP Tools(7 个,总计 17 个)

list_semantic_providers / compile_semantic_model / list_semantic_artifacts / delete_semantic_artifact / get_semantic_metadata / generate_semantic_sql(干跑)/ query_semantic_model(生成+只读校验+执行)

测试与文档

  • 88 个新增离线单元测试(Cube 28 / LookML 10 / MetricFlow 12 / 注册表+存储 14 / 工具层 12 / 冒烟散见各文件),全部接入 test/run_all_tests.sh --offline;本地 20 项离线套件全绿。
  • DESIGN.md 新增 §6A 完整设计文档;双语 README 更新;examples/semantic-models/ 提供 Cube/LookML 示例(均通过 provider 校验)。

Review 过程发现并已修复

  • bind() 原设计会修改注册表单例 → 跨工作区并发竞态,改为返回新绑定副本(3c52dd8
  • requirements.txt 漏加 lkml(CI/build.sh 不走 pyproject)→ 补齐(28a155d
  • MetricFlow 适配器零覆盖 → 补 12 个测试(3beaef9

已知边界(后续 PR)

  • Cube 跨 cube 指标查询(fact-to-fact join)暂未支持,报错信息明确
  • LookML explores/joins、derived_table 暂不支持(v1 仅 sql_table_name 视图)
  • MetricFlow provider 的 generate_sql 需要绑定 workspace 引擎,未绑定时明确报错引导至 query_metric

Introduce the plugin SPI for semantic model formats, split into a
build-time half (validate/parse/compile -> CompiledArtifact) and a
runtime half (get_metrics/get_dimensions/generate_sql).

The compiled artifact uses a standardized envelope (provider, version,
source digest) with a deliberately provider-opaque payload — this is a
Semantic Model -> Doris SQL compiler framework, not a semantic exchange
standard.
Cube-schema YAML provider modeled after cube-js/cube data models:
- parser: cubes with measures (count/count_distinct/sum/avg/min/max/number),
  dimensions (string/number/time/boolean) and many_to_one joins
- compiler: pre-resolved artifact with metric/dimension indexes and a
  source digest — queries never re-parse YAML
- runtime: injection-safe Doris SQL generation (validated names, escaped
  literals, structured filters, join resolution, GROUP BY/ORDER BY/LIMIT)
Uploads are routed by explicit provider name or by confidence-based
content sniffing (detect() >= 0.5). Optional providers (LookML/lkml)
degrade gracefully when their dependencies are missing.
Bridges the existing vendored MetricFlow engine into the provider SPI:
- build time: lightweight dbt semantic_models/metrics YAML parsing for
  validation and metadata preview
- runtime: bind() attaches a live workspace MetricFlowCompiler+manifest
  so generate_sql delegates to the real engine; unbound use fails loudly
Looker .view.lkml support built on the MIT-licensed lkml parser:
- views with sql_table_name, dimensions, dimension_groups (expanded to
  Doris date_trunc expressions) and measures (count/sum/average/...)
- ${TABLE}.x / ${x} references normalized at parse time; cross-view
  references and derived tables rejected with clear errors
- compiler translates views into the Cube artifact shape so the shared
  Cube SQL runtime is reused instead of reimplemented

Adds lkml>=1.3 as a dependency (MIT).
File-backed store under <workspace>/.artifacts/ with a provenance
envelope (provider, source filename, digest, created_at), path-traversal
safe ids, and corruption-tolerant listing.
Transport-agnostic implementations of the model lifecycle pipeline
(upload/validate/compile/store artifact, metadata discovery, SQL
generation, and execute-through-query) over the provider SPI, reusing
the existing read-only SQL validator and response envelope.
list_semantic_providers, compile_semantic_model, list_semantic_artifacts,
delete_semantic_artifact, get_semantic_metadata, generate_semantic_sql
(dry-run) and query_semantic_model (generate + execute), with the same
auth/audit/workspace handling as the existing metric-layer tools.
Cube provider (28): detection, validation errors, compilation, SQL
generation incl. join resolution, filter escaping/injection safety.
LookML provider (10): dimension_group expansion, reference rules,
shared-runtime SQL generation.
Registry + artifact store (14): routing, unknown providers, store CRUD,
path traversal and corruption handling.
Provider tools (12): lifecycle, metadata, dry-run SQL, fake-pool query.
Architecture, the no-unified-artifact-format decision, built-in provider
matrix, structured-filter injection safety, the 7 new MCP tools, and an
end-to-end agent example. Also updates the tool count to 17.
Mutating the registry singleton would race across concurrent workspaces;
bind() now returns a new bound adapter and the registered instance stays
stateless (found in review).
CI and build.sh install from requirements.txt, not pyproject — without
this the LookML provider tests would fail in CI (found in review).
Offline unit tests plus a linux-x64 build check on push/PR to main.
Build outputs are discarded — no artifacts uploaded, no releases.
(same workflow as apache/doris-mcp-server#174, comments in English)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant