diff --git a/README.md b/README.md index 75180640..bd31b67b 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ DataFoundry 0.2 turns the first usable workbench into a more complete, stateful - **Branchable, concurrent analysis** — Keep multiple sessions running, queue follow-up prompts, restore completed work, and branch from an earlier question or checkpoint without overwriting the original path. - **Evidence-first follow-ups** — Reference a complete output or a selected table/text region in the next question; resolved evidence is carried into the governed run context with diagnostics. -- **Semantic trace and first-party Data Link** — Inspect checkpoint-backed run structure in a semantic Trace DAG. With our newly open-sourced [Data Link](https://github.com/datagallery-lab/datalink), connect tables and columns to business concepts, entities, joinable paths, and confidence-scored relationships for stronger agent grounding. +- **Semantic trace and DataLink integration** — Inspect checkpoint-backed run structure in a semantic Trace DAG. Connect an external DataLink service to map tables and columns to business concepts, entities, joinable paths, and confidence-scored relationships for stronger agent grounding. - **Reusable outputs and workspace assets** — Preview and export tables, charts, reports, SQL, and files; upload files into an active session, then promote supported files for reuse across sessions. - **Production-facing Web foundation** — Built-in password authentication, same-origin API proxying, bilingual UI, model connection tests, onboarding, and an auto-provisioned DTC growth analysis case. @@ -75,7 +75,7 @@ Formal mode has two paths (do **not** run `npm run dev`). Docker / Compose is no ### Recommended: Ubuntu / Debian one-click -`./deploy.sh` generates configuration, installs dependencies, builds (Web, API, and TUI), and starts Web + API as a detached background process — closing the terminal does not stop the stack. The TUI is built and ready but is a foreground client: start it in another terminal with `./deploy.sh tui` (or `npm run start:tui`); it does **not** stay running with the stack. Semantic Data Link is an **external** component (not started by deploy) — connect it later via MCP in the Web UI if needed. No model key is required during deploy — create and enable a model profile in the Web UI after login. Does **not** support native Windows / macOS. +`./deploy.sh` generates configuration, installs dependencies, builds (Web, API, and TUI), and starts Web + API as a detached background process — closing the terminal does not stop the stack. The TUI is built and ready but is a foreground client: start it in another terminal with `./deploy.sh tui` (or `npm run start:tui`); it does **not** stay running with the stack. DataLink is an **external** component (not started by deploy) — connect it later via MCP in the Web UI if needed. No model key is required during deploy — create and enable a model profile in the Web UI after login. Does **not** support native Windows / macOS. ```bash git clone https://github.com/datagallery-lab/datafoundry.git @@ -154,7 +154,7 @@ You will see the full chain: schema inspection → read-only SQL → SQL audit For real production, also configure SMTP and a reverse proxy: [`deploy/nginx.datafoundry.conf.example`](deploy/nginx.datafoundry.conf.example) (gzip/brotli static assets; keep `/api/copilotkit` uncompressed and unbuffered for SSE). -> Full steps, the two-environment matrix, and optional external Data Link notes: [Quick Start](docs/en/quick-start.md). Contributor hot-reload (`npm run dev`) is appendix-only. Connect your own PostgreSQL / MySQL / CSV and more: [Data Sources guide](docs/en/guides/data-sources.md). +> Full steps, the two-environment matrix, and optional external DataLink notes: [Quick Start](docs/en/quick-start.md). Contributor hot-reload (`npm run dev`) is appendix-only. Connect your own PostgreSQL / MySQL / CSV and more: [Data Sources guide](docs/en/guides/data-sources.md). ## 🆚 How It Differs From Coding Agents And SQL Chatbots diff --git a/README_zh.md b/README_zh.md index dc52939d..a3878b4b 100644 --- a/README_zh.md +++ b/README_zh.md @@ -63,7 +63,7 @@ DataFoundry 0.2 在首个可用版本上,进一步补齐了有状态、可追 - **可分支的并发分析** — 多个会话可同时运行,运行中可排队后续问题;恢复历史后,可从早期问题或 checkpoint 创建新分支,不覆盖原分析路径。 - **证据驱动的追问** — 可将完整产出,或选中的表格区域、文本片段引用到下一个问题;证据解析结果和诊断信息会进入受控 run context。 -- **语义 Trace 与自研 Data Link** — 通过基于 checkpoint 的语义 Trace DAG 复盘执行结构;结合我们新近开源的 [Data Link](https://github.com/datagallery-lab/datalink),把表和字段连接到业务概念、实体、可 JOIN 路径和带置信度的关系,为 Agent 提供更强语义支撑。 +- **语义 Trace 与 DataLink 集成** — 通过基于 checkpoint 的语义 Trace DAG 复盘执行结构;连接外部 DataLink 服务,把表和字段映射到业务概念、实体、可 JOIN 路径和带置信度的关系,为 Agent 提供更强语义支撑。 - **可复用的产出与工作区资产** — 统一预览和导出表格、图表、报告、SQL 和文件;文件可先上传到当前会话,再提升为跨会话复用的工作区资产。 - **面向正式部署的 Web 基础** — 内置密码认证、同源 API 代理、中英双语界面、模型连接测试、首次引导,并自动准备 DTC 增长分析案例。 @@ -75,7 +75,7 @@ DataFoundry 0.2 在首个可用版本上,进一步补齐了有状态、可追 ### 推荐:Ubuntu / Debian 一键部署 -`./deploy.sh` 自动生成配置、安装依赖、构建(含 Web、API 与 TUI),并以 detached 后台进程启动 Web + API——关闭终端一般不会停止服务。TUI 会在部署时构建就绪,但它是前台交互客户端:请另开终端执行 `./deploy.sh tui`(或 `npm run start:tui`),**不会**随 stack 后台常驻。语义 Data Link 为**外置**组件(deploy 不会安装或启动)——如需使用,稍后在 Web 的 MCP 配置中连接外部服务即可。部署时不要求填写模型 Key——登录后在 Web 中创建并启用模型即可。**不支持**原生 Windows / macOS。 +`./deploy.sh` 自动生成配置、安装依赖、构建(含 Web、API 与 TUI),并以 detached 后台进程启动 Web + API——关闭终端一般不会停止服务。TUI 会在部署时构建就绪,但它是前台交互客户端:请另开终端执行 `./deploy.sh tui`(或 `npm run start:tui`),**不会**随 stack 后台常驻。DataLink 为**外置**组件(deploy 不会安装或启动)——如需使用,稍后在 Web 的 MCP 配置中连接外部服务即可。部署时不要求填写模型 Key——登录后在 Web 中创建并启用模型即可。**不支持**原生 Windows / macOS。 ```bash git clone https://github.com/datagallery-lab/datafoundry.git diff --git a/docs/en/guides/datalink.md b/docs/en/guides/datalink.md new file mode 100644 index 00000000..fb74333c --- /dev/null +++ b/docs/en/guides/datalink.md @@ -0,0 +1,52 @@ +# DataLink semantic service + +DataLink is an optional semantic graph service for DataFoundry. It connects schemas and profiles to business concepts, entities, join paths, and confidence-scored relationships. The implementation is included under `services/datalink`, but it runs as a separate Python service. + +## Runtime topology + +| Process | Default endpoint | Purpose | +| --- | --- | --- | +| DataLink MCP | `http://127.0.0.1:8080/mcp` | Exposes `datalink_explore` for agent grounding | +| DataLink REST | `http://127.0.0.1:8081` | Graph management and visualization API | + +DataFoundry's Web and API processes remain unchanged. DataLink is not started by `deploy.sh` or `npm run start`; start it separately when semantic grounding is needed. + +## Install and start + +From the service directory, install Python 3.10+ and [uv](https://docs.astral.sh/uv/): + +```bash +cd services/datalink +uv pip install -e . +cp datalink_config.example.json datalink_config.json +``` + +Set the LLM and embedding provider in `datalink_config.json`, then start both processes in separate terminals: + +```bash +uv run datalink serve --port 8080 --transport streamable-http +uv run datalink api --port 8081 +``` + +The service README contains provider examples and CLI commands for building and exploring a graph. + +## Connect it in DataFoundry + +In the Web workbench, open MCP settings and add an external server with: + +| Field | Example | +| --- | --- | +| `serverUrl` | `http://127.0.0.1:8080/mcp` | +| `apiUrl` | `http://127.0.0.1:8081` | +| `transport` | `streamable-http` | +| `toolManifest` | `[{ "name": "datalink_explore" }]` | + +Use a name or id containing `datalink` so the DataLink panel recognizes the server. + +## Verify + +```bash +curl http://127.0.0.1:8081/healthz +``` + +If the panel reports the service as unavailable, check both processes, ports `8080` and `8081`, the configured graph path, and the MCP transport. Keep API keys in environment variables or a secret manager; do not commit them to `datalink_config.json`. diff --git a/docs/en/quick-start.md b/docs/en/quick-start.md index e7485d6c..7a69d451 100644 --- a/docs/en/quick-start.md +++ b/docs/en/quick-start.md @@ -13,7 +13,7 @@ After deploy, configure a model in the Web UI and run an analysis against the bu - **One-click deploy**: Ubuntu or Debian (x86_64 / aarch64); Node.js 22 (the script can help install it after consent) - **Manual npm**: Linux, macOS, or Windows; Node.js >= 22 and npm -- Optional external [Data Link](https://github.com/datagallery-lab/datalink): run it as a separate process if you want semantic graph features (not required for deploy) +- Optional external [DataLink](guides/datalink.md): run it as a separate process if you want semantic graph features (not required for deploy) Install and run the project in the same environment. On Windows, do not share `node_modules` between Windows and WSL. @@ -64,11 +64,11 @@ Open `http://127.0.0.1:3000/login` (or the Web URL printed by the script if the `LLM_*` is not required during deploy. Set `AUTH_PUBLIC_BASE_URL` for remote hosts. Re-running deploy uses a maintenance window: stop the managed process group before `npm ci` and builds. -### External Data Link (optional) +### External DataLink (optional) -One-click deploy starts **only** Web + API. It does **not** install, start, or health-check Data Link. +One-click deploy starts **only** Web + API. It does **not** install, start, or health-check DataLink. -To use semantic features, run [Data Link](https://github.com/datagallery-lab/datalink) as a separate service (typically MCP on `:8080` and REST on `:8081`), then in the Web workbench MCP settings add an external server, for example: +To use semantic features, run [DataLink](guides/datalink.md) as a separate service (typically MCP on `:8080` and REST on `:8081`), then in the Web workbench MCP settings add an external server, for example: | Field | Example | | --- | --- | @@ -77,7 +77,7 @@ To use semantic features, run [Data Link](https://github.com/datagallery-lab/dat | `transport` | `streamable-http` | | `toolManifest` | `[{ "name": "datalink_explore" }]` | -Use a name/id containing `datalink` (or `datagraph`) so the Data Link panel can recognize it. +Use a name/id containing `datalink` so the DataLink panel can recognize it. ## Windows / macOS / other: manual npm deploy diff --git a/docs/javascripts/language-switcher.js b/docs/javascripts/language-switcher.js index 80e05c48..a81c844d 100644 --- a/docs/javascripts/language-switcher.js +++ b/docs/javascripts/language-switcher.js @@ -105,6 +105,7 @@ ["Web workbench", "Web 工作台"], ["TUI", "TUI"], ["Data sources", "数据源"], + ["DataLink", "DataLink"], ["Examples", "案例"], ["DTC growth operating review", "DTC 增长经营复盘"], ["Developer Guide", "开发者指南"], diff --git a/docs/zh/guides/datalink.md b/docs/zh/guides/datalink.md new file mode 100644 index 00000000..853172db --- /dev/null +++ b/docs/zh/guides/datalink.md @@ -0,0 +1,52 @@ +# DataLink 语义服务 + +DataLink 是 DataFoundry 的可选语义图服务,用于将 Schema 与数据画像连接到业务概念、实体、可 JOIN 路径和带置信度的关系。实现位于 `services/datalink`,但作为独立 Python 服务运行。 + +## 运行拓扑 + +| 进程 | 默认地址 | 用途 | +| --- | --- | --- | +| DataLink MCP | `http://127.0.0.1:8080/mcp` | 提供 `datalink_explore`,为 Agent 提供语义上下文 | +| DataLink REST | `http://127.0.0.1:8081` | 图谱管理与可视化 API | + +DataFoundry 的 Web 与 API 进程不受影响。`deploy.sh` 和 `npm run start` 不会启动 DataLink;需要语义增强时请单独启动。 + +## 安装与启动 + +进入服务目录,准备 Python 3.10+ 与 [uv](https://docs.astral.sh/uv/): + +```bash +cd services/datalink +uv pip install -e . +cp datalink_config.example.json datalink_config.json +``` + +在 `datalink_config.json` 中配置 LLM 与 embedding 提供商,然后在两个终端分别启动: + +```bash +uv run datalink serve --port 8080 --transport streamable-http +uv run datalink api --port 8081 +``` + +提供商示例、建图和探索命令见服务目录下的 README。 + +## 在 DataFoundry 中连接 + +在 Web 工作台打开 MCP 设置,添加外部服务: + +| 字段 | 示例 | +| --- | --- | +| `serverUrl` | `http://127.0.0.1:8080/mcp` | +| `apiUrl` | `http://127.0.0.1:8081` | +| `transport` | `streamable-http` | +| `toolManifest` | `[{ "name": "datalink_explore" }]` | + +名称或 id 中包含 `datalink` 时,DataLink 面板会识别该服务。 + +## 验证 + +```bash +curl http://127.0.0.1:8081/healthz +``` + +如果面板显示服务不可用,请检查两个进程、`8080`/`8081` 端口、图数据库路径和 MCP transport。API Key 应放在环境变量或 Secret 管理系统中,不要提交到 `datalink_config.json`。 diff --git a/docs/zh/quick-start.md b/docs/zh/quick-start.md index 26daae26..c677ff77 100644 --- a/docs/zh/quick-start.md +++ b/docs/zh/quick-start.md @@ -13,7 +13,7 @@ - **一键部署**:Ubuntu 或 Debian(x86_64 / aarch64);Node.js 22(缺失时脚本可在确认后协助安装) - **手动 npm**:Linux、macOS 或 Windows;Node.js >= 22 与 npm -- 可选外置 [Data Link](https://github.com/datagallery-lab/datalink):需要语义图谱能力时单独运行(一键部署不依赖) +- 可选外置 [DataLink](guides/datalink.md):需要语义图谱能力时单独运行(一键部署不依赖) 请在同一环境内安装和运行项目。Windows 用户不要在 Windows 和 WSL 之间共用 `node_modules`。 @@ -64,11 +64,11 @@ cd datafoundry 部署时不要求填写 `LLM_*`。远程主机请设置 `AUTH_PUBLIC_BASE_URL`。重复部署会进入维护窗口:先停止受管进程组,再执行 `npm ci` 与构建。 -### 外置 Data Link(可选) +### 外置 DataLink(可选) -一键部署**只启动** Web + API,**不会**安装、启动或健康检查 Data Link。 +一键部署**只启动** Web + API,**不会**安装、启动或健康检查 DataLink。 -若需要语义能力,请单独运行 [Data Link](https://github.com/datagallery-lab/datalink)(常见为 MCP `:8080` + REST `:8081`),再在 Web 工作台的 MCP 设置中添加外部服务,例如: +若需要语义能力,请单独运行 [DataLink](guides/datalink.md)(常见为 MCP `:8080` + REST `:8081`),再在 Web 工作台的 MCP 设置中添加外部服务,例如: | 字段 | 示例 | | --- | --- | @@ -77,7 +77,7 @@ cd datafoundry | `transport` | `streamable-http` | | `toolManifest` | `[{ "name": "datalink_explore" }]` | -名称/id 包含 `datalink`(或 `datagraph`)即可被 Data Link 面板识别。 +名称/id 包含 `datalink` 即可被 DataLink 面板识别。 ## Windows / macOS / 其他:手动 npm 部署 diff --git a/mkdocs.yml b/mkdocs.yml index e4aebf9d..b468d139 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -74,6 +74,7 @@ nav: - Web workbench: en/guides/web-workbench.md - TUI: en/guides/tui.md - Data sources: en/guides/data-sources.md + - DataLink: en/guides/datalink.md - Examples: - DTC growth operating review: en/examples/dtc-growth-demo.md - Developer Guide: