Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ DataFoundry 0.2 turns the first usable workbench into a more complete, stateful

- **Branchable, concurrent analysis** — Keep multiple sessions running, queue follow-up prompts, restore completed work, and branch from an earlier question or checkpoint without overwriting the original path.
- **Evidence-first follow-ups** — Reference a complete output or a selected table/text region in the next question; resolved evidence is carried into the governed run context with diagnostics.
- **Semantic trace and first-party Data Link** — Inspect checkpoint-backed run structure in a semantic Trace DAG. With our newly open-sourced [Data Link](https://github.com/datagallery-lab/datalink), connect tables and columns to business concepts, entities, joinable paths, and confidence-scored relationships for stronger agent grounding.
- **Semantic trace and DataLink integration** — Inspect checkpoint-backed run structure in a semantic Trace DAG. Connect an external DataLink service to map tables and columns to business concepts, entities, joinable paths, and confidence-scored relationships for stronger agent grounding.
- **Reusable outputs and workspace assets** — Preview and export tables, charts, reports, SQL, and files; upload files into an active session, then promote supported files for reuse across sessions.
- **Production-facing Web foundation** — Built-in password authentication, same-origin API proxying, bilingual UI, model connection tests, onboarding, and an auto-provisioned DTC growth analysis case.

Expand All @@ -75,7 +75,7 @@ Formal mode has two paths (do **not** run `npm run dev`). Docker / Compose is no

### Recommended: Ubuntu / Debian one-click

`./deploy.sh` generates configuration, installs dependencies, builds (Web, API, and TUI), and starts Web + API as a detached background process — closing the terminal does not stop the stack. The TUI is built and ready but is a foreground client: start it in another terminal with `./deploy.sh tui` (or `npm run start:tui`); it does **not** stay running with the stack. Semantic Data Link is an **external** component (not started by deploy) — connect it later via MCP in the Web UI if needed. No model key is required during deploy — create and enable a model profile in the Web UI after login. Does **not** support native Windows / macOS.
`./deploy.sh` generates configuration, installs dependencies, builds (Web, API, and TUI), and starts Web + API as a detached background process — closing the terminal does not stop the stack. The TUI is built and ready but is a foreground client: start it in another terminal with `./deploy.sh tui` (or `npm run start:tui`); it does **not** stay running with the stack. DataLink is an **external** component (not started by deploy) — connect it later via MCP in the Web UI if needed. No model key is required during deploy — create and enable a model profile in the Web UI after login. Does **not** support native Windows / macOS.

```bash
git clone https://github.com/datagallery-lab/datafoundry.git
Expand Down Expand Up @@ -154,7 +154,7 @@ You will see the full chain: schema inspection → read-only SQL → SQL audit

For real production, also configure SMTP and a reverse proxy: [`deploy/nginx.datafoundry.conf.example`](deploy/nginx.datafoundry.conf.example) (gzip/brotli static assets; keep `/api/copilotkit` uncompressed and unbuffered for SSE).

> Full steps, the two-environment matrix, and optional external Data Link notes: [Quick Start](docs/en/quick-start.md). Contributor hot-reload (`npm run dev`) is appendix-only. Connect your own PostgreSQL / MySQL / CSV and more: [Data Sources guide](docs/en/guides/data-sources.md).
> Full steps, the two-environment matrix, and optional external DataLink notes: [Quick Start](docs/en/quick-start.md). Contributor hot-reload (`npm run dev`) is appendix-only. Connect your own PostgreSQL / MySQL / CSV and more: [Data Sources guide](docs/en/guides/data-sources.md).


## 🆚 How It Differs From Coding Agents And SQL Chatbots
Expand Down
4 changes: 2 additions & 2 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ DataFoundry 0.2 在首个可用版本上,进一步补齐了有状态、可追

- **可分支的并发分析** — 多个会话可同时运行,运行中可排队后续问题;恢复历史后,可从早期问题或 checkpoint 创建新分支,不覆盖原分析路径。
- **证据驱动的追问** — 可将完整产出,或选中的表格区域、文本片段引用到下一个问题;证据解析结果和诊断信息会进入受控 run context。
- **语义 Trace 与自研 Data Link** — 通过基于 checkpoint 的语义 Trace DAG 复盘执行结构;结合我们新近开源的 [Data Link](https://github.com/datagallery-lab/datalink),把表和字段连接到业务概念、实体、可 JOIN 路径和带置信度的关系,为 Agent 提供更强语义支撑。
- **语义 Trace 与 DataLink 集成** — 通过基于 checkpoint 的语义 Trace DAG 复盘执行结构;连接外部 DataLink 服务,把表和字段映射到业务概念、实体、可 JOIN 路径和带置信度的关系,为 Agent 提供更强语义支撑。
- **可复用的产出与工作区资产** — 统一预览和导出表格、图表、报告、SQL 和文件;文件可先上传到当前会话,再提升为跨会话复用的工作区资产。
- **面向正式部署的 Web 基础** — 内置密码认证、同源 API 代理、中英双语界面、模型连接测试、首次引导,并自动准备 DTC 增长分析案例。

Expand All @@ -75,7 +75,7 @@ DataFoundry 0.2 在首个可用版本上,进一步补齐了有状态、可追

### 推荐:Ubuntu / Debian 一键部署

`./deploy.sh` 自动生成配置、安装依赖、构建(含 Web、API 与 TUI),并以 detached 后台进程启动 Web + API——关闭终端一般不会停止服务。TUI 会在部署时构建就绪,但它是前台交互客户端:请另开终端执行 `./deploy.sh tui`(或 `npm run start:tui`),**不会**随 stack 后台常驻。语义 Data Link 为**外置**组件(deploy 不会安装或启动)——如需使用,稍后在 Web 的 MCP 配置中连接外部服务即可。部署时不要求填写模型 Key——登录后在 Web 中创建并启用模型即可。**不支持**原生 Windows / macOS。
`./deploy.sh` 自动生成配置、安装依赖、构建(含 Web、API 与 TUI),并以 detached 后台进程启动 Web + API——关闭终端一般不会停止服务。TUI 会在部署时构建就绪,但它是前台交互客户端:请另开终端执行 `./deploy.sh tui`(或 `npm run start:tui`),**不会**随 stack 后台常驻。DataLink 为**外置**组件(deploy 不会安装或启动)——如需使用,稍后在 Web 的 MCP 配置中连接外部服务即可。部署时不要求填写模型 Key——登录后在 Web 中创建并启用模型即可。**不支持**原生 Windows / macOS。

```bash
git clone https://github.com/datagallery-lab/datafoundry.git
Expand Down
52 changes: 52 additions & 0 deletions docs/en/guides/datalink.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# DataLink semantic service

DataLink is an optional semantic graph service for DataFoundry. It connects schemas and profiles to business concepts, entities, join paths, and confidence-scored relationships. The implementation is included under `services/datalink`, but it runs as a separate Python service.

## Runtime topology

| Process | Default endpoint | Purpose |
| --- | --- | --- |
| DataLink MCP | `http://127.0.0.1:8080/mcp` | Exposes `datalink_explore` for agent grounding |
| DataLink REST | `http://127.0.0.1:8081` | Graph management and visualization API |

DataFoundry's Web and API processes remain unchanged. DataLink is not started by `deploy.sh` or `npm run start`; start it separately when semantic grounding is needed.

## Install and start

From the service directory, install Python 3.10+ and [uv](https://docs.astral.sh/uv/):

```bash
cd services/datalink
uv pip install -e .
cp datalink_config.example.json datalink_config.json
```

Set the LLM and embedding provider in `datalink_config.json`, then start both processes in separate terminals:

```bash
uv run datalink serve --port 8080 --transport streamable-http
uv run datalink api --port 8081
```

The service README contains provider examples and CLI commands for building and exploring a graph.

## Connect it in DataFoundry

In the Web workbench, open MCP settings and add an external server with:

| Field | Example |
| --- | --- |
| `serverUrl` | `http://127.0.0.1:8080/mcp` |
| `apiUrl` | `http://127.0.0.1:8081` |
| `transport` | `streamable-http` |
| `toolManifest` | `[{ "name": "datalink_explore" }]` |

Use a name or id containing `datalink` (or `datagraph`) so the DataLink panel recognizes the server.

## Verify

```bash
curl http://127.0.0.1:8081/healthz
```

If the panel reports the service as unavailable, check both processes, ports `8080` and `8081`, the configured graph path, and the MCP transport. Keep API keys in environment variables or a secret manager; do not commit them to `datalink_config.json`.
10 changes: 5 additions & 5 deletions docs/en/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ After deploy, configure a model in the Web UI and run an analysis against the bu

- **One-click deploy**: Ubuntu or Debian (x86_64 / aarch64); Node.js 22 (the script can help install it after consent)
- **Manual npm**: Linux, macOS, or Windows; Node.js >= 22 and npm
- Optional external [Data Link](https://github.com/datagallery-lab/datalink): run it as a separate process if you want semantic graph features (not required for deploy)
- Optional external [DataLink](guides/datalink.md): run it as a separate process if you want semantic graph features (not required for deploy)

Install and run the project in the same environment. On Windows, do not share `node_modules` between Windows and WSL.

Expand Down Expand Up @@ -64,11 +64,11 @@ Open `http://127.0.0.1:3000/login` (or the Web URL printed by the script if the

`LLM_*` is not required during deploy. Set `AUTH_PUBLIC_BASE_URL` for remote hosts. Re-running deploy uses a maintenance window: stop the managed process group before `npm ci` and builds.

### External Data Link (optional)
### External DataLink (optional)

One-click deploy starts **only** Web + API. It does **not** install, start, or health-check Data Link.
One-click deploy starts **only** Web + API. It does **not** install, start, or health-check DataLink.

To use semantic features, run [Data Link](https://github.com/datagallery-lab/datalink) as a separate service (typically MCP on `:8080` and REST on `:8081`), then in the Web workbench MCP settings add an external server, for example:
To use semantic features, run [DataLink](guides/datalink.md) as a separate service (typically MCP on `:8080` and REST on `:8081`), then in the Web workbench MCP settings add an external server, for example:

| Field | Example |
| --- | --- |
Expand All @@ -77,7 +77,7 @@ To use semantic features, run [Data Link](https://github.com/datagallery-lab/dat
| `transport` | `streamable-http` |
| `toolManifest` | `[{ "name": "datalink_explore" }]` |

Use a name/id containing `datalink` (or `datagraph`) so the Data Link panel can recognize it.
Use a name/id containing `datalink` (or `datagraph`) so the DataLink panel can recognize it.

## Windows / macOS / other: manual npm deploy

Expand Down
1 change: 1 addition & 0 deletions docs/javascripts/language-switcher.js
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@
["Web workbench", "Web 工作台"],
["TUI", "TUI"],
["Data sources", "数据源"],
["DataLink", "DataLink"],
["Examples", "案例"],
["DTC growth operating review", "DTC 增长经营复盘"],
["Developer Guide", "开发者指南"],
Expand Down
52 changes: 52 additions & 0 deletions docs/zh/guides/datalink.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# DataLink 语义服务

DataLink 是 DataFoundry 的可选语义图服务,用于将 Schema 与数据画像连接到业务概念、实体、可 JOIN 路径和带置信度的关系。实现位于 `services/datalink`,但作为独立 Python 服务运行。

## 运行拓扑

| 进程 | 默认地址 | 用途 |
| --- | --- | --- |
| DataLink MCP | `http://127.0.0.1:8080/mcp` | 提供 `datalink_explore`,为 Agent 提供语义上下文 |
| DataLink REST | `http://127.0.0.1:8081` | 图谱管理与可视化 API |

DataFoundry 的 Web 与 API 进程不受影响。`deploy.sh` 和 `npm run start` 不会启动 DataLink;需要语义增强时请单独启动。

## 安装与启动

进入服务目录,准备 Python 3.10+ 与 [uv](https://docs.astral.sh/uv/):

```bash
cd services/datalink
uv pip install -e .
cp datalink_config.example.json datalink_config.json
```

在 `datalink_config.json` 中配置 LLM 与 embedding 提供商,然后在两个终端分别启动:

```bash
uv run datalink serve --port 8080 --transport streamable-http
uv run datalink api --port 8081
```

提供商示例、建图和探索命令见服务目录下的 README。

## 在 DataFoundry 中连接

在 Web 工作台打开 MCP 设置,添加外部服务:

| 字段 | 示例 |
| --- | --- |
| `serverUrl` | `http://127.0.0.1:8080/mcp` |
| `apiUrl` | `http://127.0.0.1:8081` |
| `transport` | `streamable-http` |
| `toolManifest` | `[{ "name": "datalink_explore" }]` |

名称或 id 中包含 `datalink`(或 `datagraph`)时,DataLink 面板会识别该服务。

## 验证

```bash
curl http://127.0.0.1:8081/healthz
```

如果面板显示服务不可用,请检查两个进程、`8080`/`8081` 端口、图数据库路径和 MCP transport。API Key 应放在环境变量或 Secret 管理系统中,不要提交到 `datalink_config.json`。
10 changes: 5 additions & 5 deletions docs/zh/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

- **一键部署**:Ubuntu 或 Debian(x86_64 / aarch64);Node.js 22(缺失时脚本可在确认后协助安装)
- **手动 npm**:Linux、macOS 或 Windows;Node.js >= 22 与 npm
- 可选外置 [Data Link](https://github.com/datagallery-lab/datalink):需要语义图谱能力时单独运行(一键部署不依赖)
- 可选外置 [DataLink](guides/datalink.md):需要语义图谱能力时单独运行(一键部署不依赖)

请在同一环境内安装和运行项目。Windows 用户不要在 Windows 和 WSL 之间共用 `node_modules`。

Expand Down Expand Up @@ -64,11 +64,11 @@ cd datafoundry

部署时不要求填写 `LLM_*`。远程主机请设置 `AUTH_PUBLIC_BASE_URL`。重复部署会进入维护窗口:先停止受管进程组,再执行 `npm ci` 与构建。

### 外置 Data Link(可选)
### 外置 DataLink(可选)

一键部署**只启动** Web + API,**不会**安装、启动或健康检查 Data Link
一键部署**只启动** Web + API,**不会**安装、启动或健康检查 DataLink

若需要语义能力,请单独运行 [Data Link](https://github.com/datagallery-lab/datalink)(常见为 MCP `:8080` + REST `:8081`),再在 Web 工作台的 MCP 设置中添加外部服务,例如:
若需要语义能力,请单独运行 [DataLink](guides/datalink.md)(常见为 MCP `:8080` + REST `:8081`),再在 Web 工作台的 MCP 设置中添加外部服务,例如:

| 字段 | 示例 |
| --- | --- |
Expand All @@ -77,7 +77,7 @@ cd datafoundry
| `transport` | `streamable-http` |
| `toolManifest` | `[{ "name": "datalink_explore" }]` |

名称/id 包含 `datalink`(或 `datagraph`)即可被 Data Link 面板识别。
名称/id 包含 `datalink`(或 `datagraph`)即可被 DataLink 面板识别。

## Windows / macOS / 其他:手动 npm 部署

Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ nav:
- Web workbench: en/guides/web-workbench.md
- TUI: en/guides/tui.md
- Data sources: en/guides/data-sources.md
- DataLink: en/guides/datalink.md
- Examples:
- DTC growth operating review: en/examples/dtc-growth-demo.md
- Developer Guide:
Expand Down