Skip to content

feat: Support MySQL to SAP HANA Data Synchronization (#306) #309

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ljluestc wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

feat: Support MySQL to SAP HANA Data Synchronization (#306) #309

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
287790b
feat: Support MySQL to SAP HANA Data Synchronization (#306)
ljluestc Jan 19, 2026
bf21645
feat: Support MySQL to SAP HANA Data Synchronization (#306)
ljluestc Jan 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions PR_README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Feature: Support MySQL to SAP HANA Data Synchronization (支持 MySQL 到 SAP HANA 数据同步)

## 🇨🇳 中文详细说明 (Chinese Description)

### 1. 摘要 (Summary)
本 PR 引入了一个全新的插件 `hana`，旨在支持从 MySQL 到 SAP HANA 的高性能实时数据同步，彻底解决了 issue #306。该插件基于 SAP 官方的 `go-hdb` 驱动开发，并针对 Bifrost 的架构实现了高效的事务性批量处理机制，能够满足企业级数据同步需求。

### 2. 核心功能与变更 (Key Features & Changes)

* **原生驱动支持**: 新增 `plugin/hana` 目录，核心逻辑直接基于 `github.com/SAP/go-hdb/driver` 实现，无需中间件。
* **全量 DML 支持**: 完整支持 `INSERT`、`UPDATE`、`DELETE` 事件的同步，并针对 HANA 语法进行了适配。
* **智能批量处理**:
* 内置可配置的事件缓冲池（`BatchSize`，默认 500）。
* 能够自动将高频的小事务合并为批量事务提交，显著减少网络 RTT 和数据库 IOPS。
* **高可靠性事务设计**:
* **按表分组**: 提交时自动将同表数据归类，每个表的数据在一个独立的 `Transition` 中执行。
* **原子性保障**: 使用 `Begin()` 和 `Commit()`/`Rollback()` 确保批次数据的完整性，任一记录失败均会触发回滚并报错，避免数据不一致。
* **健壮的类型映射**:
* 自动处理 Go 语言与 HANA 数据库的类型转换。
* 复杂结构（如 JSON、Map、Slice）自动序列化为字符串存储。
* 支持 `json.Number` 高精度数值。

### 3. 配置指南 (Configuration)

在 Bifrost 管理界面添加目标库时，请使用以下格式的 URI：

```text
hdb://user:password@host:port?schema=TARGET_SCHEMA
```

**参数说明**:
* `user`: HANA 数据库用户名
* `password`: 密码
* `host`: 数据库地址
* `port`: 数据库端口 (通常为 3xx15)
* `schema`: (可选) 指定同步到的目标 Schema。如果不填，默认使用源库名或 SQL 中的 Schema。

**高级参数 (Plugin Param)**:
* `BatchSize`: 批量提交的大小。默认为 `500`。增大此值可提高吞吐量，但会增加延迟。

### 4. 测试与验证 (How to Test)

#### 4.1 单元测试 (Unit Test)
本项目包含完整的单元测试，用于验证 URI 解析、参数校验及基本逻辑。

```bash
# 在项目根目录下运行
go test -v ./plugin/hana/src/...
```

#### 4.2 集成测试 (Integration Test)
1. **环境准备**: 准备一个可访问的 SAP HANA 实例。
2. **Bifrost 配置**:
* 启动 Bifrost。
* 在 "To Server" 中添加 HANA 目标：`hdb://system:Password123@192.168.1.100:39015`。
* 创建一个同步通道，选择 MySQL 源表和 HANA 目标表。
3. **功能验证**:
* **Insert**: 在 MySQL 插入 1000 条数据，HANA 端应在数秒内可见（取决于 BatchSize）。
* **Update**: 更新 MySQL 某行数据，验证 HANA 端对应字段变更。
* **Delete**: 删除 MySQL 数据，验证 HANA 端数据消失。
* **重启测试**: 重启 Bifrost，验证断点续传功能是否正常。

### 5. 实现细节 (Implementation Details)

* **文件结构**:
* `plugin/hana/hana.go`: 插件注册入口。
* `plugin/hana/src/hana.go`: 核心连接与同步逻辑实现。
* `plugin/hana/src/hana_test.go`: 单元测试文件。
* `plugin/hana/www/`: 插件 UI 文档静态资源。
* **依赖管理**:
* `go.mod` 已更新，添加了 `github.com/SAP/go-hdb v1.14.18`。
* 已执行 `go mod tidy` 并与上游保持一致。

---

[查看代码变更 (View Changes)](https://github.com/brokercap/Bifrost/compare/master...ljluestc:Bifrost:feature/hana-support?expand=1)
92 changes: 4 additions & 88 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,152 +1,68 @@
module github.com/brokercap/Bifrost

go 1.21.8
go 1.24.0

require (
github.com/ClickHouse/clickhouse-go v1.4.3
github.com/SAP/go-hdb v1.14.18
github.com/Shopify/sarama v1.29.0
github.com/StackExchange/wmi v1.2.1
github.com/agiledragon/gomonkey/v2 v2.11.0
github.com/bradfitz/gomemcache v0.0.0-20190329173943-551aad21a668
github.com/gmallard/stompngo v1.0.11
github.com/go-redis/redis/v8 v8.7.1
github.com/hprose/hprose-golang v2.0.4+incompatible
github.com/juju/errors v0.0.0-20200330140219-3fe23663418f
github.com/olivere/elastic/v7 v7.0.24
github.com/robfig/cron/v3 v3.0.1
github.com/rwynn/gtm/v2 v2.1.2
github.com/satori/go.uuid v1.2.0
github.com/smartystreets/goconvey v1.7.2
github.com/streadway/amqp v0.0.0-20190404075320-75d898a42a94
github.com/syndtr/goleveldb v1.0.0
github.com/xdg/scram v1.0.5
go.mongodb.org/mongo-driver v1.17.2
golang.org/x/sys v0.0.0-20210112080510-489259a85091
gopkg.in/mgo.v2 v2.0.0-20190816093944-a6b53ec6cb22
)

require (
cloud.google.com/go v0.26.0 // indirect
github.com/BurntSushi/toml v0.3.1 // indirect
github.com/Shopify/toxiproxy v2.1.4+incompatible // indirect
github.com/aws/aws-sdk-go v1.38.3 // indirect
github.com/bkaradzic/go-lz4 v1.0.0 // indirect
github.com/cespare/xxhash/v2 v2.1.1 // indirect
github.com/client9/misspell v0.3.4 // indirect
github.com/cloudflare/golz4 v0.0.0-20150217214814-ef862a3cdc58 // indirect
github.com/creack/pty v1.1.9 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/eapache/go-resiliency v1.2.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/fortytw2/leaktest v1.3.0 // indirect
github.com/frankban/quicktest v1.11.3 // indirect
github.com/fsnotify/fsnotify v1.4.9 // indirect
github.com/go-ole/go-ole v1.2.5 // indirect
github.com/go-sql-driver/mysql v1.5.0 // indirect
github.com/go-stack/stack v1.8.0 // indirect
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/golang/mock v1.1.1 // indirect
github.com/golang/protobuf v1.4.3 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 // indirect
github.com/gorilla/securecookie v1.1.1 // indirect
github.com/gorilla/sessions v1.2.1 // indirect
github.com/hashicorp/go-uuid v1.0.2 // indirect
github.com/hpcloud/tail v1.0.0 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.0.0 // indirect
github.com/jcmturner/goidentity/v6 v6.0.1 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.2 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/jmespath/go-jmespath/internal/testify v1.5.1 // indirect
github.com/jmoiron/sqlx v1.2.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/jtolds/gls v4.20.0+incompatible // indirect
github.com/juju/ansiterm v0.0.0-20160907234532-b99631de12cf // indirect
github.com/juju/clock v0.0.0-20190205081909-9c5c9712527c // indirect
github.com/juju/cmd v0.0.0-20171107070456-e74f39857ca0 // indirect
github.com/juju/collections v0.0.0-20200605021417-0d0ec82b7271 // indirect
github.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d // indirect
github.com/juju/httpprof v0.0.0-20141217160036-14bf14c30767 // indirect
github.com/juju/loggo v0.0.0-20200526014432-9ce3a2e09b5e // indirect
github.com/juju/mutex v0.0.0-20171110020013-1fe2a4bf0a3a // indirect
github.com/juju/retry v0.0.0-20180821225755-9058e192b216 // indirect
github.com/juju/testing v0.0.0-20201216035041-2be42bba85f3 // indirect
github.com/juju/utils v0.0.0-20200116185830-d40c2fe10647 // indirect
github.com/juju/utils/v2 v2.0.0-20200923005554-4646bfea2ef1 // indirect
github.com/juju/version v0.0.0-20191219164919-81c1be00b9a6 // indirect
github.com/julienschmidt/httprouter v1.1.1-0.20151013225520-77a895ad01eb // indirect
github.com/kisielk/gotool v1.0.0 // indirect
github.com/klauspost/compress v1.16.7 // indirect
github.com/kr/pretty v0.2.1 // indirect
github.com/kr/pty v1.1.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/lib/pq v1.0.0 // indirect
github.com/lunixbochs/vtclean v0.0.0-20160125035106-4fbf7632a2c6 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/masterzen/azure-sdk-for-go v3.2.0-beta.0.20161014135628-ee4f0065d00c+incompatible // indirect
github.com/masterzen/simplexml v0.0.0-20160608183007-4572e39b1ab9 // indirect
github.com/masterzen/winrm v0.0.0-20161014151040-7a535cd943fc // indirect
github.com/masterzen/xmlpath v0.0.0-20140218185901-13f4951698ad // indirect
github.com/mattn/go-colorable v0.0.6 // indirect
github.com/mattn/go-isatty v0.0.0-20160806122752-66b8e73f3f5c // indirect
github.com/mattn/go-sqlite3 v1.9.0 // indirect
github.com/montanaflynn/stats v0.7.1 // indirect
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect
github.com/nu7hatch/gouuid v0.0.0-20131221200532-179d4d0c4d8d // indirect
github.com/nxadm/tail v1.4.4 // indirect
github.com/onsi/ginkgo v1.15.0 // indirect
github.com/onsi/gomega v1.10.5 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/pierrec/lz4 v2.6.0+incompatible // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/robfig/cron/v3 v3.0.1 // indirect
github.com/serialx/hashring v0.0.0-20200727003509-22c0c7ab6b1b // indirect
github.com/smartystreets/assertions v1.2.0 // indirect
github.com/smartystreets/go-aws-auth v0.0.0-20180515143844-0c1422d1fdb9 // indirect
github.com/smartystreets/gunit v1.4.2 // indirect
github.com/stretchr/objx v0.1.0 // indirect
github.com/stretchr/testify v1.7.0 // indirect
github.com/tidwall/pretty v1.0.0 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/scram v1.1.2 // indirect
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/xdg/stringprep v1.0.3 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
go.opencensus.io v0.23.0 // indirect
go.opentelemetry.io/otel v0.18.0 // indirect
go.opentelemetry.io/otel/metric v0.18.0 // indirect
go.opentelemetry.io/otel/oteltest v0.18.0 // indirect
go.opentelemetry.io/otel/trace v0.18.0 // indirect
golang.org/x/crypto v0.26.0 // indirect
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3 // indirect
golang.org/x/net v0.0.0-20210427231257-85d9c07bbe3a // indirect
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be // indirect
golang.org/x/sync v0.8.0 // indirect
golang.org/x/text v0.17.0 // indirect
golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
google.golang.org/appengine v1.1.0 // indirect
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8 // indirect
google.golang.org/grpc v1.33.2 // indirect
google.golang.org/protobuf v1.23.0 // indirect
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
gopkg.in/errgo.v1 v1.0.0-20161222125816-442357a80af5 // indirect
gopkg.in/fsnotify.v1 v1.4.7 // indirect
gopkg.in/httprequest.v1 v1.1.1 // indirect
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
gopkg.in/yaml.v2 v2.3.0 // indirect
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
honnef.co/go/tools v0.0.0-20180728063816-88497007e858 // indirect
launchpad.net/gocheck v0.0.0-20140225173054-000000000087 // indirect
launchpad.net/xmlpath v0.0.0-20130614043138-000000000004 // indirect
golang.org/x/text v0.33.0 // indirect
)

replace (
Expand Down
Loading