Skip to content

Align default BanyanDB stage segmentInterval to integer multiples #13910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mrproliu wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Align default BanyanDB stage segmentInterval to integer multiples #13910

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fc4e0e5
Align default BanyanDB stage segmentInterval to integer multiples
mrproliu Jun 14, 2026
be7c374
Address review: update banyandb.md defaults + fix stale stages.md exa…
mrproliu Jun 14, 2026
d9dc930
Trigger CI re-run
mrproliu Jun 14, 2026
ca34a0c
fix CI failure
mrproliu Jun 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/publish-docker-e2e-service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,14 +51,14 @@ jobs:
restore-keys: |
${{ runner.os }}-maven-
- name: Log in to the Container registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
registry: ${{ env.HUB }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up QEMU
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392
uses: docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3 # v4.1.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
- name: Build and push images
run: make -C test build.e2e-service docker.push-e2e-service
6 changes: 3 additions & 3 deletions .github/workflows/publish-docker.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,15 +65,15 @@ jobs:
echo "TAG=${{ github.sha }}" >> $GITHUB_ENV
fi
- name: Log in to the Container registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
registry: ${{ env.DOCKER_REGISTRY }}
username: ${{ env.DOCKER_USERNAME }}
password: ${{ env.DOCKER_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392
uses: docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3 # v4.1.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
- name: Build and push docker images based on Java 11
env:
SW_OAP_BASE_IMAGE: eclipse-temurin:11-jre
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/skywalking.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ jobs:
persist-credentials: false
- name: Filter
id: filter
uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36
uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
with:
list-files: 'shell'
predicate-quantifier: 'every'
Expand Down Expand Up @@ -765,7 +765,7 @@ jobs:
name: dist
path: dist
- name: Login to ghcr
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
Expand Down Expand Up @@ -854,7 +854,7 @@ jobs:
find docker-images -name "*.tar" -exec docker load -i {} \;
find docker-images -name "*.tar" -exec rm {} \;
- name: Login to ghcr
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
Expand Down Expand Up @@ -926,7 +926,7 @@ jobs:
find docker-images -name "*.tar" -exec docker load -i {} \;
find docker-images -name "*.tar" -exec rm {} \;
- name: Login to ghcr
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
Expand Down Expand Up @@ -1042,7 +1042,7 @@ jobs:
name: dist
path: dist
- name: Login to ghcr
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
Expand Down
31 changes: 16 additions & 15 deletions docs/en/banyandb/stages.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,31 +14,32 @@ Each group of records/metrics can be automatically migrated and stored in differ
If necessary, you also can jump the warm stage, and only use hot and cold stages. Then the data will be moved to the cold stage after the TTL of the hot stage.

## Configuration Guidelines
The lifecycle stages configuration is under each group settings of the `bydb.yml` file, for example, the `metricsMin` group:
The lifecycle stages configuration is under each group settings of the `bydb.yml` file, for example, the `metricsMinute` group:

```yaml
metricsMin:
metricsMinute:
# The settings for the default `hot` stage.
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_TTL_DAYS:7}
enableWarmStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_COLD_STAGE:false}
shardNum: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_SI_DAYS:1}
ttl: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_TTL_DAYS:7}
enableWarmStage: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_ENABLE_WARM_STAGE:false}
enableColdStage: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_ENABLE_COLD_STAGE:false}
warm:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SI_DAYS:3}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_TTL_DAYS:15}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_NODE_SELECTOR:"type=warm"}
shardNum: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_SI_DAYS:3}
ttl: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_TTL_DAYS:15}
nodeSelector: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_NODE_SELECTOR:"type=warm"}
cold:
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SI_DAYS:5}
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_TTL_DAYS:60}
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_NODE_SELECTOR:"type=cold"}
shardNum: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SHARD_NUM:2}
segmentInterval: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SI_DAYS:6}
ttl: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_TTL_DAYS:60}
nodeSelector: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_NODE_SELECTOR:"type=cold"}
```

1. **shardNum**: The number of shards for the group.
2. **segmentInterval**: The time interval in days for creating a new data segment.
- According to the freshness of the data, the `segmentInterval` days should: `hot` < `warm` < `cold`.
- Each coarser stage's `segmentInterval` should be an **integer multiple** of the immediately finer (preceding) stage's `segmentInterval` (e.g. `hot=5` → `warm=10` → `cold=20`). BanyanDB segments are epoch-anchored, so when the coarser interval is a multiple of the finer one each source segment nests entirely inside one target segment and the hot → warm → cold migration stays on the cheap whole-segment fast path. A non-multiple interval (e.g. `hot=5`, `warm=7`) makes source segments straddle target boundaries, forcing a row-by-row re-bucketing that costs significantly more CPU, IO, and memory. It does not affect correctness, only migration efficiency.
3. **ttl**: The time-to-live for data within the group, in days.
4. **enableWarmStage/enableColdStage**: Enable the warm/cold stage for the group.
- The `hot` stage is always enabled by default.
Expand Down
1 change: 1 addition & 0 deletions docs/en/changes/changes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,7 @@
* Bump up dependencies to clear CVE alerts on shipped OAP jars: log4j `2.25.3` → `2.25.4`, jackson `2.18.5` → `2.18.6`, kafka-clients `3.4.0` → `3.9.2`, postgresql `42.4.4` → `42.7.11`, commons-compress `1.21` → `1.26.2`.
* Fix: continuous profiling policy validation now rejects a threshold / count of `0` to match the error messages and rover's `value >= threshold` trigger semantics (a `0` threshold would always trigger). CPU percent and HTTP error rate are tightened from `[0-100]` to `(0-100]`.
* Fix wrong BanyanDB resource options in record data.
* Align the default BanyanDB stage `segmentInterval` values so each coarser stage is an integer multiple of the finer one (`records` cold `3` → `4`, `metricsMinute` cold `5` → `6`, `metricsHour` warm `7` → `10` and cold `15` → `20`), keeping hot → warm → cold lifecycle migration on the cheap whole-segment fast path.

#### UI
* Add Airflow layer dashboards and menu i18n under Workflow Scheduler in Horizon UI (SWIP-7).
Expand Down
8 changes: 4 additions & 4 deletions docs/en/setup/backend/configuration-vocabulary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,7 +432,7 @@ OAP will query the data from the "hot and warm" stage by default if the "warm" s
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_RECORDS_WARM_NODE_SELECTOR | type=warm |
| - | cold | - | The cold stage settings. | - | - |
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_SHARD_NUM | 1 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_SI_DAYS | 3 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_SI_DAYS | 4 |
Comment thread
mrproliu marked this conversation as resolved.
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_TTL_DAYS | 30 |
| - | - | replicas | Replicas for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_REPLICAS | 0 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_NODE_SELECTOR | type=cold |
Expand Down Expand Up @@ -527,7 +527,7 @@ OAP will query the data from the "hot and warm" stage by default if the "warm" s
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_NODE_SELECTOR | type=warm |
| - | cold | - | The cold stage settings. | - | - |
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SHARD_NUM | 2 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SI_DAYS | 5 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SI_DAYS | 6 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_TTL_DAYS | 60 |
| - | - | replicas | Replicas for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_REPLICAS | 0 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_NODE_SELECTOR | type=cold |
Expand All @@ -539,13 +539,13 @@ OAP will query the data from the "hot and warm" stage by default if the "warm" s
| - | enableColdStage | - | Activate cold stage for hour granularity metrics group. | SW_STORAGE_BANYANDB_METRICS_HOUR_ENABLE_COLD_STAGE | false |
| - | warm | - | The warm stage settings. | - | - |
| - | - | shardNum | Shards Number for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_SHARD_NUM | 1 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_SI_DAYS | 7 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_SI_DAYS | 10 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_TTL_DAYS | 30 |
| - | - | replicas | Replicas for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_REPLICAS | 0 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_NODE_SELECTOR | type=warm |
| - | cold | - | The cold stage settings. | - | - |
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_SHARD_NUM | 1 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_SI_DAYS | 15 |
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_SI_DAYS | 20 |
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_TTL_DAYS | 120 |
| - | - | replicas | Replicas for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_REPLICAS | 0 |
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_NODE_SELECTOR | type=cold |
Expand Down
Loading
Loading