Skip to content

fix(dpmodel): reject incompatible LMDB merge type maps#5838

Open
njzjz-bot wants to merge 1 commit into
deepmodeling:masterfrom
njzjz-bot:fix/lmdb-merge-type-map-5634
Open

fix(dpmodel): reject incompatible LMDB merge type maps#5838
njzjz-bot wants to merge 1 commit into
deepmodeling:masterfrom
njzjz-bot:fix/lmdb-merge-type-map-5634

Conversation

@njzjz-bot

Copy link
Copy Markdown
Contributor

Fixes #5634

Summary

  • preflight every source LMDB type map before replacing or creating the destination
  • preserve byte-for-byte merging for sources with identical explicit maps
  • preserve legacy behavior when every source omits type-map metadata
  • reject reordered maps and explicit/missing mixtures because raw atom-type indices cannot be proven compatible
  • include every source path and map in validation errors

This chooses fail-fast validation instead of decoding and rewriting frames. A remapping implementation would also need to keep atom_types, atom_names, atom_numbs, and system_info mutually consistent; rejecting incompatible metadata is the smaller safe change.

Why existing tests missed this

The merge coverage used either two mapless legacy databases or sources with the same ["O", "H"] map. Its type-map test inspected only the merged metadata and never read frames from both source segments. Because every fixture generated the same numeric 0/1 type ordering, raw frame copying preserved all tested shapes, counts, and IDs even though a reversed source map would change their species meaning.

The regular LMDB remapping tests cover one database mapped to a model type map, not multiple source maps being collapsed into one merged metadata map. Reader/dataset consistency checks would also interpret the same corrupted merged metadata and therefore agree with each other.

Validation

  • source/tests/pt/test_lmdb_dataloader.py: 53 passed
  • ruff format .
  • ruff check .

Coding agent: Codex
Codex version: codex-cli 0.144.4
Model: gpt-5.6-sol
Reasoning effort: xhigh

Preflight all source metadata before touching the destination. Permit identical explicit maps and all-mapless legacy inputs, but reject reordered maps and explicit/missing mixtures that byte-for-byte frame copying cannot preserve safely.

Cover incompatible and ambiguous sources, destination preservation on validation failure, and frame types from each segment of a valid merge.

Coding-Agent: Codex
Codex-Version: codex-cli 0.144.4
Model: gpt-5.6-sol
Reasoning-Effort: xhigh
@dosubot dosubot Bot added the bug label Jul 16, 2026
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@njzjz-bot, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 5 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 048e03d1-7ca5-46a8-a5d0-651c604fc343

📥 Commits

Reviewing files that changed from the base of the PR and between 6c3b985 and 3e38522.

📒 Files selected for processing (2)
  • deepmd/dpmodel/utils/lmdb_data.py
  • source/tests/pt/test_lmdb_dataloader.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.83333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 76.59%. Comparing base (6c3b985) to head (3e38522).

Files with missing lines Patch % Lines
deepmd/dpmodel/utils/lmdb_data.py 95.83% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5838      +/-   ##
==========================================
- Coverage   78.58%   76.59%   -1.99%     
==========================================
  Files        1050     1050              
  Lines      120637   120651      +14     
  Branches     4356     4354       -2     
==========================================
- Hits        94801    92417    -2384     
- Misses      24278    26667    +2389     
- Partials     1558     1567       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@njzjz

njzjz commented Jul 18, 2026

Copy link
Copy Markdown
Member

Possible reviewers based on changed lines, exact file history, and exact-file review history:

  • @wanghan-iapcm — 1 commits on changed files; 4 reviews on exact changed files (deepmd/dpmodel/utils/lmdb_data.py, source/tests/pt/test_lmdb_dataloader.py).

No review request was made automatically.

Coding agent: Codex
Codex version: codex-cli 0.144.4
Model: gpt-5.6-sol
Reasoning effort: xhigh

@njzjz
njzjz requested review from wanghan-iapcm and removed request for wanghan-iapcm July 18, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Code scan] Reject or remap incompatible type maps when merging LMDB datasets

2 participants