Design for integrating LanceDB into Hyrax by gitosaurus · Pull Request #683 · lincc-frameworks/hyrax

gitosaurus · 2026-02-06T23:05:02Z

Using this mechanism for comment on a proposed design for integrating LanceDB into Hyrax.

codecov · 2026-02-06T23:10:09Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.26%. Comparing base (19fb5f9) to head (85c518d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #683   +/-   ##
=======================================
  Coverage   63.26%   63.26%           
=======================================
  Files          59       59           
  Lines        5771     5771           
=======================================
  Hits         3651     3651           
  Misses       2120     2120

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

Adds a design document describing a proposed, stepwise plan for integrating LanceDB/Lance as a new storage backend for Hyrax “result datasets” (inference/UMAP/test/engine outputs), including writer/reader abstractions, verb wiring, migration, deprecation, and testing strategy.

Changes:

Introduces a new design doc laying out incremental steps for Lance-backed result writing/reading (ResultDatasetWriter / ResultDataset).
Proposes config changes (results.storage_format), verb wiring via factories, and an .npy → Lance migration verb.
Captures risks (performance, API churn, visualize incompatibility, disk layout conflicts) and test/benchmark plans.

github-actions · 2026-02-06T23:45:42Z

Before [`19fb5f9`]	After [`a13f0e7`]	Ratio	Benchmark (Parameter)
101±0.5μs	104±0.9μs	1.03	data_request_benchmarks.DatasetRequestBenchmarks.time_request_all_data
9.44±0.2ms	9.63±0.06ms	1.02	vector_db_benchmarks.VectorDBSearchBenchmarks.time_search_by_vector_many_shards(128, 'chromadb')
6.53±0s	6.57±0.01s	1.01	data_cache_benchmarks.DataCacheBenchmarks.time_preload_cache_hsc1k
16.944909319024536	17.169437703760522	1.01	data_cache_benchmarks.DataCacheBenchmarks.track_cache_hsc1k_hyrax_size_undercount
6.59±0.04s	6.65±0.03s	1.01	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(2048, 'qdrant')
9.30±0.2ms	9.43±0.2ms	1.01	vector_db_benchmarks.VectorDBSearchBenchmarks.time_search_by_vector_many_shards(64, 'chromadb')
1.75±0.01s	1.75±0.03s	1	benchmarks.time_database_connection_help
1.75±0.02s	1.75±0.01s	1	benchmarks.time_prepare_help
1.74±0.01s	1.74±0.03s	1	benchmarks.time_train_help
1.76±0.03s	1.76±0.02s	1	benchmarks.time_visualize_help

Click here to view all benchmarks.

drewoldag

Left a handful of comments. Overall I really like that approach. Were you BMAD'ing to produce this?

Copilot · 2026-02-11T21:37:23Z

@gitosaurus I've opened a new pull request, #691, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot · 2026-02-11T21:56:54Z

@gitosaurus I've opened a new pull request, #692, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 19 comments.

Copilot · 2026-02-12T00:58:24Z

@gitosaurus I've opened a new pull request, #696, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot · 2026-02-12T00:58:34Z

@gitosaurus I've opened a new pull request, #697, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot · 2026-02-12T01:01:28Z

@gitosaurus I've opened a new pull request, #698, to work on those changes. Once the pull request is ready, I'll request review from you.

* Use dedicated lance_db subdirectory for LanceDB storage --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com>

* Update lance_design.md terminology to match actual codebase Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Simplify writer factory to always use ResultDatasetWriter without config option Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com> --------- Co-authored-by: Derek T. Jones <dtj@mac.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com> Co-authored-by: Derek T. Jones <dtj1s@uw.edu> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Address feedback on lines 104-106: clarify that tensor metadata (shape, dtype) is stored in Arrow table schema's custom metadata dictionary, show proper JSON serialization using json.dumps(), and explain the serialization/deserialization process with concrete code examples. Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com> --------- Co-authored-by: Derek T. Jones <dtj@mac.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com> Co-authored-by: Derek T. Jones <dtj1s@uw.edu> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

drewoldag

This looks good to me. We've reviewed it a couple of times, good to go.

gitosaurus marked this pull request as ready for review February 6, 2026 23:05

gitosaurus requested review from Copilot, drewoldag and mtauraso February 6, 2026 23:05

Copilot started reviewing on behalf of gitosaurus February 6, 2026 23:05 View session

Copilot AI reviewed Feb 6, 2026

View reviewed changes

Comment thread lance_design.md Outdated

Comment thread specs/lance_db_spec.md

Comment thread lance_design.md Outdated

Comment thread lance_design.md Outdated

drewoldag reviewed Feb 7, 2026

View reviewed changes

gitosaurus mentioned this pull request Feb 10, 2026

Dtj lance opt #667

Closed

4 tasks

Copilot AI mentioned this pull request Feb 11, 2026

Use dedicated lance_db subdirectory for LanceDB storage #691

Merged

4 tasks

gitosaurus force-pushed the dtj-lance-design branch from 9a784c1 to 17cbe9b Compare February 11, 2026 21:43

Copilot AI mentioned this pull request Feb 11, 2026

Update lance_design.md terminology to match codebase #692

Merged

4 tasks

gitosaurus requested review from Copilot and drewoldag February 11, 2026 23:53

Copilot started reviewing on behalf of gitosaurus February 11, 2026 23:54 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

Copilot AI mentioned this pull request Feb 12, 2026

Remove storage_format config option from LanceDB design #696

Closed

4 tasks

Copilot AI mentioned this pull request Feb 12, 2026

Remove storage_format config option from LanceDB design #697

Merged

4 tasks

Copilot AI mentioned this pull request Feb 12, 2026

Clarify tensor metadata storage in LanceDB design document #698

Merged

4 tasks

gitosaurus and others added 5 commits February 11, 2026 17:03

Design for integrating LanceDB into Hyrax

5fbd9c5

Address PR comments, add CLAUDE.md

4ab4666

write_index -> commit

3b6fe19

gitosaurus and others added 8 commits February 11, 2026 17:03

Update lance_design.md

c561d88

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Per interactive discussion with Drew

757ecc1

Update lance_design.md

989ce4a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update lance_design.md

c7c1395

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update lance_design.md

04e5f00

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update lance_design.md

2499131

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update lance_design.md

91f6aa3

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update lance_design.md

cadb17c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

gitosaurus force-pushed the dtj-lance-design branch from 89d8e82 to cadb17c Compare February 12, 2026 01:03

Copilot AI and others added 4 commits February 11, 2026 17:12

Move spec to dedicated directory

3f4e096

Manual proofreading

85c518d

gitosaurus added a commit that referenced this pull request Feb 12, 2026

See #683 for the correct lance_db_spec.md

34ea1a7

drewoldag approved these changes Feb 12, 2026

View reviewed changes

gitosaurus merged commit ba1430a into main Feb 12, 2026
9 checks passed

gitosaurus deleted the dtj-lance-design branch February 12, 2026 23:44

Conversation

gitosaurus commented Feb 6, 2026

Uh oh!

codecov Bot commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drewoldag left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Feb 11, 2026

Uh oh!

Copilot AI commented Feb 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Feb 12, 2026

Uh oh!

Copilot AI commented Feb 12, 2026

Uh oh!

Copilot AI commented Feb 12, 2026

Uh oh!

drewoldag left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Feb 6, 2026 •

edited

Loading

github-actions Bot commented Feb 6, 2026 •

edited

Loading