Skip to content

Register CLEM data in ISPyB#689

Merged
tieneupin merged 21 commits intomainfrom
register-clem-data-collection
Nov 6, 2025
Merged

Register CLEM data in ISPyB#689
tieneupin merged 21 commits intomainfrom
register-clem-data-collection

Conversation

@tieneupin
Copy link
Copy Markdown
Contributor

@tieneupin tieneupin commented Oct 31, 2025

Adds logic to register incoming CLEM data to both ISPyB and its mirror tables on Murfey.

  • Adds new columns to the CLEMImageSeries table to keep track of image metadata needed for the ISPyB insert
  • Adds logic to create a data collection group entry for CLEM data in ISPyB
  • Adds logic to register grid overview datasets as atlases in ISPyB
  • Adds logic to register positional datasets as grid squares in ISPyB
  • Migrates hard-coded image scaling logic in some of the ISPyB insertion functions into the workflows that scale them as such, allowing the insertion function to be more universally used

* Linked 'CLEMImageSeries' to 'DataCollectionGroup' and 'GridSquare'
* Tried modern type hinting notation in 'CLEMImageSeries' table
* Added new columns to 'CLEMImageSeries' to keep track of image search string (for globbing images with) and type of dataset (i.e., atlas or grid square)
* Removed 'composite_image' column from 'CLEMImageSeries'
@codecov
Copy link
Copy Markdown

codecov Bot commented Oct 31, 2025

Codecov Report

❌ Patch coverage is 71.20419% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 38.24%. Comparing base (f91762e) to head (3abe4f6).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #689      +/-   ##
==========================================
+ Coverage   36.94%   38.24%   +1.29%     
==========================================
  Files          97       97              
  Lines       11459    11593     +134     
  Branches     1510     1528      +18     
==========================================
+ Hits         4234     4434     +200     
+ Misses       7063     6964      -99     
- Partials      162      195      +33     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…of image shape, pixel size, and spatial location on grid
… scaled down information about the grid squares being registered
…reprocess' workflow instead, so that the grid square registration and updating functions just register the values as-is
Comment thread src/murfey/workflows/clem/register_preprocessing_results.py Fixed
Comment thread src/murfey/workflows/clem/register_preprocessing_results.py Fixed
@tieneupin tieneupin changed the title Register clem data collection Register CLEM data collection Nov 4, 2025
@tieneupin tieneupin changed the title Register CLEM data collection Register CLEM data in ISPyB Nov 5, 2025
@tieneupin tieneupin self-assigned this Nov 5, 2025
@tieneupin tieneupin added enhancement New feature or request server Relates to the server component cryo-clem Part of the cryo-CLEM pipeline extension labels Nov 5, 2025
@tieneupin tieneupin marked this pull request as ready for review November 5, 2025 11:33
@tieneupin tieneupin force-pushed the register-clem-data-collection branch 2 times, most recently from 0b55f57 to 2c1ac4f Compare November 5, 2025 17:02
@tieneupin tieneupin force-pushed the register-clem-data-collection branch 3 times, most recently from c77516e to d206593 Compare November 5, 2025 18:07
Comment thread tests/workflows/clem/test_register_preprocessing_results.py Fixed
@tieneupin tieneupin force-pushed the register-clem-data-collection branch 6 times, most recently from d12a674 to 6bb3e69 Compare November 5, 2025 19:26
Copy link
Copy Markdown
Contributor

@stephen-riggs stephen-riggs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All seems fine, as long as you're happy it works

Comment thread src/murfey/util/db.py
pixels_x: Optional[int] = Field(default=None)
pixels_y: Optional[int] = Field(default=None)
pixel_size: Optional[float] = Field(default=None)
units: Optional[str] = Field(default=None)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need the units in the database? I can't see they are used anywhere

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a valid point. I included it as a just-in-case, so I'll revisit this in the future if no new workflows end up using this column.

@tieneupin tieneupin force-pushed the register-clem-data-collection branch from 6bb3e69 to a62941c Compare November 6, 2025 09:38
Comment thread src/murfey/workflows/clem/register_preprocessing_results.py Fixed
@tieneupin tieneupin force-pushed the register-clem-data-collection branch 8 times, most recently from 7c7cce2 to 31b26f4 Compare November 6, 2025 14:20
@tieneupin tieneupin force-pushed the register-clem-data-collection branch from 7917b40 to 72b2b1f Compare November 6, 2025 14:36
@tieneupin tieneupin force-pushed the register-clem-data-collection branch from 5c988b1 to df1f732 Compare November 6, 2025 14:52
@tieneupin tieneupin force-pushed the register-clem-data-collection branch from ba2d1d2 to 3abe4f6 Compare November 6, 2025 15:50
@tieneupin tieneupin merged commit 2e38cb2 into main Nov 6, 2025
17 checks passed
@tieneupin tieneupin deleted the register-clem-data-collection branch November 6, 2025 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cryo-clem Part of the cryo-CLEM pipeline extension enhancement New feature or request server Relates to the server component

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants