🐎 accelerate findcirc

[dmiller15]

Description

Speed improvement for find_circ. Original code can be found here: https://github.com/daaaaande/circs_snake/blob/master/scripts/pipelines/find_circ.py.

It's a 10 year old piece of software that is extremely slow. I decided to change a couple of things:

1. I examined the profile of a test run an observed that it's doing a lot of expensive numpy calls that can be better handled more cheaply through bytewise comparisons. Additionally, I added an early exit to the function so that it will stop as soon as it exceeds the mismatch limit.
2. find_breakpoints needs to be called on each pair of reads. The original implementation does this one pair at a time and that's really quite rate limiting for the pipe. The best way to unlock things here was to add multiprocessing. The multiprocessing is done in batches of 500 across as many threads as possible.

For this review, a lot of the code is copied from the original. For subworkflows/local/circs/bin/find_circ_mp.py everything can be taken as-is except for:

• mismatches_bytes_early
• find_breakpoints
• worker_init
• worker_process
• worker_process_inner
• main script lines 480-590

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Using single threaded script: https://cavatica.sbgenomics.com/u/naqvia/circrna/tasks/42df93e9-55db-49bd-97e4-7a0cdf7feb88
• Using multi threaded script: https://cavatica.sbgenomics.com/u/naqvia/circrna/tasks/1b7f3f31-31c1-44c3-89ec-271c63221d7b/

Process time drops from 5.5 to 1.5 hours!

Test Configuration:

• Environment:
• Test files:

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings
• I have committed any related changes to the PR

[migbro]

This is pretty nice overall. I mostly just comments. I assume you've compared to the original implementation to ensure functional equivalence?

[migbro]

Add docstring?

[dmiller15]

Eh, I didn't write this. I just swapped out the mismatch function.

[migbro]

Interesting...is this for speed, or do global lookups have like "changed during iteration" problems?

[dmiller15]

It is for speed. It's one of the many suggestions CHOPGPT came up with. The improvement was minimal compared to dropping numpy and doing early exit. Left it in anyway.

[migbro]

Comments addressed, thanks!