Skip to content

Memory testing #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jul 11, 2023
Merged

Memory testing #40

merged 14 commits into from
Jul 11, 2023

Conversation

Adoni5
Copy link
Owner

@Adoni5 Adoni5 commented Jun 26, 2023

Blocked, waiting on jguhlin/minimap2-rs#40 to be merged.

This PR rewrites the thread pool to be a permanently waiting pool, rather than spawning a new pool each map_batch call.
An improvement to be made would be to park the threads, rather than sleep them but the effect is the same.
There is some memory weirdness around using Py<PyAny>> to store the Dictionary on the side

Waiting until the minimap2-rs crate is updated as currently references a local version that only exists on my laptop.

Adoni5 added 3 commits June 22, 2023 18:39
…updates in minmap2-rs

Add bare clipppy allows cause I need this backed up
Kept the Py<PyAny> wrapper that messes the memory up
but no leaks
@Adoni5 Adoni5 mentioned this pull request Jun 29, 2023
2 tasks
Adoni5 added 7 commits July 3, 2023 11:19
Currently pointing at the revision commit hash 037abe5
When crates is updated, will come back and repoint at that
Was failing the tests as Multiple handlers were being set
Remove mem::drop on aligners in tests
Stop it being tested by pytest
As fun as MSRV roulette was turning out to be
@Adoni5 Adoni5 requested a review from alexomics July 3, 2023 14:04
Adoni5 added 2 commits July 3, 2023 15:18
Bump version
Add more classifiers
Add Readme.md
@Adoni5
Copy link
Owner Author

Adoni5 commented Jul 3, 2023

This release switches from spawning threads every time map_batch is called to having a long running thread pool on the Aligner class, which is created if enable_threading is called. This is a naive implementation, but it does work.

adds some complexity in passing reads into the threadpool via an ArrayQueue and then transferring them back to map_batch, where they are passed on again to the AlignmentBatchResultIter.

Some unused methods are added for catching memory leaks, which I intend to implement in PyTest. These are map_no_op on the PyMethods Aligner struct and no_op_map on the impl Aligner. Respectively, these take in a sequence and return a Pr e created Alignment and create the pre created alignment to be returned.

@Adoni5 Adoni5 merged commit 9816afa into main Jul 11, 2023
@Adoni5 Adoni5 deleted the memory_testing branch August 11, 2023 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant