Refactor and expand onepass model#300
Draft
sjfleming wants to merge 18 commits into
Draft
Conversation
Contributor
Author
|
Welford covariance computation on all 33k genes (using a batch size of 10k cells) spikes to like 31GB memory usage on my laptop (eyeballing Activity Monitor). Just wanted to make a note of this as a ballpark figure. I am guessing that's why the github actions runner went OOM on 5495ce0. The CLI test now uses a For speed purposes, the rule seems to be "make the batch as big as you can accommodate in memory". |
Contributor
Author
|
Ensure batch is not empty in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #163
Closes #296
This is a refactor of onepass to make it more extensible. It implements the Welford algorithm for online variance calculation and it also implements a gene-gene covariance computation via an online algorithm similar to Welford (https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Covariance subsection "Online"). The latter can be run using the raw data or ranks (which allows for a computation of gene-gene Spearman correlations).
Currently the Welford implementation is actually implemented in a different cellarium class, as is the covariance implementation. I thought this might be cleaner than having one huge class with more input arguments, but I'm open to opposing views. Also, the class heirarchy worked a lot better when Welford was a separate class, since Welford keeps track of different sufficient statistics than the naive/shifted algorithms. And the Welford-like gene-gene covariance keeps track of the same sufficient statistics as Welford (plus more).
Todo: