Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major refactoring of sampled_indicators #240

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Conversation

RondeauG
Copy link
Collaborator

@RondeauG RondeauG commented Dec 3, 2024

Pull Request Checklist:

  • This PR addresses an already opened issue (for bug fixes / features)
    • This PR fixes #xyz
  • (If applicable) Documentation has been added / updated (for bug fixes / features).
  • (If applicable) Tests have been added.
  • CHANGELOG.rst has been updated (with summary of main changes).
    • Link to issue (:issue:number) and pull request (:pull:number) has been added.

What kind of change does this PR introduce?

  • Turns out that sampled_indicators was too memory-inefficient for bigger applications and could quickly grow to 10s or even 100s of Gb of RAM. Thus, the function has been split in two to allow for writing the results of the sampling on disk and offload some of the RAM usage. No miracle can be done, but this should be slightly better.
  • The default number of iterations was reduced from 50,000 to 5000.
  • Uniform weights are now only added to extra dimensions specifically named in include_dims, instead of being automatic. The dimensions time and horizon are still never sampled.
  • _weighted_sampling was almost completely rewritten to hopefully make better use of chunks and multithreading through dask.

Does this PR introduce a breaking change?

  • Yes. While this does not affect the numerical results themselves, this is an almost complete refactoring of sampled_indicators, from the inputs required to the outputs produced.

Other information:

Sorry, something went wrong.

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@RondeauG RondeauG requested a review from essicolo December 3, 2024 16:35
@github-actions github-actions bot added notebooks Run tests against notebooks docs labels Dec 3, 2024
RondeauG and others added 5 commits December 3, 2024 11:36

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs notebooks Run tests against notebooks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants