Skip to content

ENH: Major update to DSL2 and nf-neuro#9

Open
levje wants to merge 32 commits intoscilus:masterfrom
levje:nf-update
Open

ENH: Major update to DSL2 and nf-neuro#9
levje wants to merge 32 commits intoscilus:masterfrom
levje:nf-update

Conversation

@levje
Copy link
Contributor

@levje levje commented Sep 16, 2025

Here it is, extractor_flow version 2.0. This is somewhat of a weird pull request as most of the development for this new version was done in a separate repo with an unrelated history (my mistake!). This work was done starting from the nf-neuro-tutorial, which includes commits from that repo, which I tried to squash into the "Initial commit" to avoid polluting the commit history. Since this new version essentially changes the entire repo, it might be easier to review this PR by directly looking at the files instead of trying to compare what changed.

The major changes are the following:

  • Nextflow DSL2 support (finally, yay!)
  • Utilize nf-neuro modules (i.e. betcrop, ants & tractogram registration and tractogram removeinvalid), enabling better updatability and improved longevity of the project.
  • Modularized the monolith pipeline code by reusing modules, simplifying the overall organisation and readability of the pipeline. This includes the filter_with_lists.sh templates as well as some other local modules.
  • Utilize the newest scilus/scilpy:dev containers!
  • Extensive configuration of each module specified in nextflow.config using each module's given name.

Note: Do we want to create a 'release' tag on the original code before merging this PR? Users could then refer to this tag to use the original implementation of the pipeline if need be.

levje and others added 30 commits February 19, 2025 15:45
add imaging data

remove some unused stuff

some updates

Update .devcontainer/devcontainer.json

Co-authored-by: Alex Valcourt Caron <[email protected]>

first try with new container

add new files and fix devcontainer

fix devcontainer with alex

fix main + add nf-test config

Update .devcontainer/devcontainer.json

Co-authored-by: Alex Valcourt Caron <[email protected]>

fix var PATH

add subworkflows load_test_data

add nextflow.config to be able to play with nf-core modules

keep only stub test

Update Dockerfile

downgrade nf-core

Update nextflow.config

Fix line ending issues in windows
https://code.visualstudio.com/docs/devcontainers/tips-and-tricks#_resolving-git-line-ending-issues-in-containers-resulting-in-many-modified-files

Update Dockerfile

Fix miniconda version. Due to some python package dependencies python 3.13 does not work yet.
@levje
Copy link
Contributor Author

levje commented Sep 16, 2025

For the tests that I've ran so far, here's the number of streamlines for the ORIGINAL IMPLEMENTATION of extractor_flow for subject t0003 (it was also giving off several failure/retry errors during runtime) using nextflow 22.10.0 on DSL1:
image

@levje
Copy link
Contributor Author

levje commented Sep 16, 2025

And here's the number of streamlines for the NEW IMPLEMENTATION of extractor_flow for subject t0003 (no errors or warning during runtime) using nextflow 25.04.6 on DSL2:
image

Copy link

@EmmaRenauld EmmaRenauld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some comments from reading the Readme

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants