Parallelizing pipeline runs on non-HPC systems

This guide shows possible ways to parallelize pipeline runs on computer systems without job schedulers supported by Nipoppy.

Getting a list of participants and sessions to run

The nipoppy bidsify, nipoppy process, and nipoppy extract commands all have a --write-list option. If this option is specified, instead of launching the pipeline, the command will write a TSV file of participant and session IDs that need to be run.

$ nipoppy <SUBCOMMAND> \
    --dataset <NIPOPPY_PROJECT_ROOT> \
    --pipeline <PIPELINE_NAME> \
    --write-list <PATH_TO_TSV_FILE>

Launching parallel processes

The TSV file created with the --write-list option can be used to launch parallel runs. On Linux systems, this can be done using the parallel tool if it is installed:

$ parallel \
    --colsep '\t' \
    --jobs <N_MAX_JOBS> \
    nipoppy <SUBCOMMAND> \
        --dataset <NIPOPPY_PROJECT_ROOT> \
        --pipeline <PIPELINE_NAME> \
        # other desired options \
        # ... \
        --participant-id {1} \
        --session-id {2} \
    :::: <PATH_TO_TSV_FILE>

If parallel or other similar utilities are not available, custom scripts would be needed to launch the runs in parallel.