Developing a Nipoppy pipeline

This guide explains how to generate and customize a pipeline configuration for use with Nipoppy.

Creating initial configuration files

Use the nipoppy pipeline create command to generate a sample configuration. Nipoppy supports three pipeline types: bidsification, processing, and extraction.

nipoppy pipeline create \
    --type processing \
    <PIPELINE_DIR>

Tip

Optionally, use the --source-descriptor flag to initialize the configuration based on a local Boutiques descriptor file. See the Creating a Boutiques descriptor section below.

This will create the following structure:

$ tree ./pipelines/howto
pipelines/howto
├── config.json
├── descriptor.json
├── hpc.json
├── invocation.json
├── pybids_ignore.json
└── tracker.json

Customizing configuration files

config.json

Edit general pipeline metadata and settings.

Example for the FSL SIENA pipeline:

 1{
 2    "NAME": "FSL SIENA",
 3    "VERSION": "6.0.4",
 4    "CONTAINER_INFO": {
 5        "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/[[PIPELINE_NAME]]_[[PIPELINE_VERSION]].sif",
 6        "URI": "docker://<OWNER>/[[PIPELINE_NAME]]:[[PIPELINE_VERSION]]"
 7    },
 8    "CONTAINER_CONFIG": {
 9        "ENV_VARS": {},
10        "ARGS": []
11    },
12    "STEPS": [
13        {
14            "INVOCATION_FILE": "invocation.json",
15            "DESCRIPTOR_FILE": "descriptor.json",
16            "HPC_CONFIG_FILE": "hpc.json",
17            "TRACKER_CONFIG_FILE": "tracker.json",
18            "PYBIDS_IGNORE_FILE": "pybids_ignore.json"
19        }
20    ],
21    "PIPELINE_TYPE": "processing",
22    "SCHEMA_VERSION": "1"
23}

Note

[[NIPOPPY_DPATH_CONTAINERS]]: is set in the nipoppy config file.

[[PIPELINE_NAME]] and [[PIPELINE_VERSION]] are set dynamically during execution using the value from the "NAME" and "VERSION" fields, respectively.

Warning

If not using a source descriptor, be sure to update the "NAME" and "VERSION" fields, and the <OWNER> placeholder of the CONTAINER_INFO’s URI. You may need to replace the entire URI if the container name does not follow the <OWNER>/[[PIPELINE_NAME]]:[[PIPELINE_VERSION]] naming convention.

descriptor.json

Define how to run your pipeline, including inputs, outputs, and command-line structure, using the Boutiques descriptor schema.

Note

You can skip this step if you used a source descriptor.

Creating a Boutiques descriptor

A Boutiques descriptor is a JSON file that describes:

  • the command-line interface of your tool ("command-line"),

  • required inputs and their types ("inputs"),

  • outputs and how to find them ("output-files"),

  • and execution environment (e.g., Docker/Singularity container via "container-image").

Example:

  1{
  2    "author": "Oxford Centre for Functional MRI of the Brain (FMRIB)",
  3    "name": "FSL SIENA",
  4    "description": "SIENA is a package for both single-time-point (\"cross-sectional\") and two-time-point (\"longitudinal\") analysis of brain change, in particular, the estimation of atrophy (volumetric loss of brain tissue)",
  5    "tool-version": "6.0.4",
  6    "schema-version": "0.5",
  7    "command-line": "/opt/fsl-6.0.4/bin/siena [INPUT1] [INPUT2] [OUTPUT_DIR] [DEBUG] [BET_OPTS] [TWO_CLASS_SEGMENTATION] [T2_WEIGHTED] [STANDARD_SPACE_MASKING] [IGNORE_UPWARDS] [IGNORE_DOWNWARDS] [SIENA_DIFF_OPTS] [VIENA] [VENTRICLE_MASK_IMAGE]",
  8    "container-image": {
  9        "image": "mathdugre/fsl:6.0.4",
 10        "index": "docker://",
 11        "type": "singularity"
 12    },
 13    "inputs": [
 14        {
 15            "id": "input1",
 16            "name": "First T1 weighted image",
 17            "optional": false,
 18            "type": "String",
 19            "value-key": "[INPUT1]"
 20        },
 21        {
 22            "id": "input2",
 23            "name": "Second T1 weighted image",
 24            "optional": false,
 25            "type": "String",
 26            "value-key": "[INPUT2]"
 27        },
 28        {
 29            "command-line-flag": "-o",
 30            "id": "output_dir",
 31            "name": "set output directory (default output is <input1>_to_<input2>_siena)",
 32            "optional": true,
 33            "type": "String",
 34            "value-key": "[OUTPUT_DIR]"
 35        },
 36        {
 37            "command-line-flag": "-d",
 38            "id": "debug",
 39            "name": "debug (don't delete intermediate files)",
 40            "optional": true,
 41            "type": "Flag",
 42            "value-key": "[DEBUG]"
 43        },
 44        {
 45            "command-line-flag": "-B",
 46            "id": "bet_opts",
 47            "name": "options to pass to BET brain extraction (inside double-quotes), e.g. -B \"-f 0.3\"",
 48            "optional": true,
 49            "type": "String",
 50            "value-key": "[BET_OPTS]"
 51        },
 52        {
 53            "command-line-flag": "-2",
 54            "id": "two_class_segmentation",
 55            "name": "two-class segmentation (don't segment grey and white matter separately)",
 56            "optional": true,
 57            "type": "Flag",
 58            "value-key": "[TWO_CLASS_SEGMENTATION]"
 59        },
 60        {
 61            "command-line-flag": "-t2",
 62            "id": "t2_weighted",
 63            "name": "T2-weighted input image (default T1-weighted)",
 64            "optional": true,
 65            "type": "Flag",
 66            "value-key": "[T2_WEIGHTED]"
 67        },
 68        {
 69            "command-line-flag": "-m",
 70            "id": "standard_space_masking",
 71            "name": "use standard-space masking as well as BET",
 72            "optional": true,
 73            "type": "Flag",
 74            "value-key": "[STANDARD_SPACE_MASKING]"
 75        },
 76        {
 77            "command-line-flag": "-t",
 78            "id": "ignore_upwards",
 79            "name": "ignore from t (mm) upwards in MNI152/Talairach space",
 80            "optional": true,
 81            "type": "Number",
 82            "value-key": "[IGNORE_UPWARDS]"
 83        },
 84        {
 85            "command-line-flag": "-b",
 86            "id": "ignore_downwards",
 87            "name": "ignore from b (mm) downwards in MNI152/Talairach space (b should probably be negative)",
 88            "optional": true,
 89            "type": "Number",
 90            "value-key": "[IGNORE_DOWNWARDS]"
 91        },
 92        {
 93            "command-line-flag": "-S",
 94            "id": "siena_diff_opts",
 95            "name": "options to pass to siena_diff timepoint differencing (inside double-quotes), e.g. -S \"-s -i 20\"",
 96            "optional": true,
 97            "type": "String",
 98            "value-key": "[SIENA_DIFF_OPTS]"
 99        },
100        {
101            "command-line-flag": "-V",
102            "id": "viena",
103            "name": "run ventricle analysis (VIENA)",
104            "optional": true,
105            "type": "Flag",
106            "value-key": "[VIENA]"
107        },
108        {
109            "command-line-flag": "-v",
110            "id": "ventricle_mask_image",
111            "name": "optional user-supplied ventricle mask (default is /opt/fsl-6.0.4/data/standard/MNI152_T1_2mm_VentricleMask)",
112            "optional": true,
113            "type": "String",
114            "value-key": "[VENTRICLE_MASK_IMAGE]"
115        }
116    ],
117    "output-files": [
118        {
119            "id": "output1",
120            "name": "Output directory",
121            "optional": false,
122            "path-template": "[OUTPUT_DIR]"
123        }
124    ],
125    "tags": {
126        "domain": [
127            "neuroimaging",
128            "mri",
129            "fmri"
130        ],
131        "toolbox": "fsl"
132    }
133}

Helpful commands

bosh is the CLI from Boutiques—already installed with Nipoppy.

Validating a descriptor:

bosh validate descriptor.json

Generating an invocation template:

bosh example descriptor.json > invocation.json

Simulating a run:

bosh exec simulate descriptor.json invocation.json

References

invocation.json

Defines input arguments to the pipeline.

1{
2    "input1": "str_input1_PV",
3    "input2": "str_input2_Yh"
4}

Note

Optionally, you can regenerate the invocation.json file to match the descriptor arguments.

bosh example ./pipelines/howto/descriptor.json > ./pipelines/howto/invocation.json

hpc.json (Optional)

Defines HPC job submission parameters.

{
    "ACCOUNT": "[[HPC_ACCOUNT_NAME]]",
    "TIME": "1:00:00",
    "CORES": "1",
    "MEMORY": "16G",
    "ARRAY_CONCURRENCY_LIMIT": ""
}

Processing pipeline specific files

tracker.json

Defines output tracking paths:

{
    "PATHS": [
        "[[NIPOPPY_BIDS_PARTICIPANT_ID]]/[[NIPOPPY_BIDS_SESSION_ID]]/anat/[[NIPOPPY_BIDS_PARTICIPANT_ID]]_[[NIPOPPY_BIDS_SESSION_ID]]*_example.txt"
    ],
    "PARTICIPANT_SESSION_DIR": "[[NIPOPPY_BIDS_PARTICIPANT_ID]]/[[NIPOPPY_BIDS_SESSION_ID]]"
}

Note

The tracked paths are relative to <NIPOPPY_PROJECT_ROOT>/derivatives/<PIPELINE_NAME>/<PIPELINE_VERSION>/output.

Assuming a participant 001 and session A, the template strings will resolve to:

  • [[NIPOPPY_PARTICIPANT_ID]]: 001

  • [[NIPOPPY_BIDS_PARTICIPANT_ID]]: sub-001

  • [[NIPOPPY_SESSION_ID]]: A

  • [[NIPOPPY_BIDS_SESSION_ID]]: ses-A

pybids_ignore.json

Defines file patterns to be excluded from the PyBIDS index. For example:

[
    "(.*_)*+T2w",  # ignores all T2w images.
    "(.*_)*+FLAIR",  # ignores all FLAIR images.
    ".*/dwi/",  # Skip any file inside a `dwi` folder.
]

Note

Patterns are relative to the BIDS root and use Unix-style wildcards (*, ?). Use this file to exclude temporary files, logs, and other non-BIDS outputs from indexing.

Tip

Optionally, You can set GENERATE_PYBIDS_DATABASE to False—skipping the database indexing—to accelerate the pipeline launch.

Uploading to Nipoppy catalog

Nipoppy provides an easy way to upload and install community-developed pipelines via Zenodo.

Important

Before uploading a pipeline to the catalog via the Nipoppy CLI, you must generate a Zenodo token.

nipoppy pipeline upload \
  --password-file <PASSWORD_FILE> \
  <PIPELINE_DIR>

zenodo.json (Optional)

By default, Nipoppy infers the value using the metadata from the user Zenodo account.

This file is optional. To provide custom metadata for your Zenodo record you must specify it in the zenodo.json file. This file is not part of the config.json file.

Note

To update an existing Zenodo record, use the --zenodo-id flag.

 1{
 2    "title": "",
 3    "description": "",
 4    "creators": [
 5        {
 6            "person_or_org": {
 7                "given_name": "",
 8                "family_name": "",
 9                "type": "personal"
10            }
11        }
12    ]
13}