Quickstart¶
This page is intended to give you a quick first run through the curation and processing stages with Nipoppy using real DICOM data from an example dataset.
Important
See the Installation instructions first if you have not yet installed Nipoppy and do not forget to activate your Nipoppy environment.
Download the example dataset¶
$ git clone https://github.com/nipoppy/tutorial-dataset.git
Initializing a new dataset¶
1. Initialize a Nipoppy dataset:
$ nipoppy init --dataset my-example-study
Important
The newly created directory tree follows the Nipoppy specification. Other Nipoppy commands expect all these directories to exist – they will throw an error if that is not the case.
2. Move the example dataset and files into your Nipoppy dataset:
$ mv tutorial-dataset/manifest.tsv my-example-study
$ mv tutorial-dataset/reorg/* my-example-study/sourcedata/imaging/pre_reorg
$ mv tutorial-dataset/bidsify/dcm2bids_config.json my-example-study/code
$ mv tutorial-dataset/bidsify/participants.tsv tutorial-dataset/bidsify/dataset_description.json my-example-study/bids
See the dcm2bids docs for info about the dcm2bids_config.json and the BIDS docs for info about valid BIDS dataset requirements.
3. Change directory into your Nipoppy dataset:
$ cd my-example-study
Creating/modifying required files¶
Customizing the global configuration file¶
The global configuration file at <NIPOPPY_PROJECT_ROOT>/global_config.json starts out like this:
1{
2 "SUBSTITUTIONS": {
3 "[[NIPOPPY_DPATH_CONTAINERS]]": "[[NIPOPPY_DPATH_ROOT]]/containers",
4 "[[HPC_ACCOUNT_NAME]]": ""
5 },
6 "DICOM_DIR_PARTICIPANT_FIRST": true,
7 "CONTAINER_CONFIG": {
8 "COMMAND": "apptainer",
9 "ARGS": [
10 "--cleanenv"
11 ],
12 "ENV_VARS": {
13 "PYTHONUNBUFFERED": "1"
14 }
15 },
16 "HPC_PREAMBLE": [
17 "# (These lines can all be removed if not using HPC functionality.)",
18 "# ========== Activate Python environment ==========",
19 "# Here we need the command to activate your Python environment in an ",
20 "# HPC job, for example:",
21 "# - venv: source <PATH_TO_VENV>/bin/activate",
22 "# - conda: source ~/.bashrc; conda activate <ENV_NAME>",
23 "# ========== Set environment variables ==========",
24 "export PYTHONUNBUFFERED=1"
25 ],
26 "PIPELINE_VARIABLES": {
27 "BIDSIFICATION": {},
28 "PROCESSING": {},
29 "EXTRACTION": {}
30 },
31 "CUSTOM": {}
32}
Fields that may need to be modified depending on your setup:
If you are on a system that uses Singularity instead of Apptainer, you need to change
CONTAINER_CONFIG->COMMANDto"singularity"instead of"apptainer"If your group uses a shared directory for storing container image files, you can replace
"[[NIPOPPY_DPATH_ROOT]]/containers"by the full path to that shared directory. Alternatively, you can create a symlink from<NIPOPPY_PROJECT_ROOT>/containersto that directory (then this line in the configuration can be deleted).
Customizing the manifest.tsv file¶
The example manifest file created at <NIPOPPY_PROJECT_ROOT>/manifest.tsv by nipoppy init always looks like this:
1participant_id visit_id session_id datatype
201 BL BL ['anat']
301 M06
401 M12 M12 ['anat']
502 BL BL ['anat','dwi']
602 M06
702 M12 M12 ['anat','dwi']
For our example study, we changed it to this:
(This is the manifest.tsv file you copied from the tutorial-dataset; you will always have to modify the manifest.tsv according to your study setup)
1participant_id visit_id session_id datatype
2ED01 BL BL ['anat', 'func', 'dwi']
3ED02 BL BL ['anat', 'func', 'dwi']
4ED03 BL BL ['anat', 'func']
5ED04 BL BL ['anat']
Prepare sourcedata for bidsification¶
1. Reorganize the sourcedata to simplify bidsification:
$ nipoppy reorg
2. Check the dataset status:
$ nipoppy status
Expected output:
...
Participant counts by session at each Nipoppy checkpoint
╷ ╷ ╷ ╷
session_id │ in_manifest │ in_pre_reorg │ in_post_reorg │ in_bids
════════════╪═════════════╪══════════════╪═══════════════╪═════════
BL │ 4 │ 4 │ 4 │ 0
╵ ╵ ╵ ╵
...
Pipeline setups¶
A newly initialized Nipoppy dataset does not contain any pipeline setups or containers.
dcm2bids example¶
1. Search for the desired pipeline:
$ nipoppy pipeline search dcm2bids
2. Copy the Zenodo ID of version 3.2.0 of the pipeline (16876754 at the time of writing) and run:
$ nipoppy pipeline install 16876754
3. Choose to install the container as well or not: y/n
4. Check pipeline installation:
$ nipoppy pipeline list
Expected output:
...
INFO Available bidsification pipelines and versions:
INFO - dcm2bids (3.2.0)
...
Bidsify the sourcedata¶
Note
Please see the dcm2bids documentation to know what dcm2bids does and how it works.
Usually you would start with running nipoppy bidsify with the first --pipeline-step (e.g. prepare). For dcm2bids this step would run the dcm2bids_helper in order to extract information from the dicom headers to create a dcm2bids_config.json. We already provided you with a dcm2bids_config.json, so we will skip this step here.
1. Replace the placeholder for "DCM2BIDS_CONFIG_FILE" in the global_config,json with the path to your code directory:
{
# ...
"PIPELINE_VARIABLES": {
"BIDSIFICATION": {
"dcm2bids": {
"3.2.0": {
"DCM2BIDS_CONFIG_FILE": "[[NIPOPPY_DPATH_ROOT]]/code/dcm2bids_config.json"
}
}
}
}
# ...
}
2. Run bidsification:
$ nipoppy bidsify --pipeline dcm2bids --pipeline-step convert
3. Track the curation status:
$ nipoppy track-curation --regenerate
The curation status file can be found at <NIPOPPY_PROJECT_ROOT>/sourcedata/imaging/curation_status.tsv.
4. Check the dataset status:
$ nipoppy status
Expected output:
...
Participant counts by session at each Nipoppy checkpoint
╷ ╷ ╷ ╷
session_id │ in_manifest │ in_pre_reorg │ in_post_reorg │ in_bids
════════════╪═════════════╪══════════════╪═══════════════╪═════════
BL │ 4 │ 4 │ 4 │ 4
╵ ╵ ╵ ╵
...
Run a processing pipeline on BIDS data¶
1. Search and install the MRIQC pipeline, version 23.1.0 (and if necessary the container) as described above.
2. Check the pipeline installation:
$ nipoppy pipeline list
Expected output:
...
INFO Available bidsification pipelines and versions:
INFO - dcm2bids (3.2.0)
INFO Available processing pipelines and versions:
INFO - mriqc (23.1.0)
...
3. Create a new directory in the Nipoppy dataset root called templateflow (required by MRIQC, see TemplateFlow docs):
$ mkdir templateflow
4. Replace the placeholders (here: the templateflow path) in the global_config.json:
{
# ...
"PIPELINE_VARIABLES": {
"BIDSIFICATION": {
"dcm2bids": {
"3.2.0": {
"DCM2BIDS_CONFIG_FILE": "[[NIPOPPY_DPATH_ROOT]]/code/dcm2bids_config.json"
}
}
},
"PROCESSING": {
"mriqc": {
"23.1.0": {
"TEMPLATEFLOW_HOME": "[[NIPOPPY_DPATH_ROOT]]/templateflow"
}
}
},
"EXTRACTION": {}
}
# ...
}
Note
You can also point to an already existing shared templateflow directory, if you have access to one.
5. Run MRIQC on one participant:
$ nipoppy process --pipeline mriqc --participant-id ED01
6. Track the processing status:
$ nipoppy track-processing --pipeline mriqc
The processing status file can be found at <NIPOPPY_PROJECT_ROOT>/derivatives/processing_status.tsv.
7. Check the dataset status:
$ nipoppy status
Expected output:
...
Participant counts by session at each Nipoppy checkpoint
╷ ╷ ╷ ╷ ╷
│ │ │ │ │ mriqc
│ │ │ │ │ 23.1.0
session_id │ in_manifest │ in_pre_reorg │ in_post_reorg │ in_bids │ default
════════════╪═════════════╪══════════════╪═══════════════╪═════════╪══════════
BL │ 4 │ 4 │ 4 │ 4 │ 1
╵ ╵ ╵ ╵ ╵
...
Tip
You can also upload your processing status file to the Neurobagel digest dashboard at https://digest.neurobagel.org/ to get a nice summary and visualization of your work!
Next steps¶
Repeat the steps of the previous section with a processing pipeline such as fmriprep to consequently use the nipoppy extract command with the fs_stats pipeline. Try it out, it will be fun!