Scripts to use smart-copy functionality on Flywheel

Overview

This repository contains scripts to perform soft-copy operations on Flywheel, enabling efficient data management across projects (for a given subject cohort, create a project that contains soft links to all source data without hard-copying). The workflow consists of two main steps:

1. Discovery: Locate matching Flywheel data based on subject identifiers
2. Copy: Perform soft-copy operations to a destination project

Prerequisites

• Python 3.x
• Flywheel SDK
• Valid Flywheel API credentials
• (Optional but likely required) Access to D3b warehouse for source data queries

Quick Start

# Step 1: Find and catalog subjects
python find_fw_data.py --subjects-csv subjects.csv --output-csv fw_data.csv

# Step 2: Soft-copy to destination
python fw_soft_copy_sdk.py --input-csv fw_data.csv --destination-project group/project --yes

Main Usage

Create CSV of subject IDs

CSV Format: A single column CBTN Subject ID with the list of subjects to include in the cohort (C-IDs)

Step 1

find_fw_data.py is a command-line script. It takes subject input (CSV), finds matching Flywheel data, and writes a CSV with project/subject/session info.

Required environment variable:

• FW_API_KEY : user's Flywheel API key (unless passed with --api-key)

Optional environment variable:

• cbtn_all_table : path to CBTN-all CSV (used when --input-mode cbtn-all and --cbtn-all-csv is not passed)

If using D3b warehouse as source (--source d3b_warehouse), also set:

• DB_URL : D3b warehouse connection URL
• DB_USER : warehouse username
• DB_PASSWORD : warehouse password

Command usage

python find_fw_data.py --subjects-csv gfac/sub_list.csv --output-csv gfac/cbtn_selected_fw.csv

python find_fw_data.py --input-mode cbtn-all --diagnosis-filter "High-Grade Glioma" --output-csv gfac/cbtn_selected_fw.csv

Main arguments:

• --source {d3b_warehouse,flywheel} (default: d3b_warehouse)
• --copy-level {subject,session} (default: session)
• --input-mode {subjects-csv,cbtn-all} (default: subjects-csv)
• --subjects-csv path to CSV containing CBTN Subject ID
• --output-csv output file path (required)
• --subject-id-column custom subject ID column name (default: CBTN Subject ID)
• --cbtn-all-csv optional path override for CBTN-all table
• --diagnosis-filter text filter on CNS Diagnosis Category in cbtn-all mode

Step 2

fw_soft_copy_sdk.py is a command-line script that takes the CSV output from Step 1 and runs Flywheel soft-copy operations into a destination project.

Behavior:

• Copies at the session level when Session column exists (or --copy-level session is set).
• Copies at the subject level when Session is absent (or --copy-level subject is set).
• Skips rows that already exist in the destination project.

Command usage

python fw_soft_copy_sdk.py --input-csv gfac/cbtn_selected_fw.csv --destination-project gfac/your_destination_project --source-group d3b --yes

Dry-run first:

python fw_soft_copy_sdk.py --input-csv gfac/cbtn_selected_fw.csv --destination-project gfac/your_destination_project --source-group d3b --dry-run --yes

Main arguments:

• --input-csv input CSV path (required)
• --destination-project destination path in form <group>/<project_label> (required)
• --source-group source group label used for lookups (default: d3b)
• --copy-level {auto,subject,session} (default: auto)
• --project-column (default: Project)
• --subject-column (default: CBTN Subject ID)
• --session-column (default: Session)
• --dry-run print planned copies without launching them
• --yes skip interactive confirmation prompt
• --strict stop on first copy error
• --api-key (default: FW_API_KEY environment variable) Flywheel API key

Additional scripts

Output Contents of Project To CSV

python pull_az_file_manifest.py

As written, this script will create a file manifest based on the contents of the az_imaging_delivery project. It is an example script that can be adapted to create manifests of other projects as needed, to support file delivery or other cases where it is useful to be able to list the contents of the project.

An example of the output of this python script is in this repoistory: AZ_manifest_2026-04-10.csv