Lab storage project bootstrap (Copier template)

Scaffolds a new project into the group's /data storage layout:

• team project → creates both source_data/<slug>/ (raw, read-only landing zone) and analysis_data/<slug>/ (your working project, optionally git + DVC)
• personal / exploratory → creates home/<you>/<slug>/ only, nothing on shared storage

Git and DVC are both optional and independent of each other. Git is recommended but never required; DVC can be initialized with --no-scm if git is off.

One-time setup (per user)

pip install --user copier          # or: pipx install copier

Bootstrap a new project

Run this from wherever you believe /data's root is (i.e. the parent of source_data/, analysis_data/, home/, etc.):

cd /data
curl -fsSL https://raw.githubusercontent.com/imsb-uke/copier-template/main/bootstrap.sh | bash

This checks you're actually in the right place before asking anything (see "Are you even in the right directory?" below), then hands off to Copier's normal questions. If you'd rather skip that check and call Copier directly, that still works too:

cd /data
copier copy --trust gh:<your-org>/<this-template-repo> .

(--trust is required because the template runs a couple of housekeeping tasks after copying: git init/git commit, and dvc init if you asked for it. Read copier.yml before trusting any template you didn't write yourself — that's the point of the flag.)

You'll be asked:

Question	Notes
project type	team (source_data + analysis_data) or personal (home/)
project name	human-readable
project slug	auto-derived, lowercase-hyphenated folder name
owner	your username, matches your home/<user> folder
PI / group	team projects only
description	one-liner, ends up in the README and manifest
data sensitivity	public / internal / patient-related — team projects only
reference data used	free text note, purely documentation
data type(s)	multi-select from common omics/imaging types (below) — also drives the modality scaffold, see below. Blank = minimal project
init git	default yes
init DVC	default no
DVC remote URL	only if DVC is on; can be set later too
backup	should Amanda pick this up? default yes for team, no for personal

Data type choices offered (pick any that apply, or "other" + free text): scRNA-seq/snRNA-seq, spatial transcriptomics (Visium/Xenium/MERFISH/CosMx), scATAC-seq, bulk RNA-seq, WGS/WES, long-read sequencing (ONT/PacBio), proteomics (mass spec), flow cytometry/CyTOF, DNA methylation (bisulfite/EM-seq), Hi-C/chromatin conformation, whole-slide histology (H&E/IHC), multiplex imaging (CODEX/PhenoCycler/Xenium/MERFISH imaging), confocal/light-sheet microscopy, MRI/CT/PET (DICOM), cryo-EM.

What you get

Every project (team or personal) gets:

• README.md — human-readable, filled in from your answers
• PROJECT.yml — the same info as structured YAML, so it's greppable / scriptable across the whole /data tree later (e.g. to build an index of what's out there, or to check who owns what and how sensitive it is)
• notebooks/, scripts/ (+ results/, env/ for team projects)
• .gitignore tuned for the folder

Team projects additionally get source_data/<slug>/ as a documented, untouched landing zone for raw/incoming files, kept separate from analysis_data/<slug>/ (which is what you'd actually put under git/DVC).

Modality scaffold

There's no separate "modalities" question — the data_type multi-select is the modality list. By default (nothing selected) a project is just README.md + PROJECT.yml. Pick one or more data types and each one becomes a modality folder:

source_data/<slug>/<data-type-slug>/
  raw/            untouched incoming files
  processed/      scripts/  logs/  output/
  metadata_table  sample/metadata table for this modality
  README.md

analysis_data/<slug>/<data-type-slug>/
  01_preprocessing/  scripts/  notebooks/  output/  environment/  README.md
  02_quality_control/  README.md
  README.md

analysis_data/<slug>/environment/   (shared, project-wide)

The folder name is the slug from the data_type choices (e.g. spatial-transcriptomics, scrna-seq). If you pick "Other / not listed" and fill in data_type_other, that free text is slugified into its own modality folder too (data_type_other isn't itself a modality slug, so "other" alone contributes nothing — you need the free-text answer for it to produce a folder).

02_quality_control/ only gets a README by default — add scripts/, notebooks/, output/, environment/ under it by hand if you want the same sub-structure as 01_preprocessing; that wasn't specified precisely enough in the source layout to assume it automatically.

Because Copier can't loop a single templated path over an arbitrary-length answer, this part isn't done via path templating like the rest of the template — it's a shell loop in _tasks that runs before git init, so the modality tree is present in the very first commit.

Are you even in the right directory?

Copier's execution model is fixed: it asks every question first, then renders/copies files, then runs _tasks — there's no supported way for a copier.yml-based check to run before the first question. So a check implemented only as a task (as this used to be) can never appear before project_type is asked; it can only run as the very last step, after you've already answered everything and files have already been written.

The actual fix is bootstrap.sh at the repo root — a plain bash wrapper you run instead of calling copier copy directly:

curl -fsSL https://raw.githubusercontent.com/imsb-uke/copier-template/main/bootstrap.sh | bash

It runs the directory check in bash before Copier is even invoked, so a wrong-directory run gets caught before the first question, not after the last one:

WARNING: none of source_data/, analysis_data/, reference_data/,
public_data/, home/ exist in /some/random/dir.
This does NOT look like the root of the shared data storage.
Continue anyway and bootstrap the project (and the standard skeleton) here? [y/N]

• Anything other than y/yes exits immediately — Copier is never even invoked, so nothing gets fetched or created.
• y/yes builds the full standard skeleton (all five top-level folders, including the nested reference_data/genomes/, reference_data/ databases/, reference_data/models/{classification,deconvolution, segmentation,llm}/) and then hands off to copier copy --trust <template-url> . for the normal interactive questions.
• If a directory already looks legitimate (has any of those folders already), it skips straight to Copier with no prompt.

copier.yml still carries its own copy of this same check as a task (runs last, prompts the same way, cleans up the just-created project files on decline) — that's a defensive fallback for anyone who runs copier copy directly rather than through bootstrap.sh, not the primary mechanism anymore.

About "No git tags found in template; using HEAD as ref"

That's informational, not an error — it means Copier didn't find any git tags on the repo, so it fell back to the latest commit on the default branch. You are getting the current version. If you want reproducible, pinned versions instead of always-latest (e.g. so an old project doesn't silently pick up template changes on copier update), tag releases:

git tag v1.0.0 && git push --tags

Backup (Amanda)

Each project's PROJECT.yml gets a backup: true|false flag, and — if true — an empty .backup file (gitignored everywhere by default). That file is not an Amanda-native mechanism; it's a plain log file meant for whatever wraps amdump on your side to append one line to per successful run, so a project directory itself shows its own backup history at a glance. Amanda never writes to it on its own.

Actual inclusion/exclusion in Amanda works differently, because of how Amanda is structured:

• Amanda backs up whole disklist entries (DLEs) — configured server-side in disklist/amanda.conf, typically one DLE per top-level directory (e.g. /data/analysis_data as one DLE, /data/source_data as another, plus one per home/<user>). Amanda has no concept of "scan this tree and figure out what changed" beyond that DLE boundary.
• What it does support at the DLE level is a single exclude list file, read from the root of that DLE, in GNU-tar --exclude-from format (paths relative to the DLE root, starting with ./) — configured via exclude list ".amanda.excludes" in the dumptype (see amanda.conf(5)).

So the practical mapping is: one exclude file per DLE root (analysis_data/.amanda.excludes, source_data/.amanda.excludes, home/<user>/.amanda.excludes), each listing the immediate subfolders to skip, e.g. ./some-project. scripts/sync-amanda-excludes.sh regenerates those files by scanning every PROJECT.yml under /data for backup: false — run it from /data (by hand, or from cron before each amdump):

scripts/sync-amanda-excludes.sh /data

On the Amanda server side, that just means each relevant dumptype needs:

define dumptype lab-standard {
    ...
    exclude list ".amanda.excludes"
}

That's a one-time config change on whoever administers Amanda; this repo only owns keeping the exclude files themselves current.

Updating projects later (why Copier over Cookiecutter)

Cookiecutter is fire-and-forget: once a project is generated, there's no supported way to pull in template improvements. Copier tracks which template + version + answers were used to generate a project, so later you can run:

cd analysis_data/<slug>
copier update --trust

...to pull in structural changes made to this template (e.g. if the group decides to add a manuscripts/ folder to every project, or tighten the .gitignore) into projects that already exist, via a 3-way merge instead of manual copy-paste.

Files in this repo

copier.yml           <- questions + validators + post-gen tasks
                         starts with `_subdirectory: template`, which tells
                         Copier "the actual template content lives one
                         folder down" — see below.
template/
  {% if project_type == 'team' %}source_data{% endif %}/{{ project_slug }}/...
  {% if project_type == 'team' %}analysis_data{% endif %}/{{ project_slug }}/...
  {% if project_type == 'personal' %}home{% endif %}/{% if project_type == 'personal' %}{{ owner }}{% endif %}/{{ project_slug }}/...
scripts/
  sync-amanda-excludes.sh   <- admin tool, run at the /data root, not per-project

The {% if ... %} folder names are Copier's way of conditionally including whole subtrees — a segment that renders to an empty string is skipped entirely, which is how one template produces either the team layout or the personal layout depending on your answer.

copier.yml sits at the repo root (not inside template/) specifically so that scripts/ and this README.md are not part of what gets copied into a new project — only whatever _subdirectory points at is. Without that split, running Copier against the repo root would copy scripts/ and this file verbatim into every new project, which isn't what you want.

Distributing this to the group — no local clone needed

Push this repo to wherever the group can reach it (internal GitLab/GitHub works, so does a plain path on shared storage). People then run Copier straight against that source — Copier fetches it internally (into its own cache, not a folder you manage), so nobody needs to git clone it themselves first:

cd /data
copier copy --trust gh:<org>/<repo> .          # GitHub shorthand
# or
copier copy --trust https://gitlab.example.org/group/lab-project-template.git .
# or, from a local/shared-storage path, also without any manual clone step:
copier copy --trust /data/reference_data/templates/lab-project-template .

Two things worth knowing about how that fetch behaves:

• Copier does its own shallow git checkout under the hood into a local cache directory it manages (not visible as a normal clone in your working directory) — that's expected and not something to clean up by hand.
• By default it checks out the latest git tag on the repo (sorted as a version number), not necessarily the tip of the default branch. Tag releases (git tag v1.1.0 && git push --tags) if you want people to get a specific, reproducible version; use --vcs-ref=HEAD (or a branch/commit) to always get the latest, untagged state instead.