Upload v2 data to GCP

[adamklie]

Context

Upload v2 results for igvf-data/igvf_sc-islet_10X-Multiome to GCP via bin/9_upload/1_upload_to_gcp.ipynb.

Upgraded to high priority 2026-04-16. This is the canonical v2 snapshot that downstream collaborators (Gaulton lab for #2, cross-dataset integrative analyses for #9/#10) pull from. Doubles as the template for the v3 upload once #8 ATAC integration lands — i.e. this defines the shared object layout that "define → GCP-ize → v3-ize" will reuse for every subsequent analysis.

Tasks

• Design step — define the manifest: exhaustive list of what gets uploaded (files + sizes + provenance) and the bucket layout / directory structure / naming convention. Write as a MANIFEST.md / LAYOUT.md before any upload happens. No upload until this is agreed.
• Format and upload single-cell matrices (RNA + ATAC)
• Upload cell-type annotations (harmony_round_2_leiden_1.0_cell_type_annotation.csv)
• Upload the 3 v2 peak sets (pseudoreplicates / timepoints / no_replicates) with post-processed + annotated consensus peaks
• Format and upload pseudobulk matrices (RNA + ATAC)
• Format and upload differential analysis results (when Finalize DESeq2 pseudobulk methodology for RNA and ATAC #4 finalized)
• README + provenance notes at the bucket root so the layout is self-documenting for downstream consumers

[adamklie]

Dependencies

• Blocked by Finalize DESeq2 pseudobulk methodology for RNA and ATAC #4 (needs finalized differential analysis), Finalize peak calling #5 (needs finalized peaks)

[adamklie]

Status Update (2026-04-09)

Still blocked on #4 (DESeq2 finalization). #5 (peaks) is done. Low priority — focusing on upstream analyses first.

[adamklie]

Status Update (2026-04-16)

Upgraded to high priority — this is the shared snapshot Sara (Gaulton lab) needs for peak-to-gene linking (#2), and the template for the v3 upload once #8 ATAC integration finishes.

New framing: GCP upload is a design step, not a dump

Rather than uploading ad-hoc as each upstream analysis finalizes, we'll treat this as the place to define the canonical object layout — what lives where, naming conventions, accompanying metadata/README — so every subsequent analysis has a fixed target to write into. That layout then gets mirrored for v3. Same pattern as the v2→v3 pipeline comparison: define the comparison structure once, reuse it.

Scope expanded

Body updated. Added: cell-type annotations, peak sets (all 3), explicit design-step task that blocks actual upload. No files go up until the manifest/layout is agreed — otherwise we'll end up reorganizing the bucket mid-project.

First concrete deliverable

Write MANIFEST.md + LAYOUT.md:

• Full file-by-file list of what's uploaded (size, provenance, upstream script)
• Bucket directory structure (e.g. gs://<bucket>/v2/{single-cell,pseudobulk,peaks,diff,annotations}/...)
• Naming conventions
• README that explains the layout to downstream consumers (Sara, integrative analyses, etc.)

Once agreed, upload finalized pieces (matrices, annotations, peaks) immediately; DE + peak-to-gene links fill in as #4 and #2 land.

[adamklie]

v2 upload archived; v3 9_upload pending (2026-04-28).

v2 upload code lives at `archive/v2/bin/9_upload/`. Bucket `gs://igvf-data/igvf_sc-islet_10X-Multiome/` (project `vocal-lead-490915-t9`) reflects the pre-archive bin/results layout, NOT the new v3 structure.

The MANIFEST.md / LAYOUT.md design task flagged on 04/16 is still the prerequisite. Worth deciding:

• Will the v3 GCP bucket follow the new v3 path structure (`bin/{1_get_data,2_sample_qc,3_cell_annotation,4_pseudobulking,5_peak_calling}/` + `results/` mirror)?
• Or does the GCP submission spec dictate a flatter layout decoupled from v3 directory structure?

Once decided, port `bin/9_upload/` from v2 — likely needs `${DATASET_ROOT}` parameterization + bucket layout map. Tracked under #8 for the v3 port mechanics.

[adamklie]

04/29 — separator convention for public files

Project convention reinforced today (per user): files we publish (IGVF portal, GCP bucket, cross-collaborator deliverables) should never have + between covariates in compound names — always -. Internal/intermediate files default to - too, but legacy notebook code that still emits + won't be proactively patched.

Practical implication for the GCP layout / manifest design: the canonical names should use - everywhere (cell_type-condition, cell_type-condition-timepoint, sample_id-cell_type, etc.). Any v2 artifacts on disk with + get renamed at upload time, not in-place in the v2 archive.

Saved as a feedback memory so future submissions don't re-introduce +. Not a unilateral move on the manifest design — just documenting the constraint.