Enable data files from the same sequencing library to be collected as a logical unit

[brianraymor]

User Story
A consumer can download a uniformly processed matrix file which reflects all the data processing results from one sequencing library such as all sequencing lane replicates for a high coverage 10x experiment being processed together

Demoable Criteria
Process a dataset with a multi lane/machine sequencing strategy for a single library and confirm our results are comparable to the outputs produced by the submitting lab.

Success Metric
We can detect and correctly process data sets where the libraries are found in multiple sequencing lanes.

[justincc]

There has been discussion about implementation but due to Barcelona the sentiment was to delay. There are a number of implementations options of varying complexity, all of which will likely require cross-component coordination.

[justincc]

To follow up, a presentation was given to Tech Arch a couple of weeks ago and @diekhans and @barkasn agreed to write up an RFC with next steps.

[morrisonnorman]

A stakeholder meeting was held on 9th Sept to identify next steps.
Present: @lauraclarke @tburdett @barkasn @diekhans @TimothyTickle @morrisonnorman

RFC Accepted:

• Representing sequencing library preparations in the HCA DCP metadata standard

RFC Due for completion of community review Friday 13th, Tech Arch Meeting:

• Processing Datasets that Span Multiple Data Collection Runs

Next steps:

1. Identify technical team to create an implementation plan (implementation spike) resulting from each RFC above.

Dependencies. It was identified that implementation for each RFC can proceed in parallel.

Note that there is a dependency for data migration of legacy data that library prep updates should be carried out first.

[justincc]

In the tech-arch call of 2019-09-13 it turns out the processing datasets rfc needs more work.

[brianraymor]

@TimothyTickle (on behalf of the DCP Project Leads) responded to the at risk escalation of this objective:

In the last PM meeting, it was requested we make a decision on the theme “Enable data files from the same sequencing library to be collected as a logical unit” indicating if it is kept or removed from the year 2019Q4 ask (and subsequent engineering plan). The theme owner Laura Clarke (with the support of PM team representatives from the PM call Ambrose Carr, Trevor Heathorn, and Tim Tickle) agrees the work should be deferred to a later quarter to free up bandwidth for the team. This work is considered important but not urgent giving the packaging ingest can perform to make data appropriately packaged for current pipelines. This work will be evaluated to be incorporated into the long term road map.