Y26-094 - As a TL (Tom), I would like email notifications to be sent to the SSR team when long-running accessioning jobs fail so that they know there has been a failure.

[TWJW-SANGER]

User story

As a TL (Tom), I would like email notifications to be sent to the SSR team when long-running accessioning jobs fail so that they know there has been a failure.

Who are the primary contacts for this story
SSRs, Stephen I, Stephen H, Dasun

Who is the nominated tester for UAT
We can test in UAT internally.

Acceptance criteria

Consider if these features can be featured flagged to decouple testing and deployment.
To be considered successful the solution must allow:

• Testing in UAT should not result in emails being sent to users.
• Any accessioning failures should result in an email notification
• Multiple accessioning failures in a batch job should result in only one email notification.
• Any calls to the Notification API are logged.

Dependencies
#5631

Additional context
It would be useful for UAT to have a different target email address than in production.

You might need to check that SequenceScape instances in UAT and prod can connect to the Notification API. (Networking, OpenStack security groups etc).

Deployment notes

[dasunpubudumal]

Please find the documentation for the Integration Hub Notification APIs below:

• User Guide
• API Documentation

[dasunpubudumal]

We (@StephenHulme and I) identified three cases for notifications:

• “In sample, alias: "9c45cb8e-2768-11f1-b6be-0242e2a7617b". No new objects can be added with MODIFY action”: This error is seemingly a configuration issue. We discussed it to be notified with a text with some like “There is a configuration mismatch” in the template. This is largely due to the ambiguity of the error response from EBI itself.

• “Sample '8295STDY16683588' cannot be accessioned: Sample does not have the required metadata: phenotype, sample-common-name, and sample-taxon-id.”: This is a well-detected error in Sequencescape. We can extract which columns are erroring because this is a validation error coming from Sequencescape (not EBI). It would be possible to include more information in the notification in this case. Information such as errored attributes and aggregated statistics – such as number of samples failed within the current notification batch – would be helpful for the stakeholders.

• “Sample: 39af8ff6-4d0b-11f0-8c5d-0242eb1badd5; Attribute: geographic location (country and~1or sea); Reason: must be equal to one of the allowed values: ["Afghanistan","Albania"…]”: These are error responses that potentially have information but are difficult to parse. In this case, we could display a truncated version of the error message for the stakeholders.

---

There are some common cases for all three of the above-identified cases.

• The “template” – in the language of the Notification API – is structured data.

  • For each case, Sequencescape decides which information to send (using the template (as structure) and the arguments (as data) provided through to the template).
  • The Notification API merely “renders and delivers”.
  • Thus, three cases identified above would require three templates.

• Sequencescape has retry pattern implemented for accessioning. By default (unless it is overridden by configuration?) it retries the accession three times and abandons the accession after the third call. The Notification API would need to be invoked after this third call.

  • If the number of retries are configurable, the API should be called after the final call.

• One of complexities identified was the data volume. There could be thousands of samples failing within a 5-minute window which the aggregation occurs.

  • Each of the templates used in Notification API calls with priority = BATCH needs to check the number of aggregated notifications using the fields attribute list. I have attached a preliminary template I’m working on currently as a reference in the documentation (it's in the collapse callout titled "Styles in templates"). If the number of aggregated notifications would exceed the readable limit in the rendered template, we should summarise and/or truncate information sent via Sequencescape.

  For more information on aggregation, please refer to the documentation.

• Aggregation in the Notification API is handled as a best-effort asynchronous process. Because it relies on the number of threads the platform allocates, aggregated messages may sometimes be distributed across multiple notifications. However, in our testing, for large batches, we have observed that the system often consolidates a high volume of notifications into a relatively small number of aggregated notifications, which is desirable.

---

Postman collection has been handed over to the team¹ and the API documentation is publicly available.

---

For authentication (as mentioned in the documentation), we recommend using a OAuth-recommended client library as the authentication (and authorisation) deals with sensitive data.

Footnotes

1. The Postman Collection that was handed over does not contain the client credentials. The client credentials were shared securely with our credentials repository, so those client credentials needs to be updated into the Postman environment for testing. ↩

[StephenHulme]

Do we have an email address we would like the production notifications to go to?

And should UAT or Training accessions be sent to anyone being trained?