Y26-093 - As a TL (Tom) I would like to create an email notification template for accessioning failures to be used with the Notification Service

[TWJW-SANGER]

User story
As a TL (Tom) I would like to create an email notification template for accessioning failures to be used with the Notification Service.

Who are the primary contacts for this story
Stephen I, Stephen H, Dasun, Katy T, Tom W

Who is the nominated tester for UAT
We can test this internally.

Acceptance criteria

Consider if these features can be featured flagged to decouple testing and deployment.
To be considered successful the solution must allow:

• An agreed and tested JINJA template for accessioning failure emails.
• The template should allow aggregation of similar messages. For example if 5000 samples in a bulk accessioning attempt fail, there should be only one email sent.
• Test recipients should differ from production recipients to enable testing

Additional context

User Guide: https://integration-hub.sanger.ac.uk/docs/notification-api/
API Documentation: https://integration-hub.sanger.ac.uk/docs/notification-api/api-docs

Deployment notes

[TWJW-SANGER]

User Guide: https://integration-hub.sanger.ac.uk/docs/notification-api/
API Documentation: https://integration-hub.sanger.ac.uk/docs/notification-api/api-docs

[dasunpubudumal]

An example MJML for accessioning template I created:

<!-- This is Accessioning MJML created to generate the Jinja template used to create the accessioning failure notification. -->
<!-- Please feel free to use it as a starting point in setting up your template. -->
<!-- MJML Documentation: https://mjml.io/ -->
<!-- In this template, you have access to a list of every attribute listed inside your `fields` attribute in each notification. -->
<!-- NB: Note the use of <mj-raw>. It is required for MJML transpiler to not to strip-off any Jinja-related syntax. -->

<mjml>
   <mj-head>
      <mj-title>Problems with Accessioning</mj-title>
      <mj-preview>There are validation errors in the samples provided</mj-preview>
      <mj-style inline="inline">
         .email-card {
         box-shadow: 0 6px 20px rgba(0, 0, 0, 0.12);
         }
      </mj-style>
      <mj-attributes>
         <mj-all font-family="Arial, Helvetica, sans-serif" />
         <mj-text font-size="14px" color="#333333" line-height="1.6" align="left" />
      </mj-attributes>
   </mj-head>
   <mj-body background-color="#f4f6f8">
      <mj-wrapper></mj-wrapper>
      <mj-wrapper css-class="email-card" background-color="#ffffff" border="1px solid #e0e0e0" border-radius="8px" padding="0">
         <!-- Header Section -->
         <mj-section background-color="#747067" padding="20px 25px 10px">
            <mj-column>
               <mj-text font-size="20px" font-weight="bold" color="#ffffff" align="left">
                  ⚠️ Problems with Accessioning ⚠️
               </mj-text>
            </mj-column>
         </mj-section>
         <mj-section background-color="#ffffff" padding="20px 25px">
            <mj-column>
               <mj-text>
                  There are validation errors in the samples provided for accessioning. This notification includes details of 
                  <b>
                     <mj-raw> {{ fields | length }} </mj-raw>
                  </b>
                  &nbsp;accessioning samples.
               </mj-text>             
            </mj-column>
         </mj-section>
         <!-- Beginning of content -->
         <!-- If 100 or more samples, display a compact summary -->
         <mj-raw>{% if fields|length >= 100 %}</mj-raw>
         <mj-raw>{% set samples = fields | map(attribute='sample-id') | list | unique | list %}</mj-raw>
         <mj-section background-color="#ffffff" padding="0 25px 12px 25px">
            <mj-column>
               <mj-text>
                  More than 100 samples have errored when accessioning. 
               </mj-text>
            </mj-column>
            <mj-column background-color="rgb(251, 233, 173)" border="1px solid #d2c85b" border-radius="6px" padding="0">
               <mj-text font-size="14px" color="#333333" align="left" background-color="#f1f3f5" padding="12px 15px">
                  <mj-raw>{% set unique_attribute_values = fields | map(attribute='attributes') | sum(start=[]) | map(attribute='key') | list | unique | list %}</mj-raw>
                  <mj-raw>{% if unique_attribute_values | length == 0 and fields["manifest-id"] %}</mj-raw>
                  Some samples in the manifest 
                  <b>
                     <mj-raw>{{ fields["manifest-id"] }}</mj-raw>
                  </b>
                  &nbsp;that have errored, due to incorrect values.
                  <mj-raw>{% elif unique_attribute_values | length > 5 %}</mj-raw>
                  Some samples in the manifest 
                  <b>
                     <mj-raw>{{ fields[0]["manifest-id"] }}</mj-raw>
                  </b>
                  &nbsp;that have errored, due to incorrect values for fields like 
                  <mj-raw>: {{ unique_attribute_values[:5] | join(", ") }}</mj-raw>
                  .
                  <mj-raw>{% else %}</mj-raw>
                  Some samples in the manifest 
                  <b>
                     <mj-raw>{{ fields[0]["manifest-id"] }}</mj-raw>
                  </b>
                  &nbsp;that have errored, due to incorrect values for fields like 
                  <mj-raw>: {{ unique_attribute_values | join(", ") }}</mj-raw>
                  .
                  <mj-raw>{% endif %}</mj-raw>
               </mj-text>
            </mj-column>
         </mj-section>
         <mj-raw>{% else %}</mj-raw>
         <!-- Display individual fields when fewer than 100 samples -->
         <mj-raw>{% for field in fields %}</mj-raw>
         <mj-section background-color="#ffffff" padding="0 25px 12px 25px">
            <mj-column background-color="#fafafa" border="1px solid #e0e0e0" border-radius="6px" padding="0">
               <mj-text font-size="14px" font-weight="bold" color="#333333" align="left" background-color="#f1f3f5" padding="12px 15px">
                  Sample ID: 
                  <mj-raw>{{ field["sample-id"] }}</mj-raw>
               </mj-text>
               <mj-divider border-color="#e0e0e0" padding="0" />
               <mj-raw>{% for attribute in field.attributes %}</mj-raw>
               <mj-text font-size="13px" color="#555555" padding="6px 15px" align="left">
                  <strong style="color:#000000;">
                     <mj-raw>{{ attribute.key }}</mj-raw>
                     : 
                  </strong>
                  &nbsp;
                  <mj-raw>{{ attribute.value }}</mj-raw>
               </mj-text>
               <mj-raw>{% endfor %}</mj-raw>
            </mj-column>
         </mj-section>
         <mj-raw>{% endfor %}</mj-raw>
         <mj-raw>{% endif %}</mj-raw>
         <!-- End of content -->
         <!-- Footer section -->
         <mj-section background-color="#ffffff" padding="8px 25px 20px 25px">
            <mj-column>
               <mj-text font-size="12px" color="#888888" align="center">
                  This is an automated message. Please do not reply.
               </mj-text>
            </mj-column>
         </mj-section>
      </mj-wrapper>
      <mj-wrapper></mj-wrapper>
   </mj-body>
</mjml>

This would then be transpiled into HTML using the MJML transpiler.

The resulting template rendered using the Jinja template renderer would look like the following:

This kind of an aggregated notification could be created by multiple notifications of the following nature:

{
    "channels": [
        {
            "type": "EMAIL",
            "recipient": [
                "X_EMAIL"
            ],
            "subject": "Accessioning Failure: Issues with the payload!!!!!!!",
            "content_type": "html",
            "template_id": "X-ACCESSIONING-BATCH",
            "fields": {
                "attributes": [
                    {
                        "key": "Country",
                        "value": "United Kindm"
                    }
                ],
                "sample-id": "40STDY246690"
            }
        }
    ],
    "priority": "BATCH",
    "aggregator_id": "324555"
}

Note: MJML has a vscode extension which was invaluable for creating this.

[dasunpubudumal]

Note

The content in this comment was adjusted by ChatGPT. I gave it my version of these information and it summarised into a more "readable" format. Although the format was generated by ChatGPT, the content was given to it by myself and it merely adjusted it to this format. I've fact-checked the content after feeding it through GPT.

Notification Templates & Rendering

The “template” – in the language of the Notification API – is structured data.

For each case, Sequencescape determines:

• What information to send, using:
  • the template (structure), and
  • the arguments (data) passed into the template

The Notification API is responsible only for:

• Rendering
• Delivering

Therefore, the three cases identified above would each require:

• Three separate templates

---

Retry Behaviour & Notification Triggering

Sequencescape implements a retry pattern for accessioning:

• By default, it:
  • Retries 3 times
  • Abandons the accession after the third attempt
• (This may be configurable)

Notification API Invocation

• The Notification API should be invoked:
  • After the final retry attempt
• If retry count is configurable:
  • The API must be called after the last configured retry

---

Handling High Data Volume

A key complexity identified is data volume:

• Potentially thousands of samples may fail within a 5-minute aggregation window

Template Considerations (Priority = BATCH)

Each template must:

• Check the number of aggregated notifications using:
  • the fields attribute list

If the rendered output exceeds a readable limit:

• Apply one or more of the following:
  • Summarisation
  • Truncation

---

Aggregation Behaviour

For more details, see:

• Notification API Aggregation Documentation

Key Points

• Aggregation is:
  • Best-effort
  • Asynchronous
• It depends on:
  • Platform thread allocation

Observed Behaviour

• Aggregated messages may be split across multiple notifications
• However, in testing:
  • Large batches are often consolidated into:
    • A small number of aggregated notifications
• This behaviour is:
  • Desirable

[dasunpubudumal]

The Postman collection along with the environments has been shared :)

[StephenHulme]

PS: to preview the rendered template in VS-Code using the recommended extension above, the leading comments need to be removed (see mjmlio/vscode-mjml#57)

[StephenHulme]

Test recipients should differ from production recipients to enable testing

@dasunpubudumal is there a test instance versus a production instance? And can they have different templates?
I'd like to replace the blue bar at the top with a red one (like in Sequencescape UAT) for emails sent by the test environment.