DevelopersGuide.md

Developers Guide

Welcome to the developer's guide for the interactive sessions repository. While this repository offers one method for creating interactive workflows in Parallel Works, please note that it's not the sole approach available. Ensure you review the information provided in these links on interactive workflows and developing workflows before proceeding with this guide.

This repository enables you to establish interactive session workflows for running various services, such as Jupyter or remote desktops, across different types of executors. The supported executors encompass your Parallel Works user workspace as well as remote on-premises or cloud clusters. When running on a cluster, we offer support for execution on the controller node, a SLURM partition, or a PBS queue. The session.sh script is launched on the selected executor to initiate the service..

The following files are executed in the specified order when a job is launched:

1. main.sh: Executes in the user workspace and contains the top-level logic for loading and preparing inputs, as well as the service.json file. This script also calls the <exec_mode>/session_wrapper.sh script.
2. <exec_mode>/session_wrapper.sh: Executes in the user workspace, generating the session.sh by wrapping the <service_name>/start_template.sh script. Subsequently, it launches the session.sh script on the designated executor. The session.sh script encapsulates the service code and oversees SSH tunnel management.

When a job is canceled, the following files are executed in this order:

1. kill.sh: Executes in the user workspace and is generated by the <exec_mode>/session_wrapper.sh script to terminate the session if the job is canceled.
2. <service_name>/kill-template.sh: Executes in the selected executor and is used created by the by the <exec_mode>/session_wrapper.sh script to create the kill.sh. This scripts contains the commands that are specific to terminate the service.

Repository Structure

The repository is structured as follows:

Execution Mode Directories

The Execution mode directories contain the wrapper script to execute the service in different types of executors:

• local: User workspace
• controller: Controller node of the cluster
• partition: SLURM partition or PBS queue

Workflow Directory

The workflow directory contains workflow-specific information and is further categorized into the following subdirectories:

• readmes: Every file contains the description of a given workflow. The files are organized by service and each service has its own subdirectory.
• xmls: Contains the workflow XML files for each workflow. Remember this file is used by Parallel Works to render the input form and execute the workflow. The files are organized by service and each service has its own subdirectory. See workflow XML section for more information.
• thumbnails: Contains the thumbnail icon for each workflow. Typically, all workflows that run the same service use the same thumbnail.

Service Directories

The rest of the directories are service directories and contain the specific code to start the service.

Workflow XML

The workflow XML encompasses both input parameters and execution details for each workflow. For a comprehensive understanding of workflow XML files, please consult the user guide. In this instance, it adheres to the subsequent structure:

• Command and cancellation items define the execution and cancellation process for the workflow.
• The service_name parameter is set as a fixed value corresponding to the service directory's name.
• The resource section adheres to the resource wrapper tool's format. For further details, refer to this link.
• Additional parameters and sections are also included.

Adding a New Service

Follow this steps to add a new service using the interactive sessions repository.

1. Create the Service Directory

You can either create a new service directory from scratch or duplicate an existing one that closely matches your requirements. Within the service directory, you'll be working with several scripts. These scripts have access to environment variables with the input parameters defined in the input form as well as any variables introduced by the workflow. Here are the key scripts:

• start-template.sh: Use this script to write the necessary code to start the service.

• url.sh: This script is responsible for writing the SLUG parameter within the file service.json. Visit this link for more information about the service JSON file. In most cases, this script sets the SLUG parameter to """.

• kill-template.sh: This script serves the purpose of terminating the service initiated by start-template.sh when the workflow is canceled. For instance, if you employ docker run -d within start-template.sh, this script becomes essential for stopping the Docker container. It is automatically executed when a job is canceled within the PW platform.

2. Create the README and XML Directories

These components serve as the front end of the workflow, encompassing the description and XML files for each workflow that initiates this service. It's important to note that a single service can have various workflows tailored to strike the right balance between customization and generality. For instance, some workflows might require the exposure or hardcoding of specific SLURM directives, such as the account or the number of tasks per node for a particular cluster. Another common scenario involves desktop sessions that launch different software applications like MATLAB or RSTUDIO. This approach alleviates the need for users to complete extensive forms when submitting jobs to a designated cluster.

When crafting the XML file, we recommend starting with an existing one and making necessary adjustments to the service section to align with your requirements. It's crucial to ensure that the service_name parameter in the XML matches the name of the service directory.

3. Add a Thumbnail for the Service

Place a 120x120 PNG image into the thumbnails directory to serve as the workflow's thumbnail.