Multi-cluster Submission

[james-s-willis]

Hi,

What is the best way to submit to different clusters that have different submission hosts?

I currently have set:

---
# Required settings
apps_dir: "./apps"
scheduler: "slurm"

# Optional settings
## General
data_dir:  <%= ENV['HOME'] %>/composer
login_node: "tri-login01"
ssh_wrapper: "ssh tri-login01"

in conf.yml.erb which successfully submits jobs via the ssh submission host, tri-login01. I thought setting:

cluster:
  - name: "trillium"
    scheduler: "slurm"
    login_node: "tri-login01"
    ssh_wrapper: "ssh tri-login01"
  - name: "trillium-gpu"
    scheduler: "slurm"
    login_node: "trig-login01"
    ssh_wrapper: "ssh trig-login01"

and defining:

  _cluster_name:
    widget: select
    label: Cluster
    required: true
    direction: horizontal
    value: trillium
    options:
      - [ trillium,  "Trillium" ]
      - [ trillium-gpu, "Trillium-GPU" ]

in the header of form.yml.erb would be sufficient. But I get an error when clicking "Submit":

Job submission failed. (No such file or directory - sbatch) 

Thanks!

[mnakao]

Thank you for reaching out!

Until now, it wasn’t possible to define ssh_wrapper and login_node inside the cluster section.

As you mentioned, such environments certainly exist, so I have updated the code to allow ssh_wrapper and login_node to be defined there. Please try it with the latest main branch.

We have also updated the documentation:
https://riken-rccs.github.io/OpenComposer/docs/install.html#cluster

Thanks again for your feedback!

[mnakao]

I thought it would be useful if you could log in to multiple clusters when you click "Shell Access" in the header, but this hasn't been implemented yet. I plan to do so soon.

[james-s-willis]

Hi @mnakao,

Thank you so much for your quick response and solution! I can confirm that the latest version of Open Composer allows me to specify ssh_wrapper and login_node under the cluster section. I can connect and submit jobs to our two separate SLURM clusters.

On a separate note, in the "History" page the "Submission at" column displays:

which has the time in UTC. However, looking at what SLURM returns via the "Job ID":

it has our local time which is EST. Is there a way for me to configure the "Submission at" column to display the time in local time to match what SLURM returns in EST?

Thanks again!

[mnakao]

First of all, as I mentioned in the previous thread, you can now specify multiple login nodes in the “Shell Access” section of the header. Please give it a try.

Regarding the time zone issue, Open Composer currently follows the system time of the server on which it is running, and that server appears to be set to UTC. While it is technically possible to allow time-zone changes within Open Composer, we are hesitant to implement this due to the following reasons:

• Changing the time zone on the Open Composer side would require the external Ruby library tzinfo, and we prefer to avoid external dependencies.
• Another possible approach would be to define something like time_offset: -5:00 in config.yml.erb and forcibly adjust the display. However, this would not automatically handle U.S. daylight saving time.

As an alternative, we are currently implementing an extension that allows users to freely customize the items displayed on the History page. With this extension, for example, you would be able to remove “Submission at” and replace it with Slurm’s “Submit” field instead.

This implementation will be completed by the end of this year.

If you have any feedback or suggestions, please feel free to let me know.

Bests.

[james-s-willis]

First of all, as I mentioned in the previous thread, you can now specify multiple login nodes in the “Shell Access” section of the header. Please give it a try.

Thank you, I have also tested this and it works nicely! The only extra change I had to make was to OOD_SSHHOST_ALLOWLIST but this is to be expected.

I totally understand not implementing the time zone changes. I will update the system time of my server.

The customisable History page sounds interesting, I look forward to trying it out.

Thanks!

[mnakao]

As I wrote in the Issues page, I have created a function to edit the History Page. I would like to continue the discussion in the Issue page, so I will close the Discussion page.
#19