Slurm support

[samuelcwilliams]

Added support for Slurm to uit.py.

[samuelcwilliams]

@sdc50 @araglu here's the draft pull request. I'll address the issues in the pipeline in a little bit, but in the meantime I'd appreciate any feedback y'all have!

[araglu]

I think some of these changes will also need to be made in async_client.py. It has very similar code to uit.py but in an async context. I noticed our app is calling the async get_raw_queue_stats(), so it did not get your changes.

[samuelcwilliams]

Will do! I started on it a while back, but ran into some bugs with what I already had in uit.py so I reverted the changes so I could start fresh once I had those bugs worked out.

I'll get those updates rolled in ASAP!

[araglu]

Is this the right spot for these two lines? I get an error on them whenever I try to submit a new job or delete that job.

In uit.py, these lines are in prepare_connect() which is called by both sync and async connect(), so async_client.py might not need these lines here.

[samuelcwilliams]

Oh, I guess I didn't realize it was calling prepare_connect() from the regular client. I'll make the changes ASAP

[araglu]

The async submit() is still using a hard-coded "qsub" command in async_client.py:590.

Edit: the error message from qsub is:

sbatch: error: invalid partition specified: background
sbatch: error: Batch job submission failed: Invalid partition name specified

I thought using the qsub compatibility command would work until we get pyuit to write out a Slurm script, but I can't get it to work, even with other queues.

[araglu]

The "Max Wall Time" and "Max Processes" don't look right to me. Wall time said 10080 when I was expecting it in a 04:00:00 format. The processes looked more like number of nodes when I'm used to seeing number of cores.