How to use Code to Cluster

Intro

The files

• using.jl: contains

1. The using_pkg() function that install the packages if not installed yet.
2. The using_mod() function to load and define the module if not already defined. Example:

include("using.jl")
using_pkg("CUDA, JLD")
using_mod(".SSH, .JulUtils")

• JulUtils.jl: contains the JulUtils module with useful functions.

Example:

JulUtils.screensize() # return the screen size in pxl
JulUtils.generate_dataframe(listname, listtab) # generate a DataFrame with all permutaions of parameters

• PictUtils.jl: PictUtils module with: 1. 2.

example

using_mod(".PictUtils.jl")
PictUtils.pngstogif(...) # Convert all the .png of a folder in a .gif

• SSH.jl: Contains the SSH module with:
  1. basic ssh() function
  2. File module to create directory, check if a file exists, ...
  3. Print module
  4. Get module
  5. SCP module

example

• Code2Cluster.jl
• infoclusterblink.jl
• scan_param_pngs.jl
• ssh_login.jl

Login

In ssh_login.jl, you have to enter your username

# Username and host for ssh connection
# the host is the remote server name, I added the '@' for simplicity
const username = "..."; const host ="@login2.baobab.hpc.unige.ch"

Then the path of your home directory will be automatically saved in cluster_home_path:

# In ssh_utilities.jl
const cluster_home_path = "/home/users/$(username[1])/$username/"

and, in order to run an ssh command, you just have to use the function ssh() of ssh_utilies.jl:

# In ssh_utilities.jl
ssh(cmd) = readchomp(`ssh $username$host $cmd`)

General command

If you want to run a ssh command on the cluster you can use the function ssh("command"). If you want to run a ssh command and want to print the result in the console you can use ssh_print("commande"). For example if you want to check the status of the gpus:

SSH.ssh_print("squeue --nodes=gpu[020-022,027-031]")

Some general commands are already written:

SSH.Print.infogpus() # Print gpu status
SSH.Print.quota(user=username) # Print disk quota

SSH.Print.squeue(; username=username, opt="") # As squeue
SSH.Print.seff(jobID) # Print efficiency of your job

SSH.Print.scancel(jobID) # scancel

SSH.Print.out(jobID) # Print the .out of the job of id jobID
SSH.Print.lastout() # Print the .out of the last job
SSH.Print.lastout(inc) # Print the .out of the (last+inc)th job

If you include("Code2Cluster.jl") then SSH.Print. is not needed.

Run one simulation on the cluster

You have to include("Code2Cluster.jl") in console with julia. Then you can install some usefull packages using install_julia_packages().

run_one_sim() function

The function is:

run_one_sim(local_code_path, julia_filename, cluster_code_dir, cluster_save_directory, stime; partitions="private-kruse-gpu,shared-gpu", mem="3000", sh_name="C2C.sh", input_param_namefile = "InputParameters.jl", constraint="DOUBLE_PRECISION_GPU", scratch = true, download_path="/")

One example:

run_one_sim("D:/Code/mycode/", "something.jl", "mycode/", "Data/test/", "1-00:00:00"; partitions="private-kruse-gpu", dowload_path="F:/mydata/test/")

It is also possible to define a "shortcut" function, example:

cd("D:/Code/")
include("Utilities/Code2Cluster.jl")

# Shortcut to run Mathieu DPD sim on all gpus
runMat_allgpus(dir_clu, stime) = run_one_sim("Mathieu/", "main.jl", "Mathieu/", "Data/Mathieu/"*dir_clu, stime)

# An other shortcut to run Mathieu DPD sim on private gpus
runMat_privategpus(dir_clu, stime) = run_one_sim("Mathieu/", "main.jl", "Mathieu/", "Data/Mathieu/"*dir_clu, stime, partitions="private-kruse-gpu")

and then call it, the data will be save on the cluster in "Data/Mathieu/sim1/", duration of sim 12h:

runMat_allgpus("sim1/", "0-12:00:00")

Scan a parameter space

Generate a DataFrame

Direct way

Call function:

JulUtils.generate_csv(saving_directory, list_col_name, list_tab; name="DF", fn="")

For specific use

...
## Rho
tar = [5.0, 7.0, 9.0]
## Friction
txi = [1.0, 10.0, 100.0]

ttype = ["P", "Q", "PQ"]

listname = ["rho0", "rhocr", "ap", "nu", "gamma", "kp", "aq", "lambda", "Gamma", "kq", "zetap", "zetaq", "a", "ar", "xi", "ttype"]
listtab = [tρ0, tρcr, tap, tν, tγ, tkp, taq, tλ, tΓ, tkq, tζp, tζq, ta, tar, txi, ttype];

df = generate_dataframe(listname, listtab; fn="NO");
for i=1:nrow(df)
    df[i, :lambda] = df[i, :nu]
    df[i, :Gamma] = df[i, :gamma]
    if df[i,:ttype]=="Q"
        df[i, :ap] = 0.0
        df[i, :kp] = 0.0
        df[i, :zetap] = 0.0
        df[i, :nu1] = 0.0
        df[i, :gamma] = 0.0
    elseif df[i,:ttype]=="P"
        df[i, :aq] = 0.0
        df[i, :kq] = 0.0
        df[i, :zetaq] = 0.0
        df[i, :lambda] = 0.0
        df[i, :Gamma] = 0.0
    end
end
unique!(df)
insertcols!(df, 1, :fn => 1:nrow(df))
CSV.write(joinpath(dir,"DF.csv"), df)

println("Number of sims :", nrow(df))

How to enter your Input Parameters

Input Parameters to explore

include("../Utilities/using.jl")
using_mod(".JulUtils")

dir = @__DIR__

# Non-dimensionalized parameters
tρ0 = [0.8, 1.0, 1.2]
tρcr = [0.2]

## Polar
tap = [1.0]
# P - v, h, ΔμP
tν1 = [-1.0]
tγ = [1.0]
tkp = [1.0e-3]

## Nematic
taq = [1.0]
# Q - v, H, ΔμQ
tλ = [-1.0]
tΓ = [1.0]
tkq = [1.0e-3]

## Stress
# Active Polar/Nematic ζpP_zP_z, ζqQ
tζp=[0.0, 0.5, 1.0]
tζq=[0.0, 0.5]
ta = [5.0, 7.0, 9.0]

## Rho
tar = [5.0]
## Friction
txi = [1.0, 10.0, 100.0]

ttype = ["P", "Q", "PQ"]

listname = ["rho0", "rhocr", "ap", "nu1", "gamma", "kp", "aq", "lambda", "Gamma", "kq", "zetap", "zetaq", "a", "ar", "xi", "ttype"]
listtab = [tρ0, tρcr, tap, tν1, tγ, tkp, taq, tλ, tΓ, tkq, tζp, tζq, ta, tar, txi, ttype];

generate_csv(dir, listname, listtab, name="DF")

Common Input Parameters

The InputParameters.jl file

include("../Utilities/using.jl")
usingpkg("FFTW, Distributions, DelimitedFiles, CSV, DataFrames, Dates, Printf, JLD, CUDA")

idx = Base.parse(Int, ENV["SLURM_ARRAY_TASK_ID"])

dir_df = @__DIR__
df = CSV.read(joinpath(dir_df,"DF.csv"), DataFrame)[idx,:]

Δx = 4e-3; Δz = 4e-3
Δx2 = Δx*Δx; Δz2 = Δz*Δz
Δt = Float64(1e-4)
L = 10.0
N = L / Δx

## Polar
ap=Float64(df[:ap])
# P - v, h, ΔμP
ν=Float64(df[:nu]); 
γ=Float64(df[:gamma])
kp=Float64(df[:kp])

## Nematic
aq=Float64(df[:aq])
# Q - v, H, ΔμQ
λ=Float64(df[:lambda]) 
Γ=Float64(df[:Gamma])
kq=Float64(df[:kq])
...

run_arra_DF() function

# npara is the maximal number of CPUs/GPUs allowed to run simultaneously in order to not use the whole cluster
function run_array_DF(local_code_path, julia_filename, cluster_code_dir, cluster_save_directory, stime; df_name="DF.csv", partitions="private-kruse-gpu,shared-gpu", mem="3000", sh_name="C2C_array.sh", input_param_namefile = "InputParameters.jl", npara=20, constraint="DOUBLE_PRECISION_GPU", scratch = true, download_path="/")

example:

# GPU by default
run_array_DF("D:/Code/mycode/", "something.jl", "mycode/", "mydata/", "0-12:00:00", download_path="F:/Data/mydata/")
# CPU (max 200)
run_array_DF("D:/Code/mycode/", "something.jl", "mycode/", "mydata/", "0-12:00:00", partitions="shared-cpu", npara=200, download_path="F:/Data/mydata/")

This function automatically modify the "InputParameters.jl" (or the input_param_namefile) by adding/modifying these lines:

dir = "/home/users/d/dumoulil/scratch/mydata/" 
localpath = "F:/Data/mydata/" 
idx = Base.parse(Int, ENV["SLURM_ARRAY_TASK_ID"]
@show fn = "$idx/" 
file = joinpath(dir, fn) 
mkpath(file) 
println("path_c = ", dir)
println("path_l = ", localpath)
println(idx) 
# End of modification
include("../Utilities/using.jl")
using_pkg("CUDA, JLD, Statistics, Printf, FileIO, CSV, DataFrames, DelimitedFiles, Dates")
using_mod(".JulUtils")

Make your plots and explore the parameter space

In the scan_param_pngs.jl enter the correct path to the .csv file at root of your data/figure:

# Enter:
# Path to csv file
path_to_csv = "F:/2D_P_Q_PQ_long/DF.csv"
# Choose where to display in: 
# 1 - web browser page
# 2 - a standalone window 
# 3 - vscode in the HTML plot panel / Notebook / Jupyterlab / Pluto
disp = 1 

Then include("scan_param_pngs.jl"), it will open this electron window:

Job info with Blink

include("Code2Cluster.jl")
infocluster()

How to use the JupyterHub

Create your password and first login

1. Type jupyter-kruse.unige.ch in your web browser.
2. Enter your first name and choose your password

Add IJulia with optimized kernel

1. Open a terminal
2. Type julia
3. copy past

using Pkg; 
Pkg.add("IJulia"); 
using IJulia; 
installkernel("Julia opt", "-O3", env=Dict("FOO"=>"yes"))

Restart your server

1. Clic on File (top left)
2. Clic on Hub Control Panel
3. Stop my server
4. Start my server

Install CUDA for Julia

1. Open a Julia opt console
2. Copy paste

using Pkg
Pkg.add("CUDA")
using CUDA
CUDA.versioninfo()

Create your Code folder and download Utilities

1. Copy this link https://github.com/Lu-Dumoulin/Utilities , to easily copy-paste you can press shift during the right-clic in order to have the usual option
2. Right-clic -> New Folder
3. Name your folder
4. Select it in order to have it open
5. Clic on the Git icon and on Clone a Repository
6. Paste the link
7. If the Utilities folder is not at the right place you can drag and drop it in the right folder

Configure the access to the cluster from the jupyter-hub

It is necessary to use SSH key login, which removes the need for a password for each login, thus ensuring a password-less, automatic login process:

1. Open a terminal
2. Type ssh-keygen and follow instruction to generate public and private key files stored in the ~/.ssh directory (do not enter un phrase-pass)
3. Then copy the public key in the ~/.ssh/authorized_keys file, on ubuntu you can use ssh-copy-id -i ~/.ssh/id_rsa.pub user@remote_host

How to delete your data files (all the files with specific .ext)

The storage on the jupyterhub is limited, if you want to delete your data but to keep your pictures or gifs you can use the JulUtils module. The function get_all_ext("directory_path", ext=".ext") get the path of all the .ext files in the folder and sub-folders of directory_path.

Example: I want to delete all my .jld files (recursively) from my Data folder.

include("/home/jupyter-ludo/Code/Utilities/using.jl")
using_mod(".JulUtils")
all_jld_path = get_all_ext("/home/jupyter-ludo/Data/", ext=".jld")

3361-element Vector{String}:
 "/home/jupyter-ludo/Data/Asters/" ⋯ 19 bytes ⋯ "h13m51s/Data/data_000000000.jld"
 "/home/jupyter-ludo/Data/Asters/" ⋯ 19 bytes ⋯ "h13m51s/Data/data_000020000.jld"
 ⋮
 "/home/jupyter-ludo/Data/Prot/testwPQ2D_8/Data/data02490000.jld"
 "/home/jupyter-ludo/Data/Prot/testwPQ2D_8/Data/data02500000.jld"

If you want to delete these files, you can just do a for loop and use the rm() function of julia: [rm(i, force=true) for i in all_jld_path]. If you want to delete the "Data" parent directory of the .ext filea you can use the JulUtils.splitpath("full_path") function.

all_folder_path = similar(all_jld_path)
for i in eachindex(all_jld_path)
    all_folder_path[i] = JulUtils.splitpath(all_jld_path[i])[1]
end
unique!(all_folder_path)

79-element Vector{String}:
 "/home/jupyter-ludo/Data/Asters/force/2023-01-18/11h13m51s/Data/"
 "/home/jupyter-ludo/Data/Asters/force/2023-01-18/11h25m25s/Data/"
 ⋮
 "/home/jupyter-ludo/Data/Prot/testwPQ2D_7/Data/"
 "/home/jupyter-ludo/Data/Prot/testwPQ2D_8/Data/"

for i in all_folder_path
    rm(i, recursive = true)
end

In case of error:

Handler error:

1. In a terminal get the list of the kernel: jupyter kernelspec list
2. Remove the julia kernels: jupyter kernelspec uninstall julia-*
3. Install the new kernel: julia

using Pkg; 
Pkg.add("IJulia"); 
using IJulia; 
installkernel("Julia", "-O3", env=Dict("FOO"=>"yes"))