Beginner's guide

GettingStarted.md

@@ -0,0 +1,47 @@
+# Getting Started at S3DF
+
+This document will guide you through the basics of using S3DF's clusters, storage systems, and services.
+
+## Get a S3DF Account
+
+To utilize the S3DF facilities, you must first [acquire a S3DF account](accounts.md#account), and your user account should be associated with a S3DF allocation to run jobs
+
+## Connect to S3DF
+There are three different ways to [access S3DF](accounts.md#connect)
+
+## Computing Resources 
+S3DF offers a variety of high-performance computing resources that are accessible. 
+Refer to the table below to find the specifications for each cluster
+
+| Partition name | CPU model | Useable cores per node | Useable memory per node | GPU model | GPUs per node | Local scratch | Number of nodes |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| [roma](systems.md#roma) | Rome 7702 | 120 | 480 GB | - | - | 300 GB | 129 |
+| [milano](systems.md#milano)| Milan 7713 | 120 | 480 GB | - | - | 6 TB | 193 |
+| [ampere](systems.md#ampere) | Rome 7542 | 112 (hyperthreaded) | 952 GB | Tesla A100 (40GB) | 4 | 14 TB | 42 |
+| [turing](systems.md#turing) | Intel Xeon Gold 5118 | 40 (hyperthreaded) | 160 GB | NVIDIA GeForce 2080Ti | 10 | 300 GB | 27 |
+| [ada](systems.md#ada) | AMD EPYC 9454 | 72 (hyperthreaded) | 702 GB | NVIDIA L40S | 10 | 21 TB | 6 |
+
+## Storage Resources
+To ensure long-term consistency, the [S3DF directory structure](storage.md) features immutable paths that are independent of the underlying file system organization and technology.
+
+## Software
+- In addition, S3DF utilizes Lmod to manage software packages installed through alternative methods. Through Lmod, S3DF provides support for a select number of software packages that are widely utilized by the SLAC communities.
+
+- S3DF encourages experts outside of the SCS to leverage Lmod for providing, supporting, maintaining, and sharing the software tools they develop.  
+
+## Running Jobs
+There are three different ways of [run jobs](run.md) on S3DF
+- [Interactive](interactive-compute.md): Commands that you issue are executed immediately.
+- [Batch](batch-compute.md): Jobs are submitted to a queue and are executed as soon as resources become available.
+- [Service](service-compute.md): Long-lived jobs that run in the background waiting for data to analyze.
+
+## Data Transfers
+s3dfdtn.slac.stanford.edu is a load-balanced DNS name which points to a pool of dedicated data transfer nodes. It is open to everyone with an S3DF account. Common tools like scp/sftp/rsync are available for casual data transfers. For serious large volume data transfer, you may consider bbcp and globus. You can refer to [this](managedata.md) for detailed information on data transfers in S3DF
+
+# Overview of the S3DF facilities
+
+![Resource](assets/Resource.png)
+
+## Getting Help
+There are many [resources](help.md) available to assist you in utilizing S3DF effectively. The S3DF support team is always here to help you with any questions or challenges you may encounter. 
+

README.md

@@ -1,13 +1,25 @@
-Welcome to the SLAC Shared Scientific Data Facility (S3DF). The S3DF
-is a compute, storage and network architecture designed to support
+# S3DF Documentation
+
+Welcome to the SLAC Shared Scientific Data Facility (S3DF). 
+
+- The S3DF is a compute, storage and network architecture designed to support
 massive scale analytics required by all SLAC experimental facilities
 and programs, including LCLS/LCLS-II, UED, cryo-EM, the accelerator,
-and the Rubin observatory. The S3DF infrastructure is optimized for
-data analytics and is characterized by large, massive throughput, high
+and the Rubin observatory.
+- The S3DF infrastructure is optimized for data analytics and is characterized by large, massive throughput, high
 concurrency storage systems.
 
 
 ## Quick Reference
+- [Get Started](GettingStarted.md) - Information for new and existing S3DF users
+- [Beginner's Guide](beginner-user.md) - Step by step guide for new users
+- [Get Help](help.md) - How to get help
+- [OnDemand Document](interactive-compute.md#ondemand) - Access S3DF through Open OnDemand via any (modern) browser
+- [Jupyter](interactive-compute.md#jupyter) - Interactive Jupyter Notebooks at S3DF
+- [Example Job Scripts](examplescripts.md) - example job scripts
+- [Multi-Factor Authentication (MFA)](multifactor.md) - How to set up and use MFA for your S3DF account
+- [Systems](systems.md) - Computing resources at S3DF
+
 
 | Access 	| Address | 
 | :--- | :--- |
@@ -22,4 +34,4 @@ concurrency storage systems.
 | S3DF Dashboard & Monitoring | https://grafana.slac.stanford.edu|
 
 
-![SRCF-II](assets/srcf-ii.png)
+![Resource](assets/Resource.png)

_sidebar.md

@@ -1,14 +1,11 @@
 * [Welcome](/)
-* [Access](accounts-and-access.md)
-* [Usage](getting-started.md)
-* [Interactive Compute](interactive-compute.md)
-* [Batch Compute](batch-compute.md)
-* [Service Compute](service-compute.md)
+* [Get Started](GettingStarted.md)
+* [Beginner's Guide](beginner-user.md)
+* [Accounts & Access](accounts.md)
+* [Systems](systems.md)
+* [Storage](storage.md)
 * [Software](software.md)
-* [Storage](data-and-storage.md)
-* [Transferring Data](data-transfer.md)
-* [Tutorials](tutorials.md)
-* [Business Model](business-model.md)
-* [Reference](reference.md)
-* [Status & Outages](changelog.md)
-* [Contact Us](contact-us.md)
+* [Run Jobs](run.md)
+* [Manage Data](managedata.md)
+* [Get Help](help.md)
+

accounts-and-access.md

@@ -40,38 +40,36 @@ If you have forgotten your password and need to reset it, [please contact the IT
 Make sure you comply with all SLAC training and cybersecurity requirements to avoid having your account disabled. You will be notified of these requirements via email.
 
 
-## How to connect
-
-There are three mechanisms to access S3DF:
-
-1. **SSH**: You can connect using any SSH client, such as
-[OpenSSH](www.openssh.com) or
-[PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/), on the
-standard TCP port 22, to connect to the S3DF load balanced bastion pool
-`s3dflogin.slac.stanford.edu`. Note that these nodes do not have access to
-storage (except for your home directory). From these bastion hosts, you
-should hop onto an [Interactive
-Node](interactive-compute.md#interactive-pools) to access S3DF batch compute
-and storage.
-
-?> Windows users may see an error message about a "*Corrupted MAC on
-input*" or "*message authentication code incorrect.*"
-The workaround is to add "*-m hmac-sha2-512*" to the ssh command, i.e.
-`ssh -m hmac-sha2-512 <username>@s3dflogin.slac.stanford.edu`
-
-2. **NoMachine**: NoMachine provides a special remote desktop that is
-specifically designed to improve, compared to ssh, the performance of
-X11 graphics over slow connection speeds. Another important feature is
-that it preserves the state of your desktop across multiple
-sessions, including when your internet session unexpectedly gets dropped. The login pool for NoMachine is
-`s3dfnx.slac.stanford.edu`. You can find more information about this
-access mode in the [NoMachine reference](reference.md#nomachine).
-
-3. **OnDemand**: If you do not have a terminal handy or you want to
-use applications like Jupyter, you can also launch a web-based
-terminal using OnDemand:\
-[`https://s3df.slac.stanford.edu/ondemand`](https://s3df.slac.stanford.edu/ondemand).\
-You can find more information about using OnDemand in the [OnDemand
+## How to connect  :id=connect 
+
+There are three primary methods to access S3DF:
+
+1. **SSH** (Secure Shell):
+
+ - You can connect using any SSH client, such as [OpenSSH](www.openssh.com) or [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/), via standard TCP port 22 to reach the S3DF load-balanced bastion pool at s3dflogin.slac.stanford.edu
+
+        ssh username@login-node-address
+
+ - Please note that these bastion hosts do not have storage access except for your home directory. After connecting, you must hop onto an [Interactive
+Node](interactive-compute.md#interactive-pools)to access S3DF batch compute resources and storage.  
+
+        ssh username@pool-node-address
+
+ - For Windows Users: If you encounter an error message regarding a “Corrupted MAC on input” or “message authentication code incorrect,” you can resolve this by adding “-m hmac-sha2-512” to your SSH command. For example:
+
+        ssh -m hmac-sha2-512 <username>@s3dflogin.slac.stanford.edu
+
+2. **NoMachine**:
+
+ - NoMachine offers a specialized remote desktop solution that enhances X11 graphics performance over slow connections compared to SSH.
+ - An added benefit is that it maintains your desktop state across sessions, even if your internet connection is dropped unexpectedly.
+ - Use the login pool for NoMachine at s3dfnx.slac.stanford.edu. Additional details about this access method can be found in the NoMachine reference documentation [NoMachine reference](reference.md#nomachine)
+
+3. **OnDemand**:
+
+ - If you prefer not to use a terminal or want to run applications such as Jupyter, you can access a web-based terminal via OnDemand [`https://s3df.slac.stanford.edu/ondemand`](https://s3df.slac.stanford.edu/ondemand).
+ - For further information on using OnDemand, please refer to the OnDemand reference documentation [OnDemand
 reference](interactive-compute.md#ondemand).
 
+
 ![S3DF users access](assets/S3DF_users_access.png)

accounts.md

@@ -0,0 +1,50 @@
+# Accounts and Access
+
+## How to get an account :id=account
+
+### Eligibility for S3DF Accounts
+SLAC employees, affiliated researchers, and experimental facility users are eligible for an S3DF account. 
+?> Please note that S3DF authentication requires a SLAC UNIX account.
+
+### Steps to Acquire a S3DF Account
+
+#### Step 1: Obtain a SLAC UNIX Account
+If you do not already have a SLAC UNIX account, follow these steps to [get a SLAC UNIX account](slac-unix-account.md)
+
+#### Step 2: Get A S3DF Account 
+After you get a SLAC UNIX account, [register Your SLAC UNIX Account in S3DF](slac-unix-account.md#register)
+
+
+## How to connect  :id=connect 
+
+There are three primary methods to access S3DF:
+
+1. **SSH** (Secure Shell):
+
+ - You can connect using any SSH client, such as [OpenSSH](www.openssh.com) or [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/), via standard TCP port 22 to reach the S3DF load-balanced bastion pool at s3dflogin.slac.stanford.edu
+
+        ssh username@s3dflogin.slac.stanford.edu
+
+ - Please note that these bastion hosts do not have storage access except for your home directory. After connecting, you must hop onto an [Interactive
+Node](interactive-compute.md#interactive-pools)to access S3DF batch compute resources and storage.  
+
+        ssh pool-node-address
+
+ - For Windows Users: If you encounter an error message regarding a “Corrupted MAC on input” or “message authentication code incorrect,” you can resolve this by adding “-m hmac-sha2-512” to your SSH command. For example:
+
+        ssh -m hmac-sha2-512 <username>@s3dflogin.slac.stanford.edu
+
+2. **NoMachine**:
+
+ - NoMachine offers a specialized remote desktop solution that enhances X11 graphics performance over slow connections compared to SSH.
+ - An added benefit is that it maintains your desktop state across sessions, even if your internet connection is dropped unexpectedly.
+ - Use the login pool for NoMachine at s3dfnx.slac.stanford.edu. Additional details about this access method can be found in the NoMachine reference documentation [NoMachine reference](reference.md#nomachine)
+
+3. **OnDemand**:
+
+ - If you prefer not to use a terminal or want to run applications such as Jupyter, you can access a web-based terminal via OnDemand [`https://s3df.slac.stanford.edu/ondemand`](https://s3df.slac.stanford.edu/ondemand).
+ - For further information on using OnDemand, please refer to the OnDemand reference documentation [OnDemand
+reference](interactive-compute.md#ondemand).
+
+
+![S3DF users access](assets/S3DF_users_access.png)

beginner-guide.md

@@ -0,0 +1,61 @@
+# A Beginner's Guide to using S3DF 
+
+Welcome to S3DF! This guide provides a clear, step-by-step workflow for all users, particularly those with limited computing experience. In this document, we will walk you through how to:
+
+- Log in to the S3DF system
+- Navigate directories and storage spaces
+- Access supported applications
+- Prepare and submit a job script
+
+These items illustrate a typical workflow for many S3DF users, particularly those utilizing our systems for extensive calculations. These calculations may encompass simulations of physical phenomena, data pre-processing or post-processing, and various forms of data generation or analysis.
+
+Before we dive into the details, please remember that you can always reach out for [assistance](contact-us.md)
+
+##  Connect to S3DF: there are three primary methods to [access](accounts-and-access.md#connect) S3DF 
+   - **SSH** (Secure Shell):
+      - You can connect Login Node using any SSH client
+
+              ssh username@login-node-address
+
+      - After successfully connecting to the Login Node, establish a second connection to a [Pool Node] (interactive-compute.md#interactive-pools) using SSH to access S3DF batch compute resources and storage.  
+
+              ssh username@pool-node-address
+   - **NoMachine**: [NoMachine reference](reference.md#nomachine) offers a specialized remote desktop solution that enhances X11 graphics performance over slow connections compared to SSH.
+   - **OnDemand**: you can access a web-based terminal via OnDemand [`https://s3df.slac.stanford.edu/ondemand`](https://s3df.slac.stanford.edu/ondemand). For further information, please refer to [OnDemand
+reference](interactive-compute.md#ondemand).
+
+##  Storage System
+
+- You can run your desired software interactively. For instance, if you need to use HFSS, launch it from the pool node.
+- Alternatively, if you're configuring input files for other software, such as ACE3P, proceed to the next step.
+
+##  Compute Nodes
+
+- Prepare and configure the necessary input files for the software you intend to use. Ensure all files are correctly set up for your simulations.
+
+## Prepare and Submit Slurm Job Scripts
+
+- Use the sbatch command to submit your jobs to a compute node for execution.
+- Example command:
+
+            sbatch your-job-script.sbatch
+
+## Accessing Supported Applications Check Status of Running Jobs (Optional)
+
+- To monitor the status of your submitted jobs, use the following command:
+
+            squeue -u username
+
+## Get Help
+
+ - Once your jobs have completed, you can view the data output directly on the pool node to ensure results are as expected.
+
+## 8. Transfer Data (if necessary)
+
+- If you need to transfer data, connect to a data transfer node to facilitate the movement of your files.
+- Use appropriate file transfer commands (e.g., scp, rsync) to move your data to the desired location.
+
+
+By following this workflow, you can effectively utilize the S3DF system for your computational needs. 
+Ensure you have all necessary software and dependencies installed before starting, 
+and refer to additional documentation for specific software setup if needed.