Skip to content

Latest commit

 

History

History
21 lines (17 loc) · 2.6 KB

SHERLOCK-GUIDE.md

File metadata and controls

21 lines (17 loc) · 2.6 KB

Guide to Sherlock

Sherlock is a computing cluster operated by the Stanford Research Computing Center for research purposes. See their full documentation here. Monte Carlo simulations in this project require parallel computing (the Python multiprocessing library) and are run on the Sherlock cluster with 16-32 CPU cores.

Preparation

  1. Prepare a job file (See this guide on how to create a job file using R/Python/Bash). submit_chirp_python.py is a Python example, which takes certain simulation parameters as arguments and submits one simulation job to Sherlock. To submit multiple jobs at once, a Python file such as submit_chirp_batch_python.py loops through a set of parameters/conditions we want to vary. In addition to the simulation parameters, the files specify demands on computing resources including no. of CPUs, memory usage, time.
  2. Data transfer: tranfer .py files (programming files and job submission files) and their dependencies to your storage folder on Sherlock.
  3. Submit jobs on Sherlock

Job submission

  1. On Terminal, ssh into Sherlock by typing ssh <SUnetID>@login.sherlock.stanford.edu. Type in the password when prompted and authorize the two-factor notification.
  2. cd into the file directory
  3. Load Python module by typing ml python/3.6.1 (provided python 3.6.1 has been installed to your Sherlock directory)
  4. To submit jobs via a Python submission file, type python3 <submission_file.py> <arguments>. You should see Submitted batch job <job number> if the job submission is successful.
  5. To see the status(es) of your job(s), type squeue -u <SUnetID>

Running Jupyter notebook on Sherlock

Simulation results are saved in .csv format. In the past I transferred the files from my Sherlock directory to my local computer before processing them. I recently adopted a new (better) workflow where I run Jupyter notebook on Sherlock and access the files from there, without having to move the files to my computer. After following these instructions, I can now start a Jupyter notebook session from my local Terminal.

Additional resources