Home
- 2023-11-30 – added SNARTomoHeatwave documentation
- 2023-09-14 – added SNARTomoEERMerge documentation
- 2023-08-30 – added SNARTomoAnimate documentation
- 2023-07-05 – PACE version writes temporary files to shared memory
- 2023-06-23 – added SNARTomoAnimate, which generates movies (MPEG, MP4, etc.) from MRC volumes/stacks
- 2023-06-08 – added deconvolution filter from IsoNet
- 2023-05-16 – moved more parameters to snartomo.bashrc
- 2023-02-28 – added support for MRC and TIFF movies
- 2022-12-09 – posted on GitHub
- 2022-11-28 – added FrameCalc to generate MotionCor frames file
- 2022-11-18 – PACE version can accept multiple targets files
- 2022-09-08 – contour-removal implemented
- 2022-07-25 – denoising using JANNI
- 2022-07-22 – dose-fitting implemented
- 2022-03-12 – can use example MDOC file as input
- 2022-03-09 – denoising (2D) using Topaz
- 2022-02-28 – can compute reconstruction using IMOD instead of AreTomo
- 2022-02-07 – exits upon unfamiliar command-line options
- 2021-12-12 – all options can be set from command line
- 2021-12-11 – times out after specified number of minutes
- 2021-12-09 – on-the-fly mode works correctly
Denoising now tested and working using multiple GPUs.
It is assumed that you have working versions of the following:
- MotionCor2
- CTFFIND4
- IMOD
- Topaz or JANNI (if denoising, in their own conda environments)
- AreTomo (unless using IMOD’s eTomo exclusively)
- IsoNet (if applying deconvolution filter)
- RELION (if compressing EERs)
- ffmpeg (if animating MRCs)
A. Create and activate conda environment
conda create --name snartomo python=3.7 matplotlib scipy tqdm mrcfile gnuplot pyqt pillow --channel conda-forge --yes
conda activate snartomo
B. Clone repository
git clone https://github.com/rubenlab/snartomo.git
C. Copy snartomo.bashrc.template to snartomo.bashrc, edit with valid paths. For example:
#!/bin/bash
# Modified 2024-09-02
##### PLEASE DEFINE THE FOLLOWING #####
export SNARTOMO_CONDA="/home/tapu/local/miniconda3/bin/conda" # Conda path (if conda is not in your $PATH)
export SNARTOMO_ENV="snartomo" # SNARTomo conda environment
export DENOISE_GPU="true" # Denoise (Topaz or JANNI) using GPUs
export MOTIONCOR2_EXE="/home/tapu/local/motioncor2/1.6.4/MotionCor2_1.6.4_Cuda115_Mar312023" # MotionCor2 exectuable
export CTFFIND4_BIN="/home/tapu/local/ctffind4/4.1.14/" # CTFFIND4 directory (not executable)
export JANNI_ENV="janni" # JANNI conda environment
export JANNI_MODEL="/home/tapu/local/janni/gmodel_janni_20190703.h5" # JANNI "h5" model file
export TOPAZ_ENV="topaz" # TOPAZ conda environment
export IMOD_BIN="/usr/local/IMOD/bin" # IMOD executable directory
export ARETOMO_EXE="/home/tapu/local/aretomo/1.3.4/AreTomo_1.3.4_Cuda115_Feb22_2023" # AreTomo executable
export CP_EER_LOCAL="false" # Copy EER file temporarily to local drive (useful for Falcon 4i data)
export SNARTOMO_LOCAL="/tmp/SNARTomo-${USER}" # Local directory for temporary EERs
export SNARTOMO_SHARE="/dev/shm/SNARTomo-${USER}" # Shared-memory directory
export ISONET_ENV="isonet" # IsoNet conda environment
# Global parameters
export SNARTOMO_VOLTAGE=300.0 # Acceleration voltage (kV) -- F20: 200, Krios: 300
export SNARTOMO_INTERVAL=2 # Monitors files & resources every N seconds
export SNARTOMO_MINFRAMES=200 # Minimum number of frames (EERs only) before warning
export SNARTOMO_MAXFRAMES=600 # Maximum number of frames (EERs only) before warning
export SNARTOMO_EER_WAIT=8 # (seconds) EER files are copied locally if header takes longer than this time
export SNARTOMO_WAIT_TIME=120.0 # (seconds) How long to wait for MotionCor2 or CTFFIND4 /tmp files to be removed
export SNARTOMO_RUOTNOCON_SD=3.0 # Contours with eTomo residuals greater than this multiple of sigma will be removed
#
# SNARTomoPACE only
export SNARTOMO_GPUS="0" # GPUs to use, delimited with spaces
export SNARTOMO_TILT_TOLERANCE=0.2 # Angle difference of last tilt +/- this value will signal end of series
export SNARTOMO_RAM_WARN=20 # Free system memory (GB) before warning printed
export SNARTOMO_RAM_KILL=10 # Free system memory (GB) before exiting
export SNARTOMO_IMOD_SLOTS=2 # Maximum number of IMOD processes to run concurrently
# MotionCor2
export SNARTOMO_MC2_PATCH="0 0" # number of patches in x y, delimited by spaces
# CTFFIND
export SNARTOMO_CTF_SLOTS=2 # Maximum number of CTFFIND4 processes to run concurrently (PACE only)
export SNARTOMO_CTF_CS=2.7 # Spherical aberration (mm) -- F20: 2.0, Krios: 2.7
export SNARTOMO_AC=0.07 # Amplitude contrast: 0.07-0.1 for cryo data, 0.14-0.2 for neg.stain data
export SNARTOMO_CTF_BOXSIZE=512 # Box size
export SNARTOMO_CTF_RESLO=30.0 # (Angstroms) Low resolution limit for CTF fitting
export SNARTOMO_CTF_RESHI=9.0 # (Angstroms) High resolution limit for CTF fitting
export SNARTOMO_CTF_DFLO=30000.0 # (Angstroms) Minimal defocus value to consider during fitting
export SNARTOMO_CTF_DFHI=70000.0 # (Angstroms) Maximal defocus value to consider during fitting
export SNARTOMO_DF_STEP=500.0 # Defocus step
export SNARTOMO_CTF_DAST=1000 # Astigmatism restraint
# JANNI
export SNARTOMO_JANNI_BATCH=4 # Number of patches predicted in parallel
export SNARTOMO_JANNI_OVERLAP=24 # (pixels) Overlap between patches
# TOPAZ
export SNARTOMO_TOPAZ_PATCH=2048 # Patch size
export SNARTOMO_TOPAZ_TIME=2h # Maximum duration, Topaz sometimes hangs, needs to be longer for CPU
# DoseDiscriminator
export SNARTOMO_DOSEFIT_MIN=0.1 # Minimum dose rate allowed, as a fraction of maximum dose rate
export SNARTOMO_DOSEFIT_RESID=0.1 # Maximum residual during dose-fitting, as a fraction of maximum dose rate
# AreTomo parameters
export SNARTOMO_BINNING=8 # Binning factor
export SNARTOMO_VOL_ZDIM=1600 # z-dimension for volume
export SNARTOMO_REC_ZDIM=1000 # z-dimension for 3D reconstruction
export SNARTOMO_TILT_AXIS=86.0 # (degrees) Estimate for tilt-axis direction
export SNARTOMO_DARKTOL=0.7 # Tolerance for dark images
export SNARTOMO_TILTCOR=1 # (boolean) Tilt-correct (1: yes, 0: no)
export SNARTOMO_BP_METHOD=1 # Reconstruction method (1: weighted backprojection, 0: SART)
export SNARTOMO_FLIPVOL=1 # (boolean) Flip coordinates axes (1: yes, 0: no)
export SNARTOMO_TRANSFILE=1 # (boolean) Generate IMOD XF files
export SNARTOMO_ARETOMO_PATCH="0 0" # Number of patches in x & y (slows down alignment)
export SNARTOMO_ARETOMO_TIME=60m # Maximum duration, AreTomo sometimes hangs
# IsoNet parameter
export SNARTOMO_SNRFALLOFF=0.7 # Signal-to-noise falloff
##### YOU SHOULDN'T NEED TO MODIFY ANYTHING BELOW THIS LINE #####
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
echo -e "\nERROR! Script '${BASH_SOURCE[0]}' is being executed!"
echo -e " Please run it with: source ${BASH_SOURCE[0]}\n"
else
# Check if conda is in your PATH
if ! [[ $(type -P conda) ]] ; then
# Ensure that exectuable exists
if ! [[ -e "${SNARTOMO_CONDA}" ]] ; then
echo -e "\nERROR! Conda executable '${SNARTOMO_CONDA}' does not exist!"
echo -e " Install Anaconda (https://www.anaconda.com/products/distribution) or Miniconda (https://docs.conda.io/en/latest/miniconda.html) and continue...\n"
return
fi
fi
# If conda isn't initialized, then initialize it
if [[ $CONDA_SHLVL -eq 0 ]] ; then
echo "Initializing conda..."
CONDA_BINDIR=$(dirname ${SNARTOMO_CONDA})
CONDA_TOPDIR=$(dirname ${CONDA_BINDIR})
__conda_setup="$('${SNARTOMO_CONDA}' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "${CONDA_TOPDIR}/etc/profile.d/conda.sh" ]; then
. "${CONDA_TOPDIR}/etc/profile.d/conda.sh"
else
export PATH="$CONDA_BINDIR:$PATH"
fi
fi
unset __conda_setup
fi
echo -e "\nScript ${BASH_SOURCE[0]} is being sourced..."
conda activate ${SNARTOMO_ENV}
export SNARTOMO_DIR="$( realpath $(dirname $BASH_SOURCE) )"
export PATH="$PATH:$SNARTOMO_DIR"
echo -e "Added '$SNARTOMO_DIR' to PATH"
# Sanity check
if [[ "${CONDA_DEFAULT_ENV}" != "${SNARTOMO_ENV}" ]] ; then
echo -e "\nERROR! Current conda environment ('${CONDA_DEFAULT_ENV}') is not '${SNARTOMO_CONDA}'!"
echo -e " Fix $(realpath ${BASH_SOURCE[0]}) or set conda environment manually\n"
return
else
echo "Activated conda environment '${CONDA_DEFAULT_ENV}'"
fi
# Conda's Gnuplot gives me a warning
export GNUPLOT_DRIVER_DIR="$CONDA_PREFIX/libexec/gnuplot/$(gnuplot --version | cut -d' ' -f 2)"
echo
fi
D. Source this newly-created file with:
source $SNARTOMO_DIRECTORY/snartomo.bashrc
Substitute the path of $SNARTOMO_DIRECTORY as needed.
I suggest creating an alias, and adding it to your ~/.bashrc file, that is:
alias use_snartomo‘source $SNARTOMO_DIRECTORY/snartomo.bashrc’=
From any new terminal, entering use_snartomo will henceforth source this file.
It is assumed that you have micrograph movies in EER, MRC, or TIFF format.
SNARTomoFrameCalc generates this file, given a few input parameters.
The frame file is required only for EER files during MotionCor2.
The frame file is used by MotionCor2 for dose-weighting. It is a text file containing the following three values, separated by spaces:
- The number of frames to include
- The number of EER frames to merge in each motion-corrected frame
- The dose per EER frame
The first value will probably simply be the total number of frames collected. However, you might limit the number frames to include if, for example, your exposure was too long, and you want to keep only the least radiation-damaged frames.
For the second value, a reasonable rule of thumb is to accumulate 0.15-0.20 electrons per Å 2. For example, at a dose of 3e/Å 2 distributed over 600 frames, the dose per EER frame would be 0.005. To accumulate 0.15e/Å 2, you would need to merge 0.15/(3/600) = 30 frames. The line in the frame file would thus be:
600 30 0.005
In theory, a frames file can have multiple lines, if for example the first N frames were to be handled differently than the next M frames. However, we haven’t tested this functionality yet.
The gain file is used to compensate for systematic errors in the camera.
The gain file can either be in TIFF or MRC format. If it’s in TIFF format, SNARTomo will automatically convert it to MRC for use with MotionCor2.
If you don’t have a gain file, include the --no_gain flag, or else SNARTomo will assume that you forgot about it.
SNARTomoGUI is a web interface to help you generate a SNARTomo command.
https://rubenlab.github.io/snartomo-gui/#/
The web interface will generate command lines for SNARTomoPACE, SNARTomoClassic, and SNARTomoFrameCalc.
Unless noted, the following options are available with both SNARTomoPACE and SNARTomoClassic.
AreTomo
AreTomo is currently the default, used if you don’t have an eTomo batch directive file (see below).
Starting with AreTomo 1.2, the program now removes dark images by default, with a parameter called “DarkTol”. This parameter (default: 0.7) can be changed with the SNARTomo flag --dark_tol.
Another important parameter is the number of patches. During alignment, AreTomo can optionally divide the micrograph into patches. This parameter is set using the flag --are_patches. The default is “0 0”. To use a 5x5 array of patches, for example, use “5 5”. (The quotes around “5 5” are required.) The alignment will be better, but the cost is speed. Without patches, AreTomo takes <2 minutes, whereas with a 5x5 array, AreTomo takes ~30 minutes.
eTomo
This option is for people who prefer the reconstruction from IMOD’s eTomo, and/or would like to refine the reconstruction later on using eTomo.
Currently, an extra input is required: a “batch directive” file, containing the various settings that you wish to use.
An additional parameter is required: --batch_directive batch_directive.adoc
NOTE: Users are responsible for making sure their batch directive files are consistent with their data. Currently, only the pixel size is confirmed.
JANNI and Topaz are two software packages for denoising.
JANNI
To perform denoising in SNARTomo using JANNI, add the --do_janni flag.
SNARTomo checks whether the JANNI executable is available (that is, whether it is in your $PATH). If it isn’t in your $PATH, SNARTomo tries to load the JANNI conda environment, as defined in your snartomo.bashrc file (see above).
Topaz
For on-the-fly processing, denoising in 2D is currently the only feasible option. Denoising in 3D would be too slow.
To perform denoising in SNARTomo using Topaz, add the --do_topaz flag.
One notable parameter is the --topaz_patch flag. If not specified, the entire micrograph would be denoised all at once. Even with our 11GB video cards, Topaz complains, so this parameter is set to 2048 by default.
To see the full set of parameters, add the --help flag, or check the SNARTomo web interface.
SNARTomo checks whether the Topaz executable is available (that is, whether it is in your $PATH). If it isn’t in your $PATH, SNARTomo tries to load the Topaz conda environment, as defined in your snartomo.bashrc file (see above).
This functionality requires MDOCs.
See the Outputs section below for the color code.
The strategy is adapted from Felix Bäuerlein. It consists of three steps:
- Fits the dose rates to a cosine curve (implemented in Python by Tat Cheng)
- Removes the points below a threshold dose rate and re-fits
- Removes the points exceeding a threshold residual
Since the dose can vary considerably, the thresholds are given as a fraction of the maximum dose rate.
-
--dosefit_min– minimum dose rate allowed (default: 0.10 * maximum dose rate) -
--dosefit_resid– maximum residual (default: 0.10 * maximum dose rate)
For eTomo reconstructions, the routine for removing bad contours is called “Ruotnocon” (reverse of “no contour”). Contours are removed on the basis of high residuals. The following is an extreme example:
Ruotnocon is controlled with the following parameters:
-
--do_ruotnocon– True or false (default: false) -
--ruotnocon_sd– Number of standard deviations for residuals, beyond which contours will be removed (default: 3)
Contour-removal
A plot of the sorted residuals can be found in Images/Contours:
The horizontal line represents the cutoff.
SNARTomoPACE
SNARTomoPACE is the PACE-tomo version of SNARTomo. Since the tilt series are collected in parallel, multiple GPUs are required to keep up.
SNARTomoClassic
SNARTomoClassic is the non-PACE version of SNARTomo. Since the images are collected at a lower rate, a single GPU is sufficient to keep up.
SNARTomoHeatwave
SNARTomoHeatwave is the graphical browser for visualization and curation of SNARTomo output.
SNARTomoFrameCalc
SNARTomoFrameCalc generates a MotionCor2 frames file for EERs.
SNARTomoAnimate
SNARTomoAnimate generates movies from MRC stacks.
SNARTomoEERMerge
SNARTomoEERMerge compresses EER files.