-
Notifications
You must be signed in to change notification settings - Fork 0
SNARTomoPACE
PACE-tomo is a data-collection scheme developed by Rado Danev and colleagues. Briefly, it uses beam shift to collect multiple tilt series in parallel, which bypasses the rate-limiting step in tomographic data collection: stage movement.
Data collection is much faster, and is organized differently, compared to typical SerialEM data collection. Thus, the rate-limiting steps have been parallelized.
In a simple use case, we will provide one or more target files, not in on-the-fly mode, and accept all default values.
snartomo-pace --target_files "target_file*.txt" --gain_file gain_reference --eer_dir frames/
- Either target files or MDOC files must be provided.
- Either a gain file must be provided or the flag
--no_gainmust be supplied. - A directory containing EERs, MRCs, or TIFFs must be provided. If micrographs are in EER format, a gain reference must be provided.
SNARTomo will warn you if there is information that you failed to provide.
More detailed usage will be described below.
SerialEM files
A key input for SNARTomoPace is one or more target files, typically with a filename of the form *_tgts.txt. To briefly summarize how SNARTomoPACE works, it reads the target file, and determines the name of the associated tilt-series MDOC files. New movie files are then scanned, in the sequence in which the corresponding tilt-series MDOC file appears in the target file. SNARTomoPACE expects the corresponding tilt-series MDOC files (typically with a filename of the form *ts*.mdoc) to be in the same directory as the target file.
The target file(s) should follow the flag --target_files
Note that if multiple target files are specified, they must be enclosed in quotes. Conversely, quotes are not requires around a single target file.
Alternatively, MDOC files can be provided instead of one or more target files, after the flag --mdoc_files
As with target files, if more than one MDOC file is specified, they must be enclosed in quotes.
Movie files
As of 2023-02-28, SNARTomoPACE accepts EER, MRC, or TIFF movies as input. The corresponding command line must include the --eer_dir, --mrc_dir, or --tif_dir, respectively.
Last angle
In on-the-fly mode, SNARTomoPACE needs a signal to decide when to start computing the 3D reconstruction. That signal is when the last angle in the last tilt series in the target file is reached, using the flag --last_tilt
If you’re not computing on-the-fly (in other words, if you leave out the --live flag), you don’t need this parameter. Instead, SNARTomo will simply search the contents of the MDOC files, assuming that they are complete.
NOTE1: If the --last_tilt flag is provided, it is assumed that all tilt series have the same number of micrographs.
NOTE2: The last angle does not necessarily correspond to the highest tilt angle, simply the chronologically last tilt angle. Under typical usage, the first tilt angle is positive, and the last tilt angle would then be the most-negative angle.
NOTE3: It is assumed that every tilt series is collected to the same last angle. If the last angle varies, the program may get stuck while looking for a movie file which will never exist. Instead, run SNARTomoPACE separately for tilt series which are collected to a different last angle.
The examples below will assume that your movies are in EER format. With EERs, a --frame_file is required.
Most settings have a default value. The following parameters are required:
- movie directory – specified with
--eer_dir,--mrc_dir, or--tif_dir, respectively -
--target_filesor--mdoc_files– If more than one file is specified, surround them in quotes. -
--gain_fileor--no_gain– If you provide neither flag, SNARTomo will assume that you forgot about the gain file. -
--last_tilt– required only in live mode
At a minimum, you could simply type snartomo-pace --target_files "target_files*.txt" --gain_file <gain_reference> without any further options specified. I recommend a “dry run”, as described below.
Even better, enter snartomo-pace --help (or snartomo-pace -h) to see the full list of settings.
More information about the specific options can be found below.
I recommend performing a dry run, with the --testing option.
snartomo-pace --target_files "target_files*.txt" --eer_dir frames/ --frame_file motioncor_frame_file --gain_file gain_reference --last_tilt last_tilt_angle --testing
The information written to the screen before SNARTomo starts looking for individual files includes summaries of how to use SNARTomo, and the settings used. Also note the Validation section. If required inputs are missing, they will be noted here. Some sanity checks will be performed here, to make sure your parameters are sensible.
If you are still collecting data, SNARTomo will continue to look for new files, and will not stop on its own. Enter ctrl-c to exit from a dry run.
Since SNARTomoPACE is parallelized, it produces output faster, and in a wider variety. Only a summary is written to the console terminal, for example, validation, warnings, and a counter for detected files. Instead, more detailed information will be written to specialized files in the Logs output subdirectory. The main log file, Logs/snartomo.txt, will closely resemble the screen output. The MotionCor2 output file, Logs/motioncor2.txt, will contain the detailed screen output of MotionCor2, et cetera. A more complete list of the log files is listed in the Outputs section.
Once you are satisfied with the results of the dry run, remove the --testing flag.
The two general ways to process data using SNARTomoPACE are: live (on-the-fly) and batch modes.
Live mode
In live mode, the --live flag tells SNARTomoPACE to look for new EER files continuously. In addition, the --last_tilt flag tells SNARTomoPACE when to start computing the 3D reconstructions.
snartomo-pace --target_files "target_files*.txt" --eer_dir frames/ --frame_file motioncor_frame_file --gain_file gain_reference --last_tilt last_tilt_angle --live
Batch mode with target files
In batch mode, the last tilt angle is not required.
snartomo-pace --target_files "target_files*.txt" --eer_dir frames/ --frame_file motioncor_frame_file --gain_file gain_reference
In batch mode, the MDOC files will be assumed to be complete, and SNARTomoPACE will not continuously look for new EER files.
Batch mode with MDOC files
snartomo-pace --mdoc_files "mdoc_files*.mdoc" --eer_dir frames/ --frame_file motioncor_frame_file --gain_file gain_reference
tmux/screen
In both live and batch modes, there will be text written continuously to the screen, so it may be impractical to continue using that console. You might want to use tmux or screen to work elsewhere. If you would like to exit the session while SNARTomo is still running, you might want to start a tmux or screen session. Alternatively, you can simply leave that console open.
For the most current settings, enter:
snartomo-pace --help
You can alternatively enter snartomo-pace -h
Any of these parameters can be overridden on the command with the use of the appropriate flag, using the form:
snartomo-pace --flag_to-override=your_new_value
The “=” is optional, and can be replaced by one or more spaces.
Data type below is as defined in Markus Stabrin’s argumentparser_dynamic.sh.
SerialEM settings
Either one or more target files, or an MDOC file, is required.
| Flag | Type | Default | Description |
|---|---|---|---|
--target_files |
ANY | None | Input target files, surrounded by quotes if more than one |
--mdoc_file |
ANY | None | Single MDOC file |
Movie settings
One of these three flags is required.
| Flag | Type | Default | Description |
|---|---|---|---|
--eer_dir |
DIR | frames | Input EER directory |
--mrc_dir |
DIR | frames | Input MRC directory |
--tif_dir |
DIR | frames | Input TIFF directory |
Global
These settings affect multiple steps.
| Flag | Type | Default | Description |
|---|---|---|---|
--gpus |
ANY | “0 1” | GPUs to use (space-delimited and in quotes if more than one) |
--live |
BOOL | false | On-the-fly mode |
--last_tilt |
FLOAT | None | Last tilt angle, degrees (required in live mode) |
--outdir |
ANY | SNARTomo | Output directory |
--apix |
FLOAT | -1.0 | Pixel size, Å/px |
--testing |
BOOL | false | Testing mode |
--overwrite |
BOOL | false | Overwrite output directory (only if no EERs) |
--verbosity |
INT | 5 | Verbosity level (0..9) |
--max_minutes |
INT | 100 | Maximum run time, minutes |
--kv |
FLOAT | set in snartomo.bashrc | Voltage, kV |
--overwrite |
BOOL | false | Overwrite output directory (only if no EERs) |
--no_redo3d |
BOOL | false | Flag to NOT overwrite pre-existing 3D reconstructions |
--denoise_gpu |
BOOL | set in snartomo.bashrc | Testing mode |
Advanced
| Flag | Type | Default | Description |
|---|---|---|---|
--search_interval |
INT | set in snartomo.bashrc | Interval to check for files and resources, seconds |
--min_tilt |
FLOAT | None | Minimum (most negative) tilt angle to keep |
--max_tilt |
FLOAT | None | Maximum tilt angle to keep |
--tilt_tolerance |
FLOAT | set in snartomo.bashrc | Pixel size, Å/px |
--slow |
BOOL | false | In testing mode, simulates a delay in file creation |
--ram_warn |
FLOAT | set in snartomo.bashrc | Free system memory (GB) before warning printed |
--ram_kill |
FLOAT | set in snartomo.bashrc | Free system memory (GB) before exiting |
--eer_local |
BOOL | set in snartomo.bashrc | Flag to temporarily copy EER files locally |
--eer_latency |
FLOAT | set in snartomo.bashrc | Maximum header read time (seconds) before micrograph is copied locally |
--debug |
BOOL | false | Flag for debugging |
MotionCor2
| Flag | Type | Default | Description |
|---|---|---|---|
--frame_file |
ANY | motioncor-frame.txt | Input MotionCor2 frame file |
--gain_file |
FILE | None | Input gain file |
--no_gain |
BOOL | false | Flag when no gain file is to be used |
--motioncor_exe |
ANY | set in snartomo.bashrc | MotionCor2 executable |
--mcor_patches |
ANY | set in snartomo.bashrc | Number of patches in x y, delimited by spaces and in quotes |
--do_splitsum |
BOOL | False | Split frames into half-sets |
--split_sum |
INT | 0 | (DEPRECATED) Split frames into half-sets (0: no, 1: yes) |
--reffrm |
INT | 1 | Reference frame (0: first, 1: middle) |
--do_outstack |
BOOL | False | Write aligned stacks |
--mc2_wait |
FLOAT | set in snartomo.bashrc | Maximum wait (seconds) for competing user’s MotionCor2_FreeGpus.txt file to be deleted |
--min_frames |
INT | set in snartomo.bashrc | Minimum number of frames (EERs only) before printing warning |
--max_frames |
INT | set in snartomo.bashrc | Maximum number of frames (EERs only) before printing warning |
CTFFIND4
| Flag | Type | Default | Description |
|---|---|---|---|
--ctffind_dir |
ANY | set in snartomo.bashrc | CTFFIND executable directory |
--ctf_slots |
INT | set in snartomo.bashrc | Maximum number of CTFFIND4 processes to run concurrently |
--cs |
FLOAT | set in snartomo.bashrc | Spherical aberration constant |
--ac |
FLOAT | set in snartomo.bashrc | Amplitude contrast |
--box |
INT | set in snartomo.bashrc | Tile size for power-spectrum calculation |
--res_lo |
FLOAT | set in snartomo.bashrc | Low-resolution limit for CTF fitting, Å |
--res_hi |
FLOAT | set in snartomo.bashrc | High-resolution limit for CTF fitting, Å |
--df_lo |
FLOAT | set in snartomo.bashrc | Minimum defocus value, Å |
--df_hi |
FLOAT | set in snartomo.bashrc | Maximum defocus value, Å |
--df_step |
FLOAT | set in snartomo.bashrc | Defocus search step during fitting, Å |
--ast_step |
FLOAT | set in snartomo.bashrc | Astigmatism search step during fitting, Å |
--ctf1d_dpi |
INT | 250 | Resolution (dpi) of PNG of 1D CTF profile |
--ctfplot_first |
INT | -1 | Plots first CTF data for first N images in tilt series (-1 -> all) |
JANNI
| Flag | Type | Default | Description |
|---|---|---|---|
--do_janni |
BOOL | false | Flag to denoise using JANNI |
--janni_batch |
INT | set in snartomo.bashrc | Number of patches predicted in parallel |
--janni_overlap |
INT | set in snartomo.bashrc | Overlap between patches, pixels |
--janni_model |
ANY | set in snartomo.bashrc | JANNI “h5” model file |
--janni_env |
ANY | set in snartomo.bashrc | Conda environment |
Topaz
| Flag | Type | Default | Description |
|---|---|---|---|
--do_topaz |
BOOL | false | Flag to denoise using Topaz |
--topaz_patch |
INT | set in snartomo.bashrc | Patch size |
--topaz_env |
ANY | set in snartomo.bashrc | Conda environment |
--topaz_time |
ANY | set in snartomo.bashrc | Maximum duration (Topaz sometimes hangs) |
DoseDiscriminator
Settings for dose-fitting.
| Flag | Type | Default | Description |
|---|---|---|---|
--dosefit_min |
FLOAT | set in snartomo.bashrc | Minimum dose rate allowed, as a fraction of maximum dose rate |
--dosefit_resid |
FLOAT | set in snartomo.bashrc | Maximum residual during dose-fitting, as a fraction of maximum |
--dosefit_verbose |
INT | 6 | Verbosity in log file (0..8) |
IMOD
| Flag | Type | Default | Description |
|---|---|---|---|
--batch_directive |
ANY | batchDirective.adoc | IMOD batch directive file |
--imod_slots |
INT | set in snartomo.bashrc | Maximum number of IMOD reconstructions to run concurrently |
--imod_dir |
ANY | set in snartomo.bashrc | IMOD executable directory |
Ruotnocon
Settings for contour removal in IMOD reconstruction.
| Flag | Type | Default | Description |
|---|---|---|---|
--do_ruotnocon |
BOOL | false | Flag to remove contours |
--ruotnocon_sd |
FLOAT | set in snartomo.bashrc | Cutoff in units of sigma for residual |
AreTomo
| Flag | Type | Default | Description |
|---|---|---|---|
--are_bin |
INT | set in snartomo.bashrc | Binning factor for reconstruction |
--are_patches |
ANY | set in snartomo.bashrc | Number of patches in x & y (delimited by spaces) |
--dark_tol |
FLOAT | set in snartomo.bashrc | Tolerance for dark images (0.0-1.0) |
--vol_zdim |
INT | set in snartomo.bashrc | z-dimension for volume |
--rec_zdim |
INT | set in snartomo.bashrc | z-dimension for reconstruction |
--tilt_cor |
INT | set in snartomo.bashrc | Tilt-correction flag (1: yes, 0: no) |
--bp_method |
INT | set in snartomo.bashrc | Reconstruction method (1: weighted backprojection, 0: SART) |
--flip_vol |
INT | set in snartomo.bashrc | Flag to flip coordinates axes (1: yes, 0: no) |
--transfile |
INT | set in snartomo.bashrc | Flag to generate IMOD XF files (1: yes, 0: no) |
--are_time |
ANY | set in snartomo.bashrc | Maximum duration (AreTomo sometimes hangs) |
--tilt_axis |
FLOAT | set in snartomo.bashrc | Estimate for tilt-axis direction, degrees |
--aretomo_exe |
ANY | set in snartomo.bashrc | AreTomo executable |