-
Notifications
You must be signed in to change notification settings - Fork 206
MMseqs2 Developer Guide
Braces are also used where they would be optional Braces are used with if, else, for, do and while statements, even when the body is empty or contains only a single statement.
For example, a preprocessor macro that could go wrong when leaving out braces:
#define MACRO test1; \
test2;
if (true)
MACRO
This would get expanded to the following:
if (true) {
test1;
}
test2;
Keep the beginning brace in the same line as control structures, functions, etc.:
if (true) {
Please avoid:
if (true)
{
Write your names as descriptive and long as necessary, but also as short as possible.
Example: Use weights instead of wg. Do not unnecessarily expand the name to sequenceWeights, if its clear from the context that you are dealing with a sequence. However, if you dealing with profileWeights and sequenceWeights in the same context, feel free to use longer names.
Only iterator variables are supposed to be one letter long (i, j, k).
Class names are written in UpperCamelCase.
Example: ClusterAlgorithm
Method names are written in lowerCamelCase.
Method names are typically verbs or verb phrases.
Example: sendMessage, stop, clusterMethod
Constant names use CONSTANT_CASE: all uppercase letters, with each word separated from the next by a single underscore.
Example: CLUSTER_ALGORITHM
Non-constant field names (static or otherwise) are written in lowerCamelCase.
These names are typically nouns or noun phrases.
Example: computedValues or index
Local variable names are written in lowerCamelCase. Even when final and immutable, local variables are not considered to be constants, and should not be styled as constants.
Be generous with white spacing horizontally, but try to keep code compact vertically.
Here a _ characters indicates where you should be placing a space character:
if_(int_name)_{
____int_test_=_1_+_(1_+_x);
}
Use empty lines to structure code in logical blocks or tasks.
Do not use printf, std::cout, std::err (etc.) for printing messages. All output have to go through the Debug logging class.
We do not use Exceptions in our code. We have two types of errors in MMseqs2. Exceptions are disabled per compile flag.
Write a descriptive error message with Debug(Debug::ERROR) and exit out immediatly with the EXIT macro.
Dot not use exit directly. EXIT handles cleaning up remaining MPI instances (if compiled with MPI).
size_t written = write(dataFilefd, buffer, bufferSize);
if (written != bufferSize) {
Debug(Debug::ERROR) << "Could not write to data file " << dataFileNames[0] << "\n";
EXIT(EXIT_FAILURE);
}
Write to Debug(Debug::WARNING) and continue with the next loop iteration or whatever is appropriate.
if (std::remove(dataFileNames[i]) != 0) {
Debug(Debug::WARNING) << "Could not remove file " << dataFileNames[i] << "\n";
}
We use OpenMP to run multiple threads for parallel computing.
Do not use pthreads or std::thread.
The standard pattern for doing anything with OpenMP looks something like this:
// Declare only thread-safe stuff here
#pragma omp parallel
{
// PER THREAD VARIABLE DECLARATION
unsigned int threadIdx = 0;
#ifdef OPENMP
threadIdx = (unsigned int) omp_get_thread_num();
#endif
// somtimes you want schedule(static)
#pragma omp for schedule(dynamic, 1)
for (...) {
// DO YOUR WORK
}
// CLEAN UP MEMORY IF NECCECARY
}
Try to avoid #pragma omp critical and #pragma omp atomic. Consider using atomic instructions instead (e.g. __sync_fetch_and_add).
Allocate memory as early as possible. Try not to allocate memory on the heap inside your hot loops:
#pragma omp parallel
{
// try to allocate once here
char MEMORY[1024 * 1024 * 1024];
// also for containers
std::vector<int> results;
results.reserve(1024);
#pragma omp for schedule(static)
for (...) {
// not here
}
Try to avoid using too many C++ features. MMseqs2 is coded in a way where we do not use not too many concepts from modern C++.
Generally you have to support GCC 4.8, this is enforced by the Continous Integration system.
It is more like C style C++. We do use classes to organize code. Some STL functionality should be used std::string, std::vector, sometimes also std::map (careful!).
However, weight any new C++ concept heavily and try to avoid them as much as possible.
Especially, do not use:
auto- streams (they can be extremely slow, instead use
std::string s; s.reserve(10000);outside a loop and insides.append(...); s.clear();) - smart pointers (try to use RAII for allocation as much as possible)
- functional programming
- inheritance (think about it very carefully, its usually a lot less useful than it appears)
You will still find some std::stringstream littered throughout our codebase, we are trying to progressivly get rid of those and not to add any new ones.
Some modern C++ features are very useful.
For example, std::vector::emplace_back can avoid memory allocations for example:
// two allocations
vector.push_back(Struct(1, 2, 3));
// one allocation
vector.emplace_back(1, 2, 3);
Take a look at all the classes in the src/common subfolder.
They contain a lot of useful stuff like Util, FileUtil, MathUtil, Itoa, etc.
Try not to reimplement stuff that exists already.
For bioinformatics, understand how to use the Sequence, QueryMatcher, Matcher, etc. classes.
To add a workflow or an util tool to MMseqs2 you need register your workflow or module in the src/mmseqs.cpp file.
A new command generally looks something like this:
{"search", search, &par.searchworkflow, COMMAND_MAIN,
"Search with query sequence or profile DB (iteratively) through target sequence DB",
"Searches with the sequences or profiles query DB through the target sequence DB by running the prefilter tool and the align tool for Smith-Waterman alignment. For each query a results file with sequence matches is written as entry into a database of search results (alignmentDB).\nIn iterative profile search mode, the detected sequences satisfying user-specified criteria are aligned to the query MSA, and the resulting query profile is used for the next search iteration. Iterative profile searches are usually much more sensitive than (and at least as sensitive as) searches with single query sequences.",
"Martin Steinegger <[email protected]>",
"<i:queryDB> <i:targetDB> <o:alignmentDB> <tmpDir>",
CITATION_MMSEQS2},
Do not leave any compiler warnings in your code. Most of the time they might be false positives.
However, sometimes they hide real issues. Continous integration runs with -Werror and will fail when it finds any warnings.
Since, the CI system runs on many compilers and compiler versions the kinf of warnigs reported might differ between your local environment and the CI>
(Shellcheck)[https://www.shellcheck.net] runs on all workflow shell scripts and will fail in the continous integration if it finds any issues.
Make sure to not use Bash specific features. #!/bin/sh means that are POSIX shell compliant.
The MMseqs2 Windows builds run with the busybox ash shell, if you are a bit careful about your scripts, you will automatically gain Windows support.
The regression test runs most workflows such as search, profile search, profile-profile, target-profile, clustering, linclustm etc. after every commit. It compares their values against known good ones and fails if they don't match.
To run a search regression test execute the following steps:
# download the runner script and set permissions
$ wget https://bitbucket.org/martin_steinegger/mmseqs-benchmark/raw/master/scripts/run_codeship_pipeline.sh
$ chmod +x run_codeship_pipeline.sh
# change three variables in this file edit the following variables:
# If you dont have AVX2 on the machine just comment all lines containing MMSEQSAVX
BASE_DIR="$HOME/clone/regression_test”
MMSEQSSSE="$HOME/clone/build/src/mmseqs”
MMSEQSAVX="$HOME/clone/build_avx2/src/mmseqs"
# run script and set CI_COMMIT_ID to some non-empty string (in our CI system this is automatically set to the git commit).
$ CI_COMMIT_ID="TESTING" ./run_codeship_pipeline.sh
# The script will return an error code != 0 if there is a regression in sensitivity of MMseqs2. The error code can be checked with "echo $?".
$ [ $? -eq 1 ] && echo "Error"
It will print a report with sensitivity AUCs it achieved and then error out if it did not achieve the minimum AUCs. Currently 0.235 for normal sequence searches and 0.331 for profile searches.
You can also use our Docker images to run this benchmark:
cd mmseqs-folder
docker build -t mmseqs2 .
git clone https://bitbucket.org/martin_steinegger/mmseqs-benchmark.git
cd mmseqs-benchmark
docker build -t mmseqs-benchmark .
The regression test passed, if the second image exits cleanly. Please note, some users don't have permissions to access the unix socket to communicate with the Docker engine. In such a case, run the commands above as "sudo docker build ..."
The script that runs the regression test is found here:
https://bitbucket.org/martin_steinegger/mmseqs-benchmark/raw/master/scripts/run_codeship_pipeline.sh
There is also a docker image that runs the regression test:
https://bitbucket.org/martin_steinegger/mmseqs-benchmark/raw/master/Dockerfile
MMseqs2 can be built with ASan/MSan/UBSan/TSan support by specifying calling:
cmake -DHAVE_SANITIZER=1 -DCMAKE_BUILD_TYPE=ASan ...
Replace ASan with MSan, UBsan or TSan for the other sanitizers. CMake will error and abort if your compiler does not support the respective sanitizer.