Milestone/4.0.1

debian/control

@@ -16,6 +16,7 @@ X-Python3-Version: >= 3.8
 Package: wordfence
 Architecture: all
 Depends: ${python3:Depends}, libpcre3
+Recommends: libhyperscan5
 Description: Command-line malware scanner powered by Wordfence
  Wordfence CLI is a multi-process malware scanner written in Python. It's
  designed to have low memory overhead while being able to utilize multiple

docker/build/entrypoint.sh

@@ -95,6 +95,8 @@ if [ "$PACKAGE_TYPE" = 'standalone' ]; then
     --hidden-import wordfence.cli.remediate.definition \
     --hidden-import wordfence.cli.countsites.countsites \
     --hidden-import wordfence.cli.countsites.definition \
+    --hidden-import wordfence.scanning.matching.pcre \
+    --hidden-import wordfence.scanning.matching.vectorscan \
     main.py
 
   # compress and copy to output volume

docs/README.md

@@ -18,6 +18,7 @@ Wordfence CLI is a high performance, multi-process, command-line malware scanner
 - **Malware Scanning**
 	- [Subcommand Configuration](malware-scan/Configuration.md)
 	- [Automatic Remediation](malware-scan/Remediation.md)
+	- [Faster Scanning with Vectorscan](malware-scan/Vectorscan.md)
 	- [Examples](malware-scan/Examples.md)
 		- [Scanning a single directory for malware](malware-scan/Examples.md#scanning-a-single-directory-for-malware)
 		- [Writing malware scan results to a CSV](malware-scan/Examples.md#writing-malware-scan-results-to-a-csv)

docs/malware-scan/Configuration.md

@@ -36,6 +36,7 @@ In order to store malware scan specific configuration in the INI file, you shoul
 - `--allow-io-errors`: Allow scanning to continue if IO errors are encountered. Files that cannot be read will be skipped and a warning will be logged.
 - `-z`, `--chunk-size`: Size of file chunks that will be scanned. Use a whole number followed by one of the following suffixes: b (byte), k (kibibyte), m (mebibyte). Defaults to 3m.
 - `-M`, `--scanned-content-limit`: The maximum amount of data to scan in each file. Content beyond this limit will not be scanned. Defaults to 50 mebibytes. Use a whole number followed by one of the following suffixes: b (byte), k (kibibyte), m (mebibyte).
+- `--match-engine`: The regex engine to use for malware scanning. Options: `pcre`, `vectorscan` (default: `pcre`)
 - `--match-all`: If set, all possible signatures will be checked against each scanned file. Otherwise, only the first matching signature will be reported
 - `--pcre-backtrack-limit`: The regex backtracking limit for signature evaluation
 - `--pcre-recursion-limit`: The regex recursion limit for signature evaluation

docs/malware-scan/README.md

@@ -2,6 +2,7 @@
 
 - [Subcommand Configuration](Configuration.md)
 - [Automatic Remediation of discovered malware](Remediation.md)
+- [Faster Scanning with Vectorscan](malware-scan/Vectorscan.md)
 - [Examples](Examples.md)
 	- [Scanning a directory for malware](Examples.md#scanning-a-directory-for-malware)
 	- [Running Wordfence CLI in a cron](Examples.md#running-wordfence-cli-in-a-cron)

docs/malware-scan/Vectorscan.md

@@ -0,0 +1,56 @@
+# Faster Scanning with Vectorscan
+
+The malware scan currently has 2 separate scan signature engines, [`libpcre`](https://www.pcre.org/) and [Vectorscan](https://github.com/VectorCamp/vectorscan). While both of these scan engines are highly performant, Vectorscan is software designed to address the specific use-case we have with malware scanning, where we need to run a large number of regular expressions against a stream of data. 
+
+## Benchmarks for Performance Increase using Vectorscan
+
+Benchmarking has indicated that Vectorscan can be more than 30 times as fast as the exact same scan on the exact same hardware using PCRE.
+
+|Files|Matches|Data|Workers|PCRE Time|Vectorscan Time|Improvement|Library|CPU|
+|-----|-------|----|-------|---------|---------------|-----------|-------|---|
+|12,006|11,998|605.4 MiB|1|388s|16s|~25x|Hyperscan 5.2.1|AMD Ryzen 7 1700|
+|3,499|1|54.7 MiB|4|42s|3s|14x|Hyperscan 5.2.1|AMD Ryzen 7 1700|
+|25905|25001|1.7 GiB|32|38s|4s|~10x|Hyperscan 5.4.0|AMD Ryzen 9 5950X|
+|25905|25001|1.7 GiB|8|98s|3s|~32x|Hyperscan 5.4.0|AMD Ryzen 9 5950X|
+|25905|25001|1.7 GiB|4|191s|6s|~32x|Hyperscan 5.4.0|AMD Ryzen 9 5950X|
+|25905|25001|1.7 GiB|1|710s|21s|~34x|Hyperscan 5.4.0|AMD Ryzen 9 5950X|
+
+*(The above benchmarks were conducted using a free Wordfence CLI license)*
+
+
+## Configuring CLI to use Vectorscan
+
+By default, CLI will use `libprce` for scanning. To configure CLI to use Vectorscan, you can use the following command-line argument:
+
+    wordfence malware-scan --match-engine=vectorscan
+
+This can also be set in the INI file:
+
+    [MALWARE-SCAN]
+    match_engine=vectorscan
+
+## Installing Vectorscan/Hyperscan
+
+On Debian-based distros, Vectorscan can be installed with `apt`:
+
+    $ sudo apt install libvectorscan5
+
+Vectorscan is a fork of Hyperscan and maintains a compatible API, so installing Hyperscan on Intel-based systems will work as well. 
+
+    $ sudo apt install libhyperscan5
+
+We do currently support both technologies, but this may change over time if Vectorscan's API diverges from Hyperscan's. We will officially support Vectorscan going forward.
+
+Additional installation instructions can be found [in the Vectorscan Wiki](https://github.com/VectorCamp/vectorscan/wiki/Installation-from-package).
+
+## Known issues
+
+### Segfaults on Ubuntu versions 22.04 and 24.04
+
+The Vectorscan package provided by Ubuntu versions 22.04 and 24.04 will produce segfaults when performing malware scanning against certain files on some ARM systems. We've identified that this is an issue specifically with the packaged version. Compiling Vectorscan from scratch does not produce the same segfaults. We've created a [ticket on LaunchPad](https://bugs.launchpad.net/ubuntu/+source/vectorscan/+bug/2064951) for this issue.
+
+### Negligible Performance Improvement over NFS
+
+CLI will oftentimes have to scan large filesystems of relatively small files. NFS is not setup particularly well to serve many small files. Vectorscan can scan these small files fast enough where there can be not much of discernible difference between using `libprce` and Vectorscan since the scanning processes are largely I/O bound while waiting on NFS to provide the file contents.
+
+We've observed in our own benchmarks that there is not a significant performance increase using Vectorscan over `libpcre` for filesystems over NFS shares.

wordfence/api/licensing.py

@@ -1,4 +1,4 @@
-from typing import Union
+from typing import Union, Optional
 
 from .exceptions import ApiException
 
@@ -36,8 +36,14 @@ def __init__(self):
 
 class LicenseSpecific:
 
-    def __init__(self, license: License):
+    def __init__(self, license: Optional[License]):
         self.license = license
 
     def is_compatible_with_license(self, license: License):
-        return self.license == license
+        return self.license is None or self.license == license
+
+    def assign_license(self, license: Optional[License]):
+        self.license = license
+
+    def clear_license(self):
+        self.assign_license(None)

wordfence/api/noc1.py

@@ -1,13 +1,17 @@
 import json
 import re
+import base64
 from typing import Callable, Optional
 
 from .noc_client import NocClient
 from .exceptions import ApiException
 from .licensing import License
 
-from ..intel.signatures import CommonString, Signature, SignatureSet
-from ..util.validation import DictionaryValidator, ListValidator, Validator
+from ..intel.signatures import CommonString, Signature, SignatureSet, \
+    PrecompiledSignatureSet, deserialize_precompiled_signature_set
+from ..util.validation import DictionaryValidator, ListValidator, Validator, \
+    OptionalValueValidator
+from ..util.platform import Platform
 
 NOC1_BASE_URL = 'https://noc1.wordfence.com/v2.27/'
 
@@ -137,6 +141,52 @@ def get_malware_signatures(self) -> SignatureSet:
                         ) from index_error
         return SignatureSet(common_strings, signatures, self.license)
 
+    def get_precompiled_patterns(
+                self,
+                platform: str,
+                library_version: str,
+                library_type: Optional[str] = None,
+                database_version: int = PrecompiledSignatureSet.VERSION
+            ) -> dict:
+        parameters = {
+                'platform': platform,
+                'library_version': library_version,
+                'database_version': database_version
+            }
+        if library_type is not None:
+            parameters['library_type'] = library_type
+        response = self.request('get_precompiled_patterns', parameters)
+        validator = DictionaryValidator({
+                'data': OptionalValueValidator(str)
+            })
+        self.validate_response(response, validator)
+        return response
+
+    def get_precompiled_malware_signatures(
+                self,
+                platform: Platform,
+                library_version: str,
+                library_type: Optional[str] = None,
+                database_version: int = PrecompiledSignatureSet.VERSION
+            ) -> Optional[PrecompiledSignatureSet]:
+        response = self.get_precompiled_patterns(
+                platform.key,
+                library_version,
+                library_type,
+                database_version
+            )
+        data = response['data']
+        if data is None:
+            return None
+        data = base64.b64decode(data)
+        signature_set = deserialize_precompiled_signature_set(data)
+        signature_set.assign_license(self.license)
+        if isinstance(signature_set, PrecompiledSignatureSet):
+            return signature_set
+        raise ApiException(
+                'Malformed signature set data received from Wordfence API'
+            )
+
     def ping_api_key(self) -> bool:
         return self.process_simple_request('ping_api_key')
 

wordfence/cli/cli.py

@@ -178,7 +178,7 @@ def invoke(self) -> int:
             return subcommand.invoke()
 
 
-def main():
+def invoke_cli():
     exception_handler = ExceptionHandler()
     try:
         cli = WordfenceCli(exception_handler)
@@ -189,6 +189,10 @@ def main():
         return 130
 
 
-if __name__ == '__main__':
-    exit_code = main()
+def main():
+    exit_code = invoke_cli()
     sys.exit(exit_code)
+
+
+if __name__ == '__main__':
+    main()

wordfence/cli/context.py

@@ -2,7 +2,8 @@
 from typing import Optional, Any, Callable, Set, Union
 
 from ..version import __version__, __version_name__
-from ..util import pcre
+from ..util import pcre, vectorscan
+from ..util.text import yes_no
 from ..api import noc1, intelligence
 from ..util.caching import Cache, CacheDirectory, RuntimeCache, \
         InvalidCachedValueException, CacheException
@@ -156,17 +157,35 @@ def get_mailer(self) -> Mailer:
             self._mailer = Mailer(self.config)
         return self._mailer
 
+    def has_pcre(self) -> bool:
+        return pcre.AVAILABLE
+
+    def has_vectorscan(self) -> bool:
+        return vectorscan.AVAILABLE
+
     def display_version(self) -> None:
         if __version_name__ is None:
             name_suffix = ''
         else:
             name_suffix = f' "{__version_name__}"'
         print(f"Wordfence CLI {__version__}{name_suffix}")
-        jit_support_text = 'Yes' if pcre.HAS_JIT_SUPPORT else 'No'
-        print(
-                f"PCRE Version: {pcre.VERSION} - "
-                f"JIT Supported: {jit_support_text}"
-            )
+        has_pcre = self.has_pcre()
+        pcre_support_text = yes_no(has_pcre)
+        if has_pcre:
+            jit_support_text = yes_no(pcre.HAS_JIT_SUPPORT)
+            pcre_support_text += (
+                    f" - PCRE Version: {pcre.VERSION}"
+                    f" (JIT Supported: {jit_support_text})"
+                )
+        print(f'PCRE Supported: {pcre_support_text}')
+        has_vectorscan = self.has_vectorscan()
+        vectorscan_support_text = yes_no(has_vectorscan)
+        if has_vectorscan:
+            vectorscan_support_text += (
+                    f' - Version: {vectorscan.VERSION} (API Version: '
+                    f'{vectorscan.API_VERSION})'
+                )
+        print(f'Vectorscan Supported: {vectorscan_support_text}')
 
     def has_terminal_output(self) -> bool:
         return has_terminal_output()

wordfence/cli/malwarescan/definition.py

@@ -2,6 +2,7 @@
         PCRE_DEFAULT_MATCH_LIMIT_RECURSION
 from wordfence.util.units import byte_length
 
+from ...scanning.matching import MatchEngine
 from ..subcommands import SubcommandDefinition, UsageExample
 from ..config.typing import ConfigDefinitions
 from .reporting import SCAN_REPORT_CONFIG_OPTIONS
@@ -146,6 +147,15 @@
             "value_type": byte_length
         }
     },
+    "match-engine": {
+        "description": "The regex engine to use for malware scanning.",
+        "context": "ALL",
+        "argument_type": "OPTION",
+        "default": MatchEngine.get_default_option(),
+        "meta": {
+            "valid_options": MatchEngine.get_options()
+        }
+    },
     "match-all": {
         "description": "If set, all possible signatures will be checked "
                        "against each scanned file. Otherwise, only the "
@@ -187,13 +197,79 @@
         "argument_type": "FLAG",
         "default": False
     },
+    "pre-compile": {
+        "description": "Pre-compile and cache the signature set without "
+                       "actually running a scan",
+        "context": "CLI",
+        "argument_type": "FLAG",
+        "default": False,
+        "category": "Signature Compilation"
+    },
+    "pre-compile-generic": {
+        "description": "Pre-compile and cache the signature set without "
+                       "any CPU-specific optimizations and without running "
+                       "a scan",
+        "context": "CLI",
+        "argument_type": "FLAG",
+        "default": False,
+        "category": "Signature Compilation"
+    },
+    "pattern-database-path": {
+        "description": "Use an alternate path for storage of the pattern "
+                       "database",
+        "context": "ALL",
+        "argument_type": "OPTION",
+        "meta": {
+            "accepts_file": True
+        },
+        "default": None,
+        "category": "Signature Compilation"
+    },
+    "compile-local": {
+        "description": "Always compile the signature set locally rather than "
+                       "attempting to download a pre-compiled set",
+        "context": "ALL",
+        "argument_type": "FLAG",
+        "default": False,
+        "category": "Signature Compilation"
+    },
+    "re-compile": {
+        "description": "Always re-compile the signature set rather than using "
+                       "a cached version (when compiling locally)",
+        "context": "CLI",
+        "argument_type": "FLAG",
+        "default": False,
+        "category": "Signature Compilation"
+    },
+    "profile": {
+        "description": "Profile scan performance",
+        "context": "CLI",
+        "argument_type": "FLAG",
+        "default": False,
+        "hidden": True
+    },
+    "profile-path": {
+        "description": "Path at which to save profiling results",
+        "context": "CLI",
+        "argument_type": "OPTION",
+        "default": None,
+        "hidden": True
+    },
+    "direct-io": {
+        "short_name": "D",
+        "description": "Use direct IO when opening files to avoid caching",
+        "context": "ALL",
+        "argument_type": "FLAG",
+        "default": False
+    }
 }
 
 
 cacheable_types = {
     'wordfence.intel.signatures.SignatureSet',
     'wordfence.intel.signatures.CommonString',
     'wordfence.intel.signatures.Signature',
+    'wordfence.intel.signatures.PrecompiledSignatureSet',
     'wordfence.api.licensing.License'
 }