Quick Start Guide

LogBoost is intended for the enrichment of logs - it contains the capability to parse certain types of structured logging to CSV as well as handling arbitrary text-based files that may or may not have an existing structure.

LogBoost always outputs to CSV format - the CSV will contain all detected 'keys' as columns if parsing a known-structured format as well as extra keys representing the per-line enrichment outputs for that specific record (row).

For all examples below, I assume that LogBoost.exe is located in the current working directory. I will also assume that the user has acquired a free MaxMind API key and has placed that key inside a file named mm_api.txt.

A free MaxMind API key for the GeoLite2 Databases can be acquired at https://www.maxmind.com/en/geolite2/signup?utm_source=kb&utm_medium=kb-link&utm_campaign=kb-create-account . This key must be provided either at the commandline (-api), in a file with LogBoost.exe (mm_api.txt) or in an environment variable (MM_API) in the format $ACCOUNTID:$APIKEY, such as '-api "222111:6ij3x2_GRChRSGRAWeHuFbu4W136UDGdrLeV_sse"' or storing "222111:6ij3x2_GRChRSGRAWeHuFbu4W136UDGdrLeV_sse" as the first line of a file named mm_api.txt.

If you already possess the MaxMind Database files for Country, City and ASN, ensure they are stored in the same directory as LogBoost.exe as GeoLite2-Country.mmdb, GeoLite2-City.mmdb and GeoLite2-ASN.mmdb.

Alternatively, if the files are stored in a separate but accessible location, this directory can be provided to LogBoost via the '-dbdir' flag. For example, if the databases are located at C:\MaxMindDBs but LogBoost is located in another directory, use '-dbdir "C:\MaxMindDBs"'.

Download the latest version of LogBoost.exe and feed_config.json from https://github.com/joeavanzato/LogBoost/releases

Enriching a CSV File containing an IP Address Column

As a core example, lets take a CSV where a known column contains an IP address ("IPAddress" for the example) that we wish to gain additional context on within a directory such as "logs". For this example, lets enrich with both the MaxMind data (Country, City, ASN) as well as DNS.

LogBoost.exe -logdir "logs" -ipcol "IPAddress" -dns

This command will perform the following actions:

• Look for any .csv files within the directory "logs"
• Look for MaxMind API key in environment variable first, then inside mm_api.txt within the current working directory
• Download MaxMind Database to CWD if they do not exist using the discovered API key
• Iterate through all discovered CSVs looking for column "IPAddress"
  • For each record, the IPAddress column is inspected - if it is determined to be a valid IP address, that value is enriched using both the downloaded MaxMind Databases as well as a live DNS reverse lookup.
• The enriched output will be stored using the original filename but inside a directory named "output".

If we are unsure of which column an IP address will be stored in or if the column isn't a 'clean' IP address value, we can use regex to identify the first non-private IP address in each record like below;

LogBoost.exe -logdir "logs" -regex -dns

Instead of telling LogBoost to pull from a specific column, now it will attempt to use IPv4/IPv6 regex patterns to extract the first public IP address from each row and that value will be used for enrichment.

What if we have multiple files to enrich?

Not a problem - just stick them all in one or more subdirectories (if required - the output directories will recreate the input directory tree) and LogBoost will find all CSVs recursively in the provided input directory.

LogBoost also features a '-combine' flag which can be used to combine outputs that are within the same input directory - this is useful if processing a directory containing many similar files (Exchange IIS logging, Linux var/log/auth.*, etc). Simply add '-combine' to the input command and LogBoost will attempt to combine all outputs in each output directory.

LogBoost.exe -logdir "logs" -regex -dns -combine

DNS information for each IP address is stored in a local cache (dns.cache directory) - this is stored indefinitely up to a 1 GB cache size until over-writing begins - this cache is used for all future enrichment purposes. If you wish to re-query existing values, just delete 'dns.cache' directory.

Incorporating Open-Source Indicator Intelligence

LogBoost is capable of enriching IP addresses with text-based indicator feeds - feed_config.json contains the relevant feeds as well as the tag that will be used when hits are detected in source files. To build the Threat Intelligence Database for the first time, use the following:

LogBoost.exe -buildti

This command will initialize the database and perform an initial feed download and ingestion to the newly-created database.

Anytime you wish to update the feeds, use the following command:

LogBoost.exe -updateti

This will perform a download and ingestion of any new indicators - indicators are not over-written and can only be stored once.

Now that we have an updated indicator database, we can use this for enrichment by adding the '-useti' flag to any command, such as below;

LogBoost.exe -logdir "logs" -regex -dns -combine -useti

Now we have covered the following topics:

• Enriching a CSV file containing an IP Address (either in a known column or dynamically)
• Enriching with Geolocation/ASN/DNS/Threat Indicators
• Enriching multiple files simultaneously
• Building, Updating and Utilizing the Threat Indicator Database during enrichment