Data Cruncher

DataCruncher is a flexible tool for simulating and testing data ingestion by sending payloads to your security information and event management (SIEM) systems. The application is built with a focus on ease of use, real-time simulation of events, and high-performance data processing. This tool is co-developed with AI, helping to optimize your data ingestion tests for different payloads and use cases.

---

Features

• Dynamic Payload Generation: Automatically adjusts data payloads to match specified target size and events per second (EPS).
• Parallel Threading: Allows adjustment of parallel threads to optimize throughput and simulate high-volume event data efficiently.
• Customizable Endpoints: Supports the ability to choose between different endpoint types (/event and /raw), depending on your SIEM configuration.
• Real-Time Feedback: Provides real-time statistics such as thread saturation, estimated events to send, bytes to send, and estimated duration.
• Interactive UI: Simplified UI with sliders to control parameters like target size, EPS, and parallel threads, while providing a clean display of results and performance metrics.

---

Installation

To get started with DataCruncher, follow these steps:

1. Clone the repository

   git clone https://github.com/miladaslaner/DataCruncher.git
   cd DataCruncher

2. Open in Xcode

   • Open the project in Xcode by double-clicking the .xcodeproj file.
   • Make sure all dependencies are set up and resolved. The project should be ready for use.

3. Run the app

   • Select the desired target (e.g., DataCruncher or simulator) in Xcode.
   • Press Cmd + R to build and run the project.

---

Usage

Once you have the app running, you can use the following features to configure your simulation:

1. HEC Endpoint URL: Provide the URL of the HTTP Event Collector (HEC) where events will be sent.
2. API Token: Enter the token required for authentication with your SIEM system.
3. Endpoint Type: Choose between /event and /raw endpoints depending on your system’s configuration.
4. Target Size (MB): Define the size of the payload you wish to send.
5. Events Per Second (EPS): Adjust the rate of events to simulate in real-time.
6. Parallel Threads: Set the number of parallel threads to optimize performance.
7. Dynamic EPS Based on Payload Size: Automatically adjusts EPS based on the payload size for more accurate simulations.

---

Example Flow

1. Target Size: Adjust the slider to specify the desired payload size.
2. Events Per Second: Choose the rate at which you want to send events.
3. Parallel Threads: Set the number of threads to run in parallel for better performance.

After selecting these configurations, click Send to SIEM to start sending events. You will receive feedback on the estimated number of events sent, thread saturation, and other helpful stats.

---

Contributing

If you’d like to contribute to DataCruncher, feel free to fork the repository and create a pull request. Contributions in the form of bug fixes, new features, and improvements are always welcome.

Steps for contribution:

1. Fork the repository.
2. Clone your fork to your local machine.
3. Create a new branch for your changes.
4. Make the necessary modifications and test them.
5. Push your changes to your forked repository.
6. Submit a pull request to the main repository.

---

License

DataCruncher is open-source software licensed under the MIT License.