Implement Log Sampling for Performance Optimization

[monilpat]

Is your feature request related to a problem? Please describe.

The current logging system captures all log entries, which can lead to performance degradation and excessive storage usage, especially in high-traffic environments.

Describe the solution you'd like

Introduce log sampling to reduce the volume of logs generated. This should involve:

• Implementing a configurable log sampling rate that can be adjusted at runtime.
• Ensuring that critical logs (e.g., errors and warnings) are always captured, while informational and debug logs are sampled based on the configured rate.
• Providing an interface or API to adjust the sampling rate dynamically.

Code Example

// Example of setting log sampling rate dynamically
logger.setSamplingRate(0.1); // Sample 10% of informational and debug logs

Describe alternatives you've considered

• Capturing all logs and filtering them post-processing, which increases complexity and storage costs.
• Using external log management tools to handle log volume, which might not integrate well with the existing system.

Additional context

Implementing log sampling will improve system performance and reduce storage costs by controlling the volume of logs generated, while still capturing critical information needed for debugging and observability.

Related Issues

• Issue #144
• Issue #143