Skip to content

Commit

Permalink
updated readme
Browse files Browse the repository at this point in the history
  • Loading branch information
daidokoro committed Aug 13, 2024
1 parent c05bf13 commit 7448835
Showing 1 changed file with 51 additions and 8 deletions.
59 changes: 51 additions & 8 deletions processor/transformprocessor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -366,29 +366,72 @@ The `convert_exponential_hist_to_explicit_hist` function converts an Exponential

This function requires 2 arguments:

- `distribution` - This argument defines the convertion algorithm used to distribute the exponential datapoints into a new Explicit Histogram. There are 4 distribution options:
- `distribution` - This argument defines the distribution algorithm used to allocate the exponential histogram datapoints into a new Explicit Histogram. There are 4 options:
<br>
- __upper__ - This approach identifies the highest possible value of each exponential bucket (_the upper bound_) and uses it to distribute the datapoints by comparing the upper bound of each bucket with the ExplicitBounds provided. This approach works better for small/narrow exponential histograms where the difference between the upper bounds and lower bounds are small.

_For example, Given:_
1. count = 10
2. Boundaries: [5, 10, 15, 20, 25]
3. Upper Bound: 15
_Process:_
4. Start with zeros: [0, 0, 0, 0, 0]
5. Iterate the boundaries and compare $upper = 15$ with each boundary:
- $15>5$ (_skip_)
- $15>10$ (_skip_)
- $15<=15$ (allocate count to this boundary)
6. Allocate count: [0, 0, __10__, 0, 0]
7. Final Counts: [0, 0, __10__, 0, 0]
<br>
- __midpoint__ - This approach works in a similar way to the __upper__ approach, but instead of using the upper bound, it uses the midpoint of each exponential bucket. The midpoint is identified by calculationg the average of the upper and lower bounds. This approach also works better for small/narrow exponential histograms.
<br>
- __uniform__ - This approach distributes the datapoints for each bucket uniformly across the overlapping __ExplicitBounds__. This approach works better for large/wide exponential histograms.

>The __uniform__ and __random__ distribution algorithms both utilise the concept of intersecting boundaries.
Intersecting boundaries are any boundary in the `boundaries array` that falls between or on the lower and upper values of the Exponential Histogram bounderies.
_For Example:_ if you have an Exponential Histogram bucket with a lower bound of 10 and upper of 20, and your boundaries array is [5, 10, 15, 20, 25], the intersecting boundaries are 10, 15, and 20 because they lie within the range [10, 20].
<br>
- __uniform__ - This approach distributes the datapoints for each bucket uniformly across the intersecting __ExplicitBounds__. The alogrithm works as follows:

- If there are valid intersecting boundaries, the function evenly distributes the count across these boundaries.
- Calculate the count to be allocated to each boundary.
- If there is a remainder after dividing the count equally, it distributes the remainder by incrementing the count for some of the boundaries until the remainder is exhausted.

_For example Given:_
1. count = 10
2. Exponential Histogram Bounds: [10, 20]
3. Boundaries: [5, 10, 15, 20, 25]
4. Intersecting Boundaries: [10, 15, 20]
5. Number of Intersecting Boundaries: 3
6. Using the formula: $count/numOfIntersections=10/3=3r1$
_Uniform Allocation:_
7. Start with zeros: [0, 0, 0, 0, 0]
8. Allocate 3 to each: [0, 3, 3, 3, 0]
9. Distribute remainder $r$ 1: [0, 4, 3, 3, 0]
10. Final Counts: [0, 4, 3, 3, 0]
<br>
- __random__ - This approach distributes the datapoints for each bucket randomly across the overlapping __ExplicitBounds__. This approach works better for large/wide exponential histograms.
- __random__ - This approach distributes the datapoints for each bucket randomly across the intersecting __ExplicitBounds__. This approach works in a similar manner to the uniform distribution algorithm with the main difference being that points are distributed randomly instead of uniformly. This works as follows:
- If there are valid intersecting boundaries, calculate the proportion of the count that should be allocated to each boundary based on the overlap of the boundary with the provided range (lower to upper).
- For each boundary, a random fraction of the calculated proportion is allocated.
- Any remaining count (_due to rounding or random distribution_) is then distributed randomly among the intersecting boundaries.
- If the bucket range does not intersect with any boundaries, the entire count is assigned to the start boundary.
<br>
- `ExplicitBounds` represents the list of bucket boundaries for the new histogram. This argument is __required__ and __cannot be empty__.

__WARNING:__
__WARNINGS:__

- The process of converting an ExponentialHistogram to an Explicit Histogram is not perfect and may result in a loss of precision. It is important to define an appropriate set of bucket boundaries and identify the best distribution approach for your data in order to minimize this loss.

For example, selecting Boundaries that are too high or too low may result histogram buckets that are too wide or too narrow, respectively.

The process of converting an ExponentialHistogram to an Explicit Histogram is not perfect and may result in a loss of precision. It is important to define an appropriate set of bucket boundaries and identify the best distribution approach for your data in order to minimize this loss.
- __Negative Bucket Counts__ are not supported in Explicit Histograms, as such negative bucket count are ignored.

For example, selecting Boundaries that are too high or too low may result histogram buckets that are too wide or too narrow, respectively.
- __ZeroCounts__ are only allocated if the ExplicitBounds array contains a zero boundary. That is, if the Explicit Boundaries that you provide does not start with `0`, the function will not allocate any zero counts from the Exponential Histogram.

This function should only be used when Exponential Histograms are not suitable for the downstream consumers or if upstream metric sources are unable to generate Explicit Histograms.

Example:
__Example__:

- `convert_exponential_hist_to_explicit_hist("random", [10.0, 100.0, 1000.0, 10000.0])`
- `convert_exponential_hist_to_explicit_hist("random", [0.0, 10.0, 100.0, 1000.0, 10000.0])`

### scale_metric

Expand Down

0 comments on commit 7448835

Please sign in to comment.