Apache Release: DataSketches-Java 7.0.1 (for Java 17)

What's Changed

• This is a release to fix a bug found in the Theta Sketch compression algorithm in release 7.0.0.
• Otherwise, please refer to the release notes for DataSketches-Java 7.0.0.

Apache Release DataSketches-java 6.2.0 (for Java 8,11)

This release:

• This is a retrograde release to fix a bug that was found in the Theta compression algorithm.
• Includes new methods: getCDF(...) and getPMF(...) for the Tdigest sketch.
• Includes other minor improvements.

Full Changelog: 6.1.1...6.2.0

Apache Release DataSketches-Java 7.0.0 (for Java 17)

What's Changed

• This release runs on Java 17.

• This release leverages the new Java Foreign Function & Memory (FFM) API capability (JEP 412), which is in incubation in Java 17. This means that Java 17 is required to compile this code and at runtime. The location of the FFM code was moved from jdk.incubator.foreign in Java 17 to its position for preview in Java 21 at java.base/java.lang.foreign. As a result, code compiled with Java 17 FFM will not run with Java 21, the next LTS. Java does not provide backward compatibility for older incubation code.

• There are some changes in the DataSketches-Java API, due to the switch to leveraging the new FFM code, especially for methods that must deal with off-heap memory. These changes are a result of the DataSketches-Java-7.0.0 dependency on DataSketches-Memory-4.1.0, which manages all the direct memory.

For example, to allocate direct (off-heap) memory:

DataSketches-Java 6.1.1 using DataSketches-Memory 3.0.2 (Java 8, 11)

    try (WritableMemory wmem = WritableMemory.allocateDirect(4096)) {
      ...
    } //wmem is closed

where WritableMemory is a class of DataSketches-Memory.

In DataSketches-Java 7.0.0 using DataSketches-Memory 4.1.0 (Java 17)

    try (ResourceScope scope = (wmem = WritableMemory.allocateDirect(4096).scope()) {
      ...
    } //wmem is closed

where ResourceScope is an FFM class.

---

Full Changelog: 6.0.0...7.0.0

Apache Release DataSketches Java 6.1.1

Update dependency DataSketches-Memory version to 3.0.2

See DataSketches-Memory 3.0.2

Apache Release 6.1.0

What's Changed since 6.0.0

• Added new KLL Longs Sketch
• Optimized HLL Union for merge HLL to HLL
• Change Memory dependency to version 3.0.1
• Added one exclusion to FindBugsExcludeFilter.xml
• Fixed bug in Memory that was preventing limited runtime operation with Java 17, see Memory PR#209. (Java 17 is not formally supported by Memory. See datasketches-memory README).

New Contributors

• @ZacBlanco made their first contribution in #556
• @Claudenw made their first contribution in #574

Full Changelog: 6.0.0...6.1.0

6.1.0-RC1

Release Process: change POM to 6.1.0

Apache Release 6.0.0

• New: quantiles T-Digest sketch
• New: BloomFilter
• New: Exact and Bounded Sampling Proportional to Size (EB-PPS) sketch
• Added Weighted Inputs to quantiles KllFloatsSketch, KllDoublesSketch and KllItemsSketch
• Added Vector Inputs to quantiles KLLFloatsSketch and KllDoublesSketch
• Enhanced quantiles Sorted Views for KLL and Classic quantiles sketches.
• Enhanced Partitioning API.

5.0.2

This is a PATCH release. No new functionality has been introduced. There are a number of changes stemming from two issues:

• Issue 527: Properly use the comparator for sorting level 0 in the KllItemsSketch
• A new version of SpotBugs created a number of potential security warnings around Finalizer Attacks. Having done our best to look into the matter, we do not believe sketches are meaningfully vulnerable -- any data in the sketches is already available via reflection and there are no methods with special conditional access. Regardless, we felt that good code hygiene meant that we should prioritize fixing any issues found.

Apache Release 5.0.1

5.01 fixed two issues:

PR 482: The HLL Union :: toString(), which prints out a simple diagnostic summary of the sketch, might change the internal state of the union. This was not intended.
PR 485: The KllItemsSketch<Boolean> was not serializing and deserializing the min and max values properly. It only affects this specific generic case of <Boolean>. This is a rather bizarre use case for quantiles -- but nonetheless it is fixed! :)

Apache Release 5.0.0

• A new Example Partitioner Tool is useable in its own right for partitioning medium sized data sets up to about 1E9 items. But the same algorithm could be used in a parallel environment for partitioning data sets many orders-of-magnitude larger.
• Lots of internal cleanup and a few API improvements for consistency across the different quantile sketches, for example. These changes in the API, although relatively minor, were the reason to move to a major release.
• Fixed an integer overflow bug caught by Karan Kumar (via Druid), where very large partitioning datasets using the classic quantiles DoublesSketch::getPartitionBoundaries() would fail.