Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TEST] Add release notes for 0.254 #16165

Closed
wants to merge 1 commit into from
Closed

[TEST] Add release notes for 0.254 #16165

wants to merge 1 commit into from

Conversation

ajaygeorge
Copy link
Contributor

Missing Release Notes

Abhisek Gautam Saikia

Akhil Umesh Mehendale

Mayank Garg

  • d9eff9a Add support to introduce custom prerequisite waiting logic
  • 84ade2c Add new query state WAITING_FOR_PREREQUISITES

Tal Galili

Tim Meehan

  • 0d08545 Temporarily disable test
  • 14c859b Add resource manager profile to presto-tests and enable in CI
  • ec0b015 Fix TestDistributedQueryResource
  • 6833296 Fix TestDistributedClusterStatsResource

Vic Zhang

guhanjie

Extracted Release Notes

  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Add Presto iceberg connector #15836 (Author: Chunxu Tang): Add Presto iceberg connector
    • Add iceberg connector.
  • Do not allocate resources within test constructor #15937 (Author: v-jizhang): Do not allocate resources within test constructor
    • Do not allocate resources within test constructor for a cleaner code.
  • Fix Athena Glue table compatibility issue #15993 (Author: v-jizhang): Fix Athena Glue table compatibility issue
    • Allow accessing tables in Glue metastore that do not have a table type.
  • Add missing support for bucket sort order in Glue #16003 (Author: v-jizhang): Add missing support for bucket sort order in Glue
    • Add support for bucket sort order in Glue when creating or updating a table or partition.
  • Partition schema evolution for Parquet #16011 (Author: Jalpreet Singh Nanda (:imjalpreet)): Partition schema evolution for Parquet
    • Add support for partition schema evolution for parquet.
  • Partition schema evolution for Parquet #16011 (Author: Jalpreet Singh Nanda (:imjalpreet)): Partition schema evolution for Parquet
    • Add support for partition schema evolution for parquet.
  • Add support for MaxResults on Glue Hive Metastore #16012 (Author: v-jizhang): Add support for MaxResults on Glue Hive Metastore
    • Add support for MaxResults on Glue Hive Metastore.
  • Add support for Glue endpoint URL #16014 (Author: v-jizhang): Add support for Glue endpoint URL
    • Add support for Glue endpoint URL :doc:/connector/hive.
  • Add max error retry config option to glue client #16018 (Author: v-jizhang): Add max error retry config option to glue client
    • Add max error retry config option to glue client. Defaults to 10. :doc:/connector/hive.
  • Add intelligent tiering storage class #16028 (Author: v-jizhang): Add intelligent tiering storage class
    • Added intelligent tiering storage class. The S3 storage class to use when writing the data. STANDARD and INTELLIGENT_TIERING storage classes are supported. Default storage class is STANDARD :doc:/connector/hive.
  • Add documentation for Glue Catalog support in Hive #16058 (Author: v-jizhang): Add documentation for Glue Catalog support in Hive
    • Add documentation for Glue Catalog support in Hive. :doc:/connector/hive.
  • Combine spill strategies #16069 (Author: Rebecca Schlussel): Combine spill strategies
    • Remove spilling strategy PER_QUERY_MEMORY_LIMIT and instead add configuration property experimental.query-limit-spill-enabled and session property query_limit_spill_enabled. When this property is set to true, and the spill strategy is not PER_TASK_MEMORY_THRESHOLD, then we will spill whenever a query uses more than the per-node total memory limit in combined revocable and non-revocable memory, in addition to whenever the memory pool exceeds the spill threshold. This fixes an issue where using the PER_QUERY_MEMORY_LIMIT spilling strategy could prevent the oom killer from running when the memory pool was full. The issue is still present for the PER_TASK_MEMORY_THRESHOLD spilling strategy.
  • Combine spill strategies #16069 (Author: Rebecca Schlussel): Combine spill strategies
    • Remove spilling strategy PER_QUERY_MEMORY_LIMIT and instead add configuration property experimental.query-limit-spill-enabled and session property query_limit_spill_enabled. When this property is set to true, and the spill strategy is not PER_TASK_MEMORY_THRESHOLD, then we will spill whenever a query uses more than the per-node total memory limit in combined revocable and non-revocable memory, in addition to whenever the memory pool exceeds the spill threshold. This fixes an issue where using the PER_QUERY_MEMORY_LIMIT spilling strategy could prevent the oom killer from running when the memory pool was full. The issue is still present for the PER_TASK_MEMORY_THRESHOLD spilling strategy.
  • Combine spill strategies #16069 (Author: Rebecca Schlussel): Combine spill strategies
    • Remove spilling strategy PER_QUERY_MEMORY_LIMIT and instead add configuration property experimental.query-limit-spill-enabled and session property query_limit_spill_enabled. When this property is set to true, and the spill strategy is not PER_TASK_MEMORY_THRESHOLD, then we will spill whenever a query uses more than the per-node total memory limit in combined revocable and non-revocable memory, in addition to whenever the memory pool exceeds the spill threshold. This fixes an issue where using the PER_QUERY_MEMORY_LIMIT spilling strategy could prevent the oom killer from running when the memory pool was full. The issue is still present for the PER_TASK_MEMORY_THRESHOLD spilling strategy.
  • Combine spill strategies #16069 (Author: Rebecca Schlussel): Combine spill strategies
    • Remove spilling strategy PER_QUERY_MEMORY_LIMIT and instead add configuration property experimental.query-limit-spill-enabled and session property query_limit_spill_enabled. When this property is set to true, and the spill strategy is not PER_TASK_MEMORY_THRESHOLD, then we will spill whenever a query uses more than the per-node total memory limit in combined revocable and non-revocable memory, in addition to whenever the memory pool exceeds the spill threshold. This fixes an issue where using the PER_QUERY_MEMORY_LIMIT spilling strategy could prevent the oom killer from running when the memory pool was full. The issue is still present for the PER_TASK_MEMORY_THRESHOLD spilling strategy.
  • Support Unnest in fragment result caching #16071 (Author: Shixuan Fan): Support Unnest in fragment result caching
    • Add support to fragment result caching for unnest.
  • Add properties to update memory in TableFinishOperator #16095 (Author: Vic Zhang): Add properties to update memory in TableFinishOperator
    • Memory tracking in TableFinishOperator can be enabled by setting the table-finish-operator-memory-tracking-enabled configuration property to true.
  • Add support for partition cache verification #16113 (Author: Nikhil Collooru): Add support for partition cache verification
    • Add support for partition cache validation. This can be enabled by setting hive.partition-cache-validation-percentage configuration parameter.
  • Reduce memory utilization on the Driver during the commit phase #16120 (Author: Andrii Rosa): Reduce memory utilization on the Driver during the commit phase
    • Reduce commit memory footprint on the Driver.
  • Reduce memory utilization on the Driver during the commit phase #16120 (Author: Andrii Rosa): Reduce memory utilization on the Driver during the commit phase
    • Reduce commit memory footprint on the Driver.
  • Reduce memory utilization on the Driver during the commit phase #16120 (Author: Andrii Rosa): Reduce memory utilization on the Driver during the commit phase
    • Reduce commit memory footprint on the Driver.
  • Fix PlanRemoteProjections #16136 (Author: Rongrong Zhong): Fix PlanRemoteProjections
    • Fix a bug where queries that have both remote functions and a local function with only constant arguments could throw an IndexOutOfBoundException during planning. The bug was introduced in release 0.253 by :pr:16039.

All Commits

  • d9eff9a Add support to introduce custom prerequisite waiting logic (Mayank Garg)
  • 84ade2c Add new query state WAITING_FOR_PREREQUISITES (Mayank Garg)
  • 0d08545 Temporarily disable test (Tim Meehan)
  • 14c859b Add resource manager profile to presto-tests and enable in CI (Tim Meehan)
  • ec0b015 Fix TestDistributedQueryResource (Tim Meehan)
  • 6833296 Fix TestDistributedClusterStatsResource (Tim Meehan)
  • 8aa2961 Fix PlanRemoteProjections (Rongrong Zhong)
  • e2944c6 Disable flaky test AbstractTestQueries.testEmptyJoins (Vic Zhang)
  • 8d7d0a9 Add spilling threshold for Presto on Spark (Vic Zhang)
  • 5f11dd2 Include revocable memory for peak node total memory (Vic Zhang)
  • 5f692c0 Include revocable memory for memory limit error (Vic Zhang)
  • 4d17bb0 Support spilling for Presto on Spark (Vic Zhang)
  • 129aab3 Rename TestPrestoSparkAbstractTestAggregations (Vic Zhang)
  • 4428cf7 Fix the partition cache invalidation logic to be more efficient (Nikhil Collooru)
  • a6d5270 Fix ZeroRowFileCreator to use DataSink instead of OutputStream interface (Arjun Gupta)
  • 5eb36ae Add session property for temp storage buffer size (Vic Zhang)
  • 100c120 Disable flaky TestQueues.testQueuedQueryInteraction (Vic Zhang)
  • 4bd5dc5 Log sizes for pages received on the Driver (Andrii Rosa)
  • 2692f30 Release inmemory input pages incrementally (Andrii Rosa)
  • a31785b Support compression for PartitionUpdate in Hive connector (Andrii Rosa)
  • 81e45ab Fix MetastoreContext creation in presto-iceberg module (Nikhil Collooru)
  • baea41b Allow StorageOrcFileTailSource to read DWRF stripe cache data (Sergii Druzkin)
  • 9958e19 Make ORC read tail size configurable (Sergii Druzkin)
  • 6681e91 Fix Athena Glue table compatibility issue (v-jizhang)
  • e87999e Add missing support for bucket sort order in Glue (v-jizhang)
  • b8b0e52 Upgrade to 0.254 (Chunxu Tang)
  • 97953b1 Adjust the test change of AbstractTestDistributedQueries (Chunxu Tang)
  • 1769297 Include presto-iceberg in example config (Chunxu Tang)
  • 70fd449 Remove unnecessary comments (Chunxu Tang)
  • cea7117 Add MetastoreContext in the iceberg HiveTableOperations (Chunxu Tang)
  • 656f7af Adjust the usage of Hive Metastore with MetastoreContext (Chunxu Tang)
  • 6800304 Fix review comments (Chunxu Tang)
  • 8180ce3 Upgrade Presto to 0.252 (Chunxu Tang)
  • d749f3f Refactor HdfsFileIo names (Chunxu Tang)
  • b6ba913 Fix parquet version conflict (Chunxu Tang)
  • ddb67db Include iceberg connector in the root POM (Chunxu Tang)
  • 2d37a11 Set up query tests (Chunxu Tang)
  • f738ed3 Add iceberg connector (Chunxu Tang)
  • 28f558b upgrade parquet to 1.11.0 (Chunxu Tang)
  • a7a9812 Use username for comparison only when impersonation is enabled (Nikhil Collooru)
  • fca1f24 Add support for partition cache verification (Nikhil Collooru)
  • 8f4ef27 Add support for MaxResults on Glue Hive Metastore (v-jizhang)
  • 8be15cf Add a flag for HttpRemoteTask to avoid eager but unnecessary sendUpdate (guhanjie)
  • e9c4a30 Decode Parquet dictionary faster (Zhenxiao Luo)
  • c9571e1 Fix generics in declaration of ValueSet.copyOf (Zhenxiao Luo)
  • 6a4f0ab Benchmark Parquet dictionary to Domain conversion (Zhenxiao Luo)
  • b302af1 Add a test for BenchmarkSortedRangeSet (Zhenxiao Luo)
  • ca09cd3 Simplify Range consumption (Zhenxiao Luo)
  • d887f69 Add benchmark for SortedRangeSet's getOrderedRanges (Zhenxiao Luo)
  • 3faafe1 Benchmark SortedRangeSet methods (Zhenxiao Luo)
  • 494b7f5 Plumb DWRF stripe cache info from proto to model (Sergii Druzkin)
  • 54c3304 Add queryId to MetastoreContext (Nikhil Collooru)
  • c63a7c7 Use double quotes for all column names in SHOW CREATE TABLE (Abhisek Gautam Saikia)
  • 18f7e17 Use alias in case of predicate stitching (Rohit Jain)
  • 2e180ba Add support for partition schema evolution for Parquet (Jalpreet Singh Nanda (:imjalpreet))
  • 9e07362 Introduce TableToPartitionMapping (Jalpreet Singh Nanda (:imjalpreet))
  • 865753c Allow classes that inherit from AbstractTestQueryFramework to supply expectedQueryRunner. (Sergey Pershin)
  • d441316 Revert "Revert "Bring retry queries to the beginning of the queue"" (Mayank Garg)
  • 779afcc Add properties to update memory in TableFinishOperator (Vic Zhang)
  • 7f9e242 Catch UncheckedIOException while reading broadcast table from storage (Arjun Gupta)
  • aa59a8d Do not allocate resources within test constructor (v-jizhang)
  • f434ea1 Disable flaky test in TestJdbcClient (Rebecca Schlussel)
  • 970e67e Disable flaky test TestPrestoDriver.testQueryCancelByInterrupt (Rebecca Schlussel)
  • 44240cc Combine memory pool and query based spill strategies (Rebecca Schlussel)
  • eae5266 Remove periodic check from MemoryRevokingScheduler (Rebecca Schlussel)
  • dec3195 Support Unnest in fragment result caching (Shixuan Fan)
  • 39c66e3 Adding the Poisson distribution (Tal Galili)
  • 77989bf Add max error retry config option to glue client (v-jizhang)
  • 012e9bd Add support for Glue endpoint URL (v-jizhang)
  • 486f4e2 Define custom json serializer for DataSize and Duration (Abhisek Gautam Saikia)
  • 54bde4c Fix iterator in IndexedPriorityQueue (Tim Meehan)
  • 88a14ba Include queuing time while computing query completion deadline (Arjun Gupta)
  • cdc9dca Release note for 0.251.1 (Ke Wang)
  • 150fb77 Revert "Bring retry queries to the beginning of the queue" (Bhavani Hari)
  • 408c2a1 Add documentation for Glue Catalog support in Hive (v-jizhang)
  • 7e4fe3d Add intelligent tiering storage class (v-jizhang)
  • 3d78f13 Add cluster-wide endpoints for query and cluster information (Tim Meehan)
  • f6611cb Support Dwrf Sequence Ids in Writer (Akhil Umesh Mehendale)
  • 54a7ec7 Fix typo in connectors.rst (linjunhua)
  • 2999330 Short-circuit TupleDomain columnWiseUnion and intersect (Zhenxiao Luo)

@ajaygeorge ajaygeorge changed the title Add release notes for 0.254 [TEST] Add release notes for 0.254 May 26, 2021
@ajaygeorge ajaygeorge closed this May 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ajaygeorge