[SPARK-35116][SQL][TESTS] The generated data fits the precision of Da…

…yTimeIntervalType in spark ### What changes were proposed in this pull request? The precision of `java.time.Duration` is nanosecond, but when it is used as `DayTimeIntervalType` in Spark, it is microsecond. At present, the `DayTimeIntervalType` data generated in the implementation of `RandomDataGenerator` is accurate to nanosecond, which will cause the `DayTimeIntervalType` to be converted to long, and then back to `DayTimeIntervalType` to lose the accuracy, which will cause the test to fail. For example: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137390/testReport/org.apache.spark.sql.hive.execution/HashAggregationQueryWithControlledFallbackSuite/udaf_with_all_data_types/ ### Why are the changes needed? Improve `RandomDataGenerator` so that the generated data fits the precision of DayTimeIntervalType in spark. ### Does this PR introduce _any_ user-facing change? 'No'. Just change the test class. ### How was this patch tested? Jenkins test. Closes #32212 from beliefer/SPARK-35116. Authored-by: beliefer <[email protected]> Signed-off-by: Max Gekk <[email protected]>
apache · Apr 18, 2021 · 03191e8 · 03191e8
1 parent 7f6dee8
commit 03191e8
Show file tree

Hide file tree

Showing 2 changed files with 17 additions and 11 deletions.
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
@@ -20,6 +20,7 @@ package org.apache.spark.sql
 import java.math.MathContext
 import java.sql.{Date, Timestamp}
 import java.time.{Duration, Instant, LocalDate, LocalDateTime, Period, ZoneId}
+import java.time.temporal.ChronoUnit
 
 import scala.collection.mutable
 import scala.util.{Random, Try}
@@ -272,17 +273,8 @@ object RandomDataGenerator {
         val ns = rand.nextLong()
         new CalendarInterval(months, days, ns)
       })
-      case DayTimeIntervalType => Some(() => {
-        val maxSeconds = Duration.ofDays(106751991).getSeconds
-        val seconds = rand.nextLong() % maxSeconds
-        val nanoAdjustment = rand.nextLong() % 999999000
-        Duration.ofSeconds(seconds, nanoAdjustment)
-      })
-      case YearMonthIntervalType => Some(() => {
-        val years = rand.nextInt() % 178956970
-        val months = rand.nextInt() % 12
-        Period.of(years, months, 0)
-      })
+      case DayTimeIntervalType => Some(() => Duration.of(rand.nextLong(), ChronoUnit.MICROS))
+      case YearMonthIntervalType => Some(() => Period.ofMonths(rand.nextInt()).normalized())
       case DecimalType.Fixed(precision, scale) => Some(
         () => BigDecimal.apply(
           rand.nextLong() % math.pow(10, precision).toLong,

diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGeneratorSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGeneratorSuite.scala
@@ -143,4 +143,18 @@ class RandomDataGeneratorSuite extends SparkFunSuite with SQLHelper {
     assert(!Arrays.equals(array1, arrayExpected))
     assert(Arrays.equals(array2, arrayExpected))
   }
+
+  test("SPARK-35116: The generated data fits the precision of DayTimeIntervalType in spark") {
+    Seq(DayTimeIntervalType, YearMonthIntervalType).foreach { dt =>
+      for (seed <- 1 to 1000) {
+        val generator = RandomDataGenerator.forType(dt, false, new Random(seed)).get
+        val toCatalyst = CatalystTypeConverters.createToCatalystConverter(dt)
+        val toScala = CatalystTypeConverters.createToScalaConverter(dt)
+        val data = generator.apply()
+        val catalyst = toCatalyst(data)
+        val convertedBack = toScala(catalyst)
+        assert(data == convertedBack)
+      }
+    }
+  }
 }