Skip to content

Commit

Permalink
Add comment
Browse files Browse the repository at this point in the history
Signed-off-by: Nghia Truong <[email protected]>
  • Loading branch information
ttnghia committed Sep 14, 2022
1 parent 1e7a24e commit 780fc96
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions integration_tests/src/main/python/orc_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,8 @@ def read_orc_df(data_path):
def read_orc_sql(data_path):
return lambda spark : spark.sql('select * from orc.`{}`'.format(data_path))

# ORC has issues reading timestamps where it is off by 1 second if the timestamp is before
# epoch in 1970 and the microsecond value is between 0 and 1000.
# See https://github.com/rapidsai/cudf/issues/11525.
# Once https://github.com/NVIDIA/spark-rapids/issues/131-140 are fixed
# we should remove this and use timestamp_gen.
def get_orc_timestamp_gen(nullable=True):
return TimestampGen(start=datetime(1590, 1, 1, tzinfo=timezone.utc), nullable=nullable)

Expand Down

0 comments on commit 780fc96

Please sign in to comment.