You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
This is very closely related to #6832, and if #6832 does everything properly we might not need this issue. Cast currently will fall back to the CPU if we are in any timezone that is not UTC. But not all cast operations actually use the timezone, despite Cast asking for it. we need to be sure that we have updated our code so that cast will only fall back to the CPU if a timezone is set, and the cast operation uses the time zone to get the right answer. i.e. cast short to int does not care at all about timezones. But cast timestamp to string does.
Is your feature request related to a problem? Please describe.
This is very closely related to #6832, and if #6832 does everything properly we might not need this issue. Cast currently will fall back to the CPU if we are in any timezone that is not UTC. But not all cast operations actually use the timezone, despite Cast asking for it. we need to be sure that we have updated our code so that cast will only fall back to the CPU if a timezone is set, and the cast operation uses the time zone to get the right answer. i.e. cast short to int does not care at all about timezones. But cast timestamp to string does.
https://github.com/apache/spark/blob/2acfc1dbca975a2a4a38124fe8ebe464aa1663a9/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L261-L284
is what Spark has to solve this problem not sure if we should reuse it or what.
The text was updated successfully, but these errors were encountered: