Read offloaded literals #2685
Read offloaded literals #2685
Conversation
Signed-off-by: Eduardo Apolinario <[email protected]>
eapolinario
requested review from
wild-endeavor,
kumare3,
pingsutw,
cosmicBboy,
samhita-alla,
thomasjpfan and
Future-Outlier
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2685 +/- ##
==========================================
- Coverage 66.44% 58.40% -8.05%
==========================================
Files 9 302 +293
Lines 453 24748 +24295
Branches 0 2867 +2867
==========================================
+ Hits 301 14453 +14152
- Misses 152 9777 +9625
- Partials 0 518 +518 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
| @@ -852,6 +852,52 @@ def from_flyte_idl(cls, pb2_object): | |||
| ) | |||
|
|
|||
|
|
|||
| class LiteralOffloadedMetadata(_common.FlyteIdlEntity): | |||
There was a problem hiding this comment.
Not your change, but a generator class for this could be useful
There was a problem hiding this comment.
do you mean like a utility function to help in tests?
Signed-off-by: Eduardo Apolinario <[email protected]>
| ) | ||
|
|
||
| # Write offloaded_lv as bytes to a temp file | ||
| with open(f"{tmp_path}/offloaded_proto.pb", "wb") as f: |
There was a problem hiding this comment.
I'm surprised using / works with windows. I usually go with:
| with open(f"{tmp_path}/offloaded_proto.pb", "wb") as f: | |
| with (tmp_path / "offloaded_proto.pb").open("wb") as f: |
There was a problem hiding this comment.
that's a good call. I'm going to fix this in a separate PR.
As for what happens on windows, my guess is that the / becomes part of the file name.
Tracking issue
Linking to the RFC: flyteorg/flyte#5103
Why are the changes needed?
As discussed in flyteorg/flyte#5103, flytekit needs to be able to read offloaded literals.
What changes were proposed in this pull request?
The type engine "unrolls" literals that contain offloaded literals prior to turning those into python values.
How was this patch tested?
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link