[query/vds] Add LEN field to VDS #14675
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
patrick-schultz
requested changes
Sep 11, 2024
cjllanwarne
requested changes
Sep 17, 2024
| .or_error( | ||
| hl.str( | ||
| 'cannot create VDS from merged representation -' ' found END field with non-reference genotype at ' |
There was a problem hiding this comment.
TIL python lets you concatenate strings by just putting them next to each other like this 🤯
cjllanwarne
approved these changes
Sep 17, 2024
chrisvittal
added a commit
to chrisvittal/hail
that referenced
this pull request
Sep 18, 2024
After split_multi, LGT is dropped from the variant data of a VDS. After PR hail-is#14560, LGT is added to datasets after creation via the combiner. After hail-is#14675 the same is true for `from_merged_representation`. We should keep the GT/LGT field consistent across ref and var data. This change does so for split_multi. Resolves hail-is#14694
chrisvittal
added a commit
to chrisvittal/hail
that referenced
this pull request
Sep 19, 2024
After split_multi, LGT is dropped from the variant data of a VDS. After PR hail-is#14560, LGT is added to datasets after creation via the combiner. After hail-is#14675 the same is true for `from_merged_representation`. We should keep the GT/LGT field consistent across ref and var data. This change does so for split_multi. Resolves hail-is#14694
hail-ci-robot
pushed a commit
that referenced
this pull request
Sep 19, 2024
After split_multi, LGT is dropped from the variant data of a VDS. After PR #14560, LGT is added to datasets after creation via the combiner. After #14675 the same is true for `from_merged_representation`. We should keep the GT/LGT field consistent across ref and var data. This change does so for split_multi. Resolves #14694
chrisvittal
force-pushed
the
vds/len-end-translation
branch
from
9bce5c9 to
d7d31cb
Compare
We can't do this anymore since genotype may be something other than diploid. Missed this in the original VDS ploidy changes
This is the beginning of a series of changes to support export of VDS to VCF 4.5, the version of VCF that contains the standardized form of our work that culminated in SVCR/VDS. Reference blocks were standardized with a LEN rather than an END. So, now, by default, add LEN to all VDS reads and drop END in favor of LEN on all VDS writes. Our optimizer will be able to take care of pruning away the dead field. We make sure that all VDS creation (other than the combiner), such as read_vds and from_merged_representation, contains both LEN and END preserving user code that depends on the presence of the END field.
Co-authored-by: Patrick Schultz <[email protected]>
chrisvittal
force-pushed
the
vds/len-end-translation
branch
from
473e90f to
f39364c
Compare
chrisvittal
dismissed
patrick-schultz’s stale review
dismissing this since I'd like another look after working around one of the core issues discoverd here
chrisvittal
force-pushed
the
vds/len-end-translation
branch
from
a8961c7 to
56b6c23
Compare
patrick-schultz
approved these changes
Oct 1, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the beginning of a series of changes to support export of VDS to VCF 4.5, the version of VCF that contains the standardized form of our work that culminated in SVCR/VDS.
Reference blocks were standardized with a LEN rather than an END. So, now, by default, add LEN to all VDS reads and drop END in favor of LEN on all VDS writes. Our optimizer will be able to take care of pruning away the dead field in pipelines that don't use it.
We make sure that all VDS creation (other than the combiner), such as read_vds and from_merged_representation, contains both LEN and END preserving user code that depends on the presence of the END field.
Furthermore, this change contains necessary combiner updates to prefer LEN over END, and to use LEN in the combiner itself.