Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ADAM-1576] Allow translation between two different GenomicRDD types. #1598

Merged
merged 1 commit into from
Jul 17, 2017
Merged

[ADAM-1576] Allow translation between two different GenomicRDD types. #1598

merged 1 commit into from
Jul 17, 2017
+2,803 −21

Conversation

fnothaft
Copy link
Member

@fnothaft fnothaft commented Jul 11, 2017 •
edited
Loading

Resolves #1576. Adds transmute function, which is analogous to transform, but which allows any return type. This uses an implicit function a la pipe to build the output GenomicRDD.

Can I get a quick review pass on this? If this general direction looks good, I will flesh out the implicits for all the allowable transformations.

  • Extend to all primary types
  • Add docs

@fnothaft fnothaft added this to the 0.23.0 milestone Jul 11, 2017
@coveralls
Copy link

Coverage Status

Coverage decreased (-0.6%) to 83.492% when pulling 2b72fff on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

1 similar comment
@coveralls
Copy link

coveralls commented Jul 11, 2017 •
edited
Loading

Coverage Status

Coverage decreased (-0.6%) to 83.492% when pulling 2b72fff on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2203/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1598/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 6b5f30589ede34f153dbac174c5b234aff21bb64 # timeout=10Checking out Revision 6b5f30589ede34f153dbac174c5b234aff21bb64 (origin/pr/1598/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 6b5f30589ede34f153dbac174c5b234aff21bb64First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

devin-petersohn
devin-petersohn approved these changes Jul 11, 2017
View reviewed changes
Copy link
Member

@devin-petersohn devin-petersohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Looks great.

@fnothaft
Copy link
Member Author

OK great! I'll flesh this out today. Don't merge as is!

@fnothaft fnothaft force-pushed the issues/1576-generic-rdd-to-specific branch from 2b72fff to f27ee3d Compare July 11, 2017 23:39
@coveralls
Copy link

coveralls commented Jul 11, 2017 •
edited
Loading

Coverage Status

Coverage increased (+0.2%) to 84.29% when pulling f27ee3d on fnothaft:issues/1576-generic-rdd-to-specific into 324ae74 on bigdatagenomics:master.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2206/
Test PASSed.

@fnothaft fnothaft force-pushed the issues/1576-generic-rdd-to-specific branch from f27ee3d to e25d5a6 Compare July 12, 2017 06:17
@fnothaft
Copy link
Member Author

This is good to go from my end.

@coveralls
Copy link

coveralls commented Jul 12, 2017 •
edited
Loading

Coverage Status

Coverage increased (+0.5%) to 84.525% when pulling e25d5a6 on fnothaft:issues/1576-generic-rdd-to-specific into 238e044 on bigdatagenomics:master.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2207/
Test PASSed.

@fnothaft fnothaft force-pushed the issues/1576-generic-rdd-to-specific branch from e25d5a6 to 30a6119 Compare July 13, 2017 04:23
@fnothaft
Copy link
Member Author

fnothaft commented Jul 13, 2017 •
edited
Loading

Rebased and cleared conflicts. This is good to go from my side.

@coveralls
Copy link

coveralls commented Jul 13, 2017 •
edited
Loading

Coverage Status

Coverage increased (+0.4%) to 84.533% when pulling 30a6119 on fnothaft:issues/1576-generic-rdd-to-specific into 607cd50 on bigdatagenomics:master.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2215/
Test PASSed.

@fnothaft fnothaft mentioned this pull request Jul 13, 2017
Merged
heuermh
heuermh approved these changes Jul 17, 2017
View reviewed changes
Copy link
Member

@heuermh heuermh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty cool and fairly powerful. It also may lead to confusion, as it allows users to create GenomicRDDs with records that break preconditions and validation checks.

@fnothaft fnothaft force-pushed the issues/1576-generic-rdd-to-specific branch from 30a6119 to 751da13 Compare July 17, 2017 15:41
@fnothaft
Copy link
Member Author

Rebased and cleared conflicts.

@fnothaft
Copy link
Member Author

Also, WRT @heuermh's comment, there's nothing that stops an enterprising user from breaking our preconditions today. This might make it easier, but not really any easier than just using transform today is.

@coveralls
Copy link

coveralls commented Jul 17, 2017 •
edited
Loading

Coverage Status

Coverage increased (+0.4%) to 84.58% when pulling 751da13 on fnothaft:issues/1576-generic-rdd-to-specific into 406d1e3 on bigdatagenomics:master.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2227/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1598/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains c24fe77ef226e0041a2f34d4212526c893402e9c # timeout=10Checking out Revision c24fe77ef226e0041a2f34d4212526c893402e9c (origin/pr/1598/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f c24fe77ef226e0041a2f34d4212526c893402e9cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft
Copy link
Member Author

Looks like a transient s3 failure.

@fnothaft
Copy link
Member Author

Jenkins, test this please.

@coveralls
Copy link

coveralls commented Jul 17, 2017 •
edited
Loading

Coverage Status

Coverage increased (+0.4%) to 84.58% when pulling 751da13 on fnothaft:issues/1576-generic-rdd-to-specific into 406d1e3 on bigdatagenomics:master.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2230/
Test PASSed.

@heuermh heuermh merged commit 961f90a into bigdatagenomics:master Jul 17, 2017
@heuermh
Copy link
Member

heuermh commented Jul 17, 2017

Thank you, @fnothaft!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@heuermh heuermh heuermh approved these changes

@devin-petersohn devin-petersohn devin-petersohn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.23.0
Development

Successfully merging this pull request may close these issues.
5 participants
@fnothaft @coveralls @AmplabJenkins @heuermh @devin-petersohn