BPE tokenizer #389
Merged
BPE tokenizer #389
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks! Will take a look! One note, it might be nice to add Jesse as a co-author on the commit, he did some incredible work on this and we should make sure to credit it. |
|
@mattdangerw Definitely! Will add in the next commit. |
mattdangerw
requested changes
Oct 17, 2022
mattdangerw
reviewed
Oct 18, 2022
mattdangerw
requested changes
Oct 19, 2022
chenmoneygithub
force-pushed
the
bpe-tokenizer
branch
from
23d6eb0 to
205efa0
Compare
mattdangerw
approved these changes
Oct 21, 2022
There was a problem hiding this comment.
This looks good to me! Left some docstring comments to fix up. Still plenty of follow ups to do, but I think landing and iterating is a good plan.
Let's make sure to fix the co-author so github properly registers this.
Let's also open some follow up issues once this is in:
- Any outstanding places we know our output differs should for sure be tracked in an issue.
- An issue for removing the
tf.functionwe shouldn't force compile when running eagerly. - An issue for adding some testing that does not require network (and works off small simplified vocabs).
Thanks!
Add more test cases. Co-authored-by: jessechancy <[email protected]> add merge file Make cache a tf module Delete testdata address comments address comments fix docstring fix docstring
chenmoneygithub
force-pushed
the
bpe-tokenizer
branch
from
883dcd6 to
2cc9977
Compare
mattdangerw
pushed a commit
to mattdangerw/keras-hub
that referenced
this pull request
Nov 10, 2022
Add more test cases. Co-authored-by: jessechancy <[email protected]> add merge file Make cache a tf module Delete testdata address comments address comments fix docstring fix docstring
mattdangerw
pushed a commit
that referenced
this pull request
Nov 10, 2022
Add more test cases. Co-authored-by: jessechancy <[email protected]> add merge file Make cache a tf module Delete testdata address comments address comments fix docstring fix docstring
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a rework on #303.
Recreate the PR instead of direct editing for clear remote-local tracking.