Skip to content

Commit

Permalink
fix docstring
Browse files Browse the repository at this point in the history
  • Loading branch information
chenmoneygithub committed Oct 21, 2022
1 parent 205efa0 commit 883dcd6
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions keras_nlp/tokenizers/byte_pair_tokenizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -200,8 +200,9 @@ class BytePairTokenizer(tokenizer.Tokenizer):
>>> tokenizer = keras_nlp.tokenizers.BytePairTokenizer(
... vocab, merge, sequence_length=2)
>>> tokenizer(["butterfly", "butter"])
<tf.Tensor: shape=(2, 2), dtype=int32, numpy=array([[1, 2],[1, 0]],
dtype=int32)>
<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
array([[1, 2],
[1, 0]], dtype=int32)>
Use hosted vocabluary and merge list.
Expand Down

0 comments on commit 883dcd6

Please sign in to comment.