Skip to content

v2.1.0-rc0

Pre-release
Pre-release
Compare
Choose a tag to compare
@broken broken released this 17 Dec 22:13
· 4 commits to 2.1 since this release

Major Updates

  • Added SplitMergeTokenizer.
  • Add support for token offsets to BertTokenizer.

Minor Updates

  • Give BertTokenizer ability to read in a vocab file directly.
  • Migrate from std::string to tensorflow::tstring.
  • Many build script improvements.
  • Update ToDense layer with ragged support attribute.

Bug Fixes

  • Update SentencePiece to inherit from TokenizerWithOffsets.
  • Fix ICU data linking issue.