v2.1.0-rc0
Pre-release
Pre-release
Major Updates
- Added SplitMergeTokenizer.
- Add support for token offsets to BertTokenizer.
Minor Updates
- Give BertTokenizer ability to read in a vocab file directly.
- Migrate from std::string to tensorflow::tstring.
- Many build script improvements.
- Update ToDense layer with ragged support attribute.
Bug Fixes
- Update SentencePiece to inherit from TokenizerWithOffsets.
- Fix ICU data linking issue.