gotokenizer v1.1.0 Release Notes
Release Date: 2018-10-17 // about 6 years ago-
- โ Add Support Custom word Filter (MM RMM BMM)
- โ Add Support filter stop tokens (MM RMM BMM)
- โ Add tests
- ๐ Fixed test error
- โก๏ธ Update ReadMe
Previous changes from v1.0.0
-
- ๐ Support Maximum Matching Method
- ๐ Support Minimum Matching Method
- ๐ Support Reverse Maximum Matching
- ๐ Support Reverse Minimum Matching
- ๐ Support Bidirectional Maximum Matching
- ๐ Support Bidirectional Minimum Matching
- ๐ Support using Stop Tokens