gotokenizer alternatives and similar packages

Based on the "Natural Language Processing" category.
Alternatively, view gotokenizer alternatives based on common mentions on social networks and blogs.

prose

8.7 2.2 gotokenizer VS prose

:book: A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.
gojieba

8.3 0.4 gotokenizer VS gojieba

"结巴"中文分词的Golang版本

Scout APM: A developer's best friend. Try free for 14-days

Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.

Sponsored scoutapm.com

go-i18n

8.3 4.6 gotokenizer VS go-i18n

Translate your Go program into multiple languages.
gse

8.2 7.3 gotokenizer VS gse

Go efficient text segmentation and NLP; support english, chinese, japanese and other. Go 语言高性能分词
when

7.7 0.0 gotokenizer VS when

A natural language date/time parser with pluggable rules
go-pinyin

7.7 0.0 gotokenizer VS go-pinyin

汉字转拼音
spaGO

7.5 9.7 gotokenizer VS spaGO

Self-contained Machine Learning and Natural Language Processing library in Go
kagome

6.9 8.2 gotokenizer VS kagome

Self-contained Japanese Morphological Analyzer written in pure Go
whatlanggo

6.8 0.0 gotokenizer VS whatlanggo

Natural language detection library for Go
nlp

6.4 0.0 gotokenizer VS nlp

[UNMANTEINED] Extract values from strings and fill your structs with nlp.
sentences

6.1 5.3 gotokenizer VS sentences

A multilingual command line sentence tokenizer in Golang
universal-translator

5.7 0.0 gotokenizer VS universal-translator

:speech_balloon: i18n Translator for Go/Golang using CLDR data + pluralization rules
locales

5.6 0.6 gotokenizer VS locales

:earth_americas: a set of locales generated from the CLDR Project which can be used independently or within an i18n package; these were built for use with, but not exclusive to https://github.com/go-playground/universal-translator
getlang

4.7 1.1 gotokenizer VS getlang

Natural language detection package in pure Go
RAKE.go

4.4 0.0 gotokenizer VS RAKE.go

A Go port of the Rapid Automatic Keyword Extraction algorithm (RAKE)
gounidecode

4.3 0.0 gotokenizer VS gounidecode

Unicode transliterator for #golang
go-nlp

4.3 0.0 gotokenizer VS go-nlp

Utilities for working with discrete probability distributions and other tools useful for doing NLP work.
go-unidecode

4.2 0.0 gotokenizer VS go-unidecode

ASCII transliterations of Unicode text.
segment

4.1 0.0 gotokenizer VS segment

A Go library for performing Unicode Text Segmentation as described in Unicode Standard Annex #29
go-stem

4.0 0.0 gotokenizer VS go-stem

Word Stemming in Go
MMSEGO

3.9 0.0 gotokenizer VS MMSEGO

Chinese word splitting algorithm MMSEG in GO
textcat

3.9 0.0 gotokenizer VS textcat

A Go package for n-gram based text categorization, with support for utf-8 and raw text
go2vec

3.3 0.0 gotokenizer VS go2vec

Read and use word2vec vectors in Go
stemmer

3.2 0.0 gotokenizer VS stemmer

Stemmer packages for Go programming language. Includes English, German and Dutch stemmers.
dpar

3.0 0.6 gotokenizer VS dpar

Neural network transition-based dependency parser (in Rust)
porter2

3.0 0.0 gotokenizer VS porter2

High Performance Porter2 Stemmer
go-mystem

2.7 0.0 gotokenizer VS go-mystem

CGo bindings to Yandex.Mystem
paicehusk

2.6 0.0 gotokenizer VS paicehusk

Golang implementation of the Paice/Husk Stemming Algorithm
petrovich

2.5 0.0 gotokenizer VS petrovich

Golang port of Petrovich - an inflector for Russian anthroponyms.
go-localize

2.5 1.9 gotokenizer VS go-localize

i18n (Internationalization and localization) engine written in Go, used for translating locale strings.
address

2.2 4.3 gotokenizer VS address

Address handling for Go.
snowball

2.2 0.0 L1 gotokenizer VS snowball

Cgo binding for Snowball C library
golibstemmer

2.1 0.0 gotokenizer VS golibstemmer

Go bindings for the snowball libstemmer library including porter 2
go-tinydate

2.0 0.0 gotokenizer VS go-tinydate

A tiny date object in Go. Tinydate uses only 4 bytes of memory
iuliia-go

2.0 0.0 gotokenizer VS iuliia-go

Transliterate Cyrillic → Latin in every possible way
libtextcat

1.9 0.0 gotokenizer VS libtextcat

Cgo binding for libtextcat C library
icu

1.9 0.0 gotokenizer VS icu

Cgo binding for icu4c library
govader

1.6 3.1 gotokenizer VS govader

vader sentiment analysis in go
shamoji

1.3 0.0 gotokenizer VS shamoji

The shamoji (杓文字) is a word filtering package
detectlanguage

1.2 0.0 gotokenizer VS detectlanguage

Detect Language API Go Client
porter

0.9 0.0 gotokenizer VS porter

porter stemmer
gosentiwordnet

0.8 4.0 gotokenizer VS gosentiwordnet

💬 Sentiment analyzer library using SentiWordnet in Go
go-eco

0.6 0.0 gotokenizer VS go-eco

Automatically exported from code.google.com/p/go-eco

* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest. Visit our partner's website for more details.

Do you think we are missing an alternative of gotokenizer or a related project?

Add another 'Natural Language Processing' Package

Popular Comparisons

README

gotokenizer

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

Motivation

I wanted a simple tokenizer that has no unnecessary overhead using the standard library only, following good practices and well tested code.

Features

Support Maximum Matching Method
Support Minimum Matching Method
Support Reverse Maximum Matching
Support Reverse Minimum Matching
Support Bidirectional Maximum Matching
Support Bidirectional Minimum Matching
Support using Stop Tokens
Support Custom word Filter

Installation

go get -u github.com/xujiajun/gotokenizer

Usage

package main

import (
    "fmt"

    "github.com/xujiajun/gotokenizer"
)

func main() {
    text := "gotokenizer是一款基于字典和Bigram模型纯go语言编写的分词器，支持6种分词算法。支持stopToken过滤和自定义word过滤功能。"

    dictPath := "/Users/xujiajun/go/src/github.com/xujiajun/gotokenizer/data/zh/dict.txt"
    // NewMaxMatch default wordFilter is NumAndLetterWordFilter
    mm := gotokenizer.NewMaxMatch(dictPath)
    // load dict
    mm.LoadDict()

    fmt.Println(mm.Get(text)) //[gotokenizer 是 一款 基于 字典 和 Bigram 模型 纯 go 语言 编写 的 分词器 ， 支持 6 种 分词 算法 。 支持 stopToken 过滤 和 自定义 word 过滤 功能 。] <nil>

    // enabled filter stop tokens 
    mm.EnabledFilterStopToken = true
    mm.StopTokens = gotokenizer.NewStopTokens()
    stopTokenDicPath := "/Users/xujiajun/go/src/github.com/xujiajun/gotokenizer/data/zh/stop_tokens.txt"
    mm.StopTokens.Load(stopTokenDicPath)

    fmt.Println(mm.Get(text)) //[gotokenizer 一款 字典 Bigram 模型 go 语言 编写 分词器 支持 6 种 分词 算法 支持 stopToken 过滤 自定义 word 过滤 功能] <nil>
    fmt.Println(mm.GetFrequency(text)) //map[6:1 种:1 算法:1 过滤:2 支持:2 Bigram:1 模型:1 编写:1 gotokenizer:1 go:1 分词器:1 分词:1 word:1 功能:1 一款:1 语言:1 stopToken:1 自定义:1 字典:1] <nil>

}