nlp alternatives and similar packages
Based on the "Natural Language Processing" category.
Alternatively, view nlp alternatives based on common mentions on social networks and blogs.
-
prose
A library for text processing that supports tokenization, part-of-speech tagging, named-entity extraction, and more. -
whatlanggo
A natural language detection package for Go. Supports 84 languages and 24 scripts (writing systems e.g. Latin, Cyrillic, etc). -
locales
๐ a set of locales generated from the CLDR Project which can be used independently or within an i18n package; these were built for use with, but not exclusive to https://github.com/go-playground/universal-translator -
go-nlp
Utilities for working with discrete probability distributions and other tools useful for doing NLP work. -
segment
A Go library for performing Unicode Text Segmentation as described in Unicode Standard Annex #29 -
snowball
Snowball stemmer port (cgo wrapper) for Go. Provides word stem extraction functionality Snowball native. -
icu
Cgo binding for icu4c C library detection and conversion functions. Guaranteed compatibility with version 50.1. -
gotokenizer
A tokenizer based on the dictionary and Bigram language models for Golang. (Now only support chinese segmentation) -
porter
This is a fairly straightforward port of Martin Porter's C implementation of the Porter stemming algorithm. -
go-eco
Similarity, dissimilarity and distance matrices; diversity, equitability and inequality measures; species richness estimators; coenocline models. -
detectlanguage
Language Detection API Go Client. Supports batch requests, short phrase or single word language detection.
Get performance insights in less than 4 minutes
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest. Visit our partner's website for more details.
Do you think we are missing an alternative of nlp or a related project?
Popular Comparisons
README
nlp
nlp
is a general purpose any-lang Natural Language Processor that parses the data inside a text and returns a filled model
Supported types
int int8 int16 int32 int64
uint uint8 uint16 uint32 uint64
float32 float64
string
time.Time
time.Duration
Installation
// go1.8+ is required
go get -u github.com/shixzie/nlp
Feel free to create PR's and open Issues :)
How it works
You will always begin by creating a NL type calling nlp.New(), the NL type is a Natural Language Processor that owns 3 funcs, RegisterModel(), Learn() and P().
RegisterModel(i interface{}, samples []string, ops ...ModelOption) error
RegisterModel takes 3 parameters, an empty struct, a set of samples and some options for the model.
The empty struct lets nlp know all possible values inside the text, for example:
type Song struct {
Name string // fields must be exported
Artist string
ReleasedAt time.Time
}
err := nl.RegisterModel(Song{}, someSamples, nlp.WithTimeFormat("2006"))
if err != nil {
panic(err)
}
// ...
tells nlp that inside the text may be a Song.Name, a Song.Artist and a Song.ReleasedAt.
The samples are the key part about nlp, not just because they set the limits between keywords but also because they will be used to choose which model use to handle an expression.
Samples must have a special syntax to set those limits and keywords.
songSamples := []string{
"play {Name} by {Artist}",
"play {Name} from {Artist}",
"play {Name}",
"from {Artist} play {Name}",
"play something from {ReleasedAt}",
}
In the example below, you can see we're reffering to the Name and Artist fields
of the Song
type declared above, both {Name}
and {Artist}
are our keywords
and yes! you guessed it! Everything between play
and by
will be treated as a
{Name}
, and everything that's after by
will be treated as an {Artist}
meaning
that play
and by
are our limits.
limits
โโโโโโโดโโโโโโ
โโดโโ โโดโ
play {Name} by {Artist}
โโโฌโโโ โโโโโฌโโโ
โโโโโโโโฌโโโโโโ
keywords
Any character can be a limit, a ,
for example can be used as a limit.
keywords as well as limits are CaseSensitive
so be sure to type them right.
Note that putting 2 keywords together will cause that only 1 or none of them will be detected
limits are important - Me :3
Learn() error
Learn maps all models samples to their respective models using the NaiveBayes
algorithm based on those samples. Learn()
also trains all registered models
so they're able to fit expressions in the future.
// must call after all models are registrated and before calling nl.P()
err := nl.Learn()
if err != nil {
panic(err)
}
// ...
Once the algorithm has finished learning, we're now ready to start Processing those texts.
Note that you must call NL.Learn() after all models are registrated and before calling NL.P()
P(expr string) interface{}
P first asks the trained algorithm which model should be used, once we get the right and already trained model, we just make it fit the expression.
Note that everything in the expression must be separated by a space or tab
When processing an expression, nlp searches for the limits inside that
expression and evaluates which sample fits better the expression, it doesn't
matter if the text has trash
. In this example:
limits
โโโโโโโดโโโโโโ
โโดโโ โโดโ
play {Name} by {Artist}
โโโฌโโโ โโโโโฌโโโ
โโโโโโโโฌโโโโโโ
keywords
we have 2 limits, play
and by
, it doesn't matter if we had an expression
hello sir can you pleeeeeease play King by Lauren Aquilina, since:
limits
trash โโโโโโดโโโโโ
โโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ โโดโโ โโดโ
hello sir can you pleeeeeease play King by Lauren Aquilina
โโฌโโ โโโโโโโฌโโโโโโโโ
{Name} {Artist}
โโโฌโโโ โโโโโฌโโโ
โโโโโโโโฌโโโโโโโโ
keywords
{Name}
would be replaced with King
,
{Artist}
would be replaced with Lauren Aquilina
,
trash
would be ignored as well as the limits play
and by
,
and then a pointer to a filled struct with the type used to register the model (Song
)
( Song.Name
being {Name}
and Song.Artist
beign {Artist}
)
will be returned.
Usage
type Song struct {
Name string
Artist string
ReleasedAt time.Time
}
songSamples := []string{
"play {Name} by {Artist}",
"play {Name} from {Artist}",
"play {Name}",
"from {Artist} play {Name}",
"play something from {ReleasedAt}",
}
nl := nlp.New()
err := nl.RegisterModel(Song{}, songSamples, nlp.WithTimeFormat("2006"))
if err != nil {
panic(err)
}
err = nl.Learn() // you must call Learn after all models are registered and before calling P
if err != nil {
panic(err)
}
// after learning you can call P the times you want
s := nl.P("hello sir can you pleeeeeease play King by Lauren Aquilina")
if song, ok := s.(*Song); ok {
fmt.Println("Success")
fmt.Printf("%#v\n", song)
} else {
fmt.Println("Failed")
}
// Prints
//
// Success
// &main.Song{Name: "King", Artist: "Lauren Aquilina"}