Popularity

7.2

Stable

Activity

0.0

Declining

Stars 873

Watchers 19

Forks 66

Last Commit 21 days ago

Programming language: Go

License: MIT License

Tags: Data Structures

hyperloglog alternatives and similar packages

Based on the "Data Structures" category.
Alternatively, view hyperloglog alternatives based on common mentions on social networks and blogs.

gods

9.7 3.9 hyperloglog VS gods

GoDS (Go Data Structures) - Sets, Lists, Stacks, Maps, Trees, Queues, and much more
go-datastructures

9.5 6.6 hyperloglog VS go-datastructures

A collection of useful, performant, and threadsafe Go datastructures.

Collect and Analyze Billions of Data Points in Real Time

Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.

Promo www.influxdata.com

golang-set

8.7 0.0 hyperloglog VS golang-set

A simple, battle-tested and generic set type for the Go language. Trusted by Docker, 1Password, Ethereum and Hashicorp.
gota

8.6 0.0 hyperloglog VS gota

Gota: DataFrames and data wrangling in Go (Golang)
FSM for Go

8.5 0.0 hyperloglog VS FSM for Go

Finite State Machine for Go
willf/bloom

8.3 0.0 hyperloglog VS willf/bloom

Go package implementing Bloom filters
roaring

8.3 0.0 hyperloglog VS roaring

Roaring bitmaps in Go (golang)
gocache

8.2 0.0 hyperloglog VS gocache

☔️ A complete Go cache library that brings you multiple ways of managing your caches
boomfilters

7.9 0.0 hyperloglog VS boomfilters

Probabilistic data structures for processing continuous, unbounded streams.
bitset

7.7 0.0 hyperloglog VS bitset

Go package implementing bitsets
gostl

7.4 0.0 hyperloglog VS gostl

Data structure and algorithm library for go, designed to provide functions similar to C++ STL
cuckoofilter

7.4 0.0 hyperloglog VS cuckoofilter

Cuckoo Filter: Practically Better Than Bloom
algorithms

7.3 0.0 hyperloglog VS algorithms

CLRS study. Codes are written with golang.
trie

7.1 0.0 hyperloglog VS trie

Data structure and relevant algorithms for extremely fast prefix/fuzzy string searching.
merkletree

6.7 0.0 hyperloglog VS merkletree

A Merkle Tree implementation written in Go.
go-geoindex

6.6 0.0 hyperloglog VS go-geoindex

Go native library for fast point tracking and K-Nearest queries
ttlcache

6.5 6.4 hyperloglog VS ttlcache

An in-memory cache with item expiration and generics [Moved to: https://github.com/jellydator/ttlcache]
conjungo

6.2 0.0 hyperloglog VS conjungo

A small flexible merge library in go
mafsa

6.0 0.0 hyperloglog VS mafsa

MA-FSA implementation with Minimal Perfect Hashing
go-adaptive-radix-tree

6.0 0.0 hyperloglog VS go-adaptive-radix-tree

Adaptive Radix Trees implemented in Go
Bloomfilter

6.0 0.0 hyperloglog VS Bloomfilter

Face-meltingly fast, thread-safe, marshalable, unionable, probability- and optimal-size-calculating Bloom filter in go
hilbert

6.0 0.0 hyperloglog VS hilbert

Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves.
goconcurrentqueue

5.9 0.0 hyperloglog VS goconcurrentqueue

Go concurrent-safe, goroutine-safe, thread-safe queue
goskiplist

5.9 0.0 hyperloglog VS goskiplist

A skip list implementation in Go
levenshtein

5.8 0.0 hyperloglog VS levenshtein

Go implementation to calculate Levenshtein Distance.
cuckoo-filter

5.6 0.0 hyperloglog VS cuckoo-filter

Cuckoo Filter go implement, better than Bloom Filter, configurable and space optimized 布谷鸟过滤器的Go实现，优于布隆过滤器，可以定制化过滤器参数，并进行了空间优化
binpacker

5.5 0.0 hyperloglog VS binpacker

A binary stream packer and unpacker
bitmap

5.4 0.0 hyperloglog VS bitmap

Simple dense bitmap index in Go with binary operators
bit

5.1 0.0 hyperloglog VS bit

Bitset data structure
iter

5.0 0.0 hyperloglog VS iter

Go implementation of C++ STL iterators and algorithms.
bloom

4.9 0.0 hyperloglog VS bloom

Bloom filters implemented in Go.
encoding

4.7 0.0 hyperloglog VS encoding

Integer Compression Libraries for Go
ring

4.6 0.0 hyperloglog VS ring

Package ring provides a high performance and thread safe Go implementation of a bloom filter.
deque

4.6 0.0 hyperloglog VS deque

A highly optimized double-ended queue
remember-go

4.6 0.0 hyperloglog VS remember-go

Cache Slow Database Queries
skiplist

4.4 0.0 hyperloglog VS skiplist

skiplist for golang
go-rquad

4.4 0.0 hyperloglog VS go-rquad

:pushpin: State of the art point location and neighbour finding algorithms for region quadtrees, in Go
go-mcache

4.2 0.0 hyperloglog VS go-mcache

Fast in-memory key:value store/cache with TTL
memlog

4.0 0.0 hyperloglog VS memlog

A Kafka log inspired in-memory and append-only data structure
crunch

4.0 0.0 hyperloglog VS crunch

take bytes out of things easily ✨🍪
set

3.9 0.0 hyperloglog VS set

A simple Set data structure implementation in Go (Golang) using LinkedHashMap.
nan

3.7 0.0 hyperloglog VS nan

Zero allocation Nullable structures in one library with handy conversion functions, marshallers and unmarshallers
timedmap

3.5 0.0 hyperloglog VS timedmap

A thread safe map which has expiring key-value pairs.
goset

3.5 0.0 hyperloglog VS goset

Set is a useful collection but there is no built-in implementation in Go lang.
count-min-log

3.3 0.0 hyperloglog VS count-min-log

Go implementation of Count-Min-Log
cmap

3.2 0.0 hyperloglog VS cmap

a thread-safe concurrent map for go
hide

3.2 0.0 hyperloglog VS hide

ID type with marshalling to/from hash to prevent sending IDs to clients.
go-tuple

3.2 0.0 hyperloglog VS go-tuple

Go 1.18 generic tuples
pipeline

3.1 0.0 hyperloglog VS pipeline

Pipelines using goroutines
ptrie

3.1 0.0 hyperloglog VS ptrie

A prefix tree implementation in go

Do you think we are missing an alternative of hyperloglog or a related project?

Add another 'Data Structures' Package

Popular Comparisons

README

Hyperloglog Logo

An improved version of HyperLogLog for the count-distinct problem, approximating the number of distinct elements in a multiset using 33-50% less space than other usual HyperLogLog implementations.

This work is based on "Better with fewer bits: Improving the performance of cardinality estimation of large data streams - Qingjun Xiao, You Zhou, Shigang Chen".

Implementation

The core differences between this and other implementations are:

use metro hash instead of xxhash
sparse representation for lower cardinalities (like HyperLogLog++)
loglog-beta for dynamic bias correction medium and high cardinalities.
4-bit register instead of 5 (HLL) and 6 (HLL++), but most implementations use 1-byte registers out of convenience

In general it borrows a lot from InfluxData's fork of Clark Duvall's HyperLogLog++ implementation, but uses 50% less space.

Results

A direct comparison with the HyperLogLog++ implementation used by InfluxDB yielded the following results:

Exact	Axiom (8.2 KB)	Influx (16.39 KB)
10	10 (0.0% off)	10 (0.0% off)
50	50 (0.0% off)	50 (0.0% off)
250	250 (0.0% off)	250 (0.0% off)
1250	1249 (0.08% off)	1249 (0.08% off)
6250	6250 (0.0% off)	6250 (0.0% off)
31250	31008 (0.7744% off)	31565 (1.0080% off)
156250	156013 (0.1517% off)	156652 (0.2573% off)
781250	782364 (0.1426% off)	775988 (0.6735% off)
3906250	3869332 (0.9451% off)	3889909 (0.4183% off)
10000000	9952682 (0.4732% off)	9889556 (1.1044% off)

Note

A big thank you to Prof. Shigang Chen and his team at the University of Florida who are actively conducting research around "Big Network Data".

An Axiom production.

Do you enjoy solving problems like these? If so, get in touch with us at [email protected]!

hyperloglog

HyperLogLog with lots of sugar (Sparse, LogLog-Beta bias correction and TailCut space reduction) brought to you by Axiom