slowpoke

Low-level key/value store in pure Go.

Programming language: Go

License: MIT License

Tags: Database    

Latest version: v2.1.0

README

Description

Package slowpoke is a simple key/value store written using Go's standard library only. Keys are stored in memory (with persistence), values stored on disk.

Description on russian: https://habr.com/post/354224/

Motivation

Replace Bolt with a simpler and more efficient engine.

Slowpoke (from version 2.0) based on pudge

How it works

Keys are stored in memory with persistence to disk. Values stored on disk only.

Slowpoke is parallel

Server

GRPC Server example: okdb

Complex examples

typegram

zen platform for authors and their subscribers with a minimalistic design and user-friendly interface.

golang-gin-realworld-example-app

This codebase was created to demonstrate a fully fledged fullstack application built with Golang/Gin/Slowpoke including CRUD operations, authentication, routing, pagination, and more.

Basic example

package main

import (
    "fmt"

    "github.com/recoilme/slowpoke"
)

func main() {
    // create database
    file := "test/example.db"
    // close all opened database
    defer slowpoke.CloseAll()
    // init key/val
    key := []byte("foo")
    val := []byte("bar")
    //store
    slowpoke.Set(file, key, val)
    // get
    res, _ := slowpoke.Get(file, key)
    //result
    fmt.Println(string(res))
}

Lazy example

func TestGob(t *testing.T) {
    file := "test/gob.db"
    DeleteFile(file)
    defer CloseAll()
    type Post struct {
        Id       int
        Content  string
        Category string
    }

    for i := 0; i < 20; i++ {
        post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i)}
        err := SetGob(file, post.Id, post)
        ch(err, t)
    }

    for i := 0; i < 20; i++ {
        var post = new(Post)
        err := GetGob(file, i, post)
        ch(err, t)
        fmt.Println("i:", i, "Post:", post)
    }
}

Advanced example

type Post struct {
    Id       int
    Content  string
    Category string
}

func main() {
    posts := "test/posts"
    tags := "test/tags"
    var pairs [][]byte
    for i := 0; i < 40; i++ {
        id := make([]byte, 4)
        binary.BigEndian.PutUint32(id, uint32(i))
        post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i), Category: "Category:" + strconv.Itoa(i/10)}
        b, _ := json.Marshal(post)
        pairs = append(pairs, id)
        pairs = append(pairs, b)
        tag := fmt.Sprintf("%s:%08d", strconv.Itoa(i/10), i)
        //store only tags keys
        slowpoke.Set(tags, []byte(tag), nil)
    }
    //store posts fast
    slowpoke.Sets(posts, pairs)

    //get last 2 post key with offset 2
    limit := uint32(2)
    offset := uint32(2)
    order := false //desc
    keys, _ := slowpoke.Keys(posts, nil, limit, offset, order)
    fmt.Println(keys) //[[0 0 0 37] [0 0 0 36]]

    //get key/ values
    res := slowpoke.Gets(posts, keys)
    for k, v := range res {
        if k%2 == 0 {
            fmt.Print(binary.BigEndian.Uint32(v))
        } else {
            var p Post
            json.Unmarshal(v, &p)
            fmt.Println(p)
        }
    }
    //37{37 Content:37 Category:3}
    //36{36 Content:36 Category:3}

    //free from memory
    slowpoke.Close(posts)
    slowpoke.Close(tags)

    //open Db and read tags by prefix 2:* in ascending order
    tagsKeys, _ := slowpoke.Keys(tags, []byte("2:*"), 0, 0, true)
    for _, v := range tagsKeys {
        fmt.Print(string(v) + ", ")
    }
    //2:00000020, 2:00000021, 2:00000022, 2:00000023, 2:00000024, 2:00000025, 2:00000026, 2:00000027, 2:00000028, 2:00000029,
}

Api

All methods are thread-safe.

• Set/Sets/SetGob

Store val and key. If the file does not exist it will be created.

• Get/Gets/GetGob

Return the value for the given key or nil and an error. Get will open the database if necessary.

• Keys

Return keys in ascending/descending order.

With limit and offset.

If from is not nil, return keys lexicographically greater than the from value.

If from ends with asterix *, return keys with the prefix equal to from without the asterix.

Documentation

Status

Used in production (master branch)

Benchmark

All tests here

Some tests, MacBook Pro (Retina, 13-inch, Early 2015)

Test 1

Number of keys: 1000000 Minimum key size: 16, maximum key size: 64 Minimum value size: 128, maximum value size: 512 Concurrency: 2

	pogreb	goleveldb	bolt	badgerdb	pudge	slowpoke	pudge(mem)
1M (Put+Get), seconds	187	38	126	34	23	23	2
1M Put, ops/sec	5336	34743	8054	33539	47298	46789	439581
1M Get, ops/sec	1782423	98406	499871	220597	499172	445783	1652069
FileSize,Mb	568	357	552	487	358	358	358