pudge

Fast and simple key/value store written using Go's standard library

Programming language: Go

License: MIT License

Tags: Database    

Latest version: v1.0.3

README

Table of Contents

• Description
• Usage
• Cookbook
• Disadvantages
• Motivation
• Benchmarks
  • Test 1
  • Test 4

Description

Package pudge is a fast and simple key/value store written using Go's standard library.

It presents the following:

• Supporting very efficient lookup, insertions and deletions
• Performance is comparable to hash tables
• Ability to get the data in sorted order, which enables additional operations like range scan
• Select with limit/offset/from key, with ordering or by prefix
• Safe for use in goroutines
• Space efficient
• Very short and simple codebase
• Well tested, used in production

Usage

package main

import (
    "log"

    "github.com/recoilme/pudge"
)

func main() {
    // Close all database on exit
    defer pudge.CloseAll()

    // Set (directories will be created)
    pudge.Set("../test/test", "Hello", "World")

    // Get (lazy open db if needed)
    output := ""
    pudge.Get("../test/test", "Hello", &output)
    log.Println("Output:", output)

    ExampleSelect()
}


//ExampleSelect
func ExampleSelect() {
    cfg := &pudge.Config{
        SyncInterval: 1} // every second fsync
    db, err := pudge.Open("../test/db", cfg)
    if err != nil {
        log.Panic(err)
    }
    defer db.DeleteFile()
    type Point struct {
        X int
        Y int
    }
    for i := 100; i >= 0; i-- {
        p := &Point{X: i, Y: i}
        db.Set(i, p)
    }
    var point Point
    db.Get(8, &point)
    log.Println(point)
    // Output: {8 8}
    // Select 2 keys, from 7 in ascending order
    keys, _ := db.Keys(7, 2, 0, true)
    for _, key := range keys {
        var p Point
        db.Get(key, &p)
        log.Println(p)
    }
    // Output: {8 8}
    // Output: {9 9}
}

Cookbook

• Store data of any type. Pudge uses Gob encoder/decoder internally. No limits on keys/values size.

pudge.Set("strings", "Hello", "World")
pudge.Set("numbers", 1, 42)

type User struct {
    Id int
    Name string
}
u := &User{Id: 1, Name: "name"}
pudge.Set("users", u.Id, u)

• Pudge is stateless and safe for use in goroutines. You don't need to create/open files before use. Just write data to pudge, don't worry about state. web server example

• Pudge is parallel. Readers don't block readers, but a writer - does, but by the stateless nature of pudge it's safe to use multiples files for storages.

• Default store system: like memcache + file storage. Pudge uses in-memory hashmap for keys, and writes values to files (no value data stored in memory). But you may use inmemory mode for values, with custom config: golang cfg = pudge.DefaultConfig() cfg.StoreMode = 2 db, err := pudge.Open(dbPrefix+"/"+group, cfg) ... db.Counter(key, val) In that case, all data is stored in memory and will be stored on disk only on Close.

Example server for highload, with http api

• You may use pudge as an engine for creating databases.

Example database

• Don't forget to close all opened databases on shutdown/kill.

  // Wait for interrupt signal to gracefully shutdown the server 
  quit := make(chan os.Signal)
  signal.Notify(quit, os.Interrupt, os.Kill)
  <-quit
  log.Println("Shutdown Server ...")
  if err := pudge.CloseAll(); err != nil {
      log.Println("Pudge Shutdown err:", err)
  }
  

  example recovery function for gin framework

• Pudge has a primitive select/query engine.

  // Select 2 keys, from 7 in ascending order
  keys, _ := db.Keys(7, 2, 0, true)
  // select keys from db where key>7 order by keys asc limit 2 offset 0
  

• Pudge will work well on SSD or spined disks. Pudge doesn't eat memory or storage or your sandwich. No hidden compaction/rebalancing/resizing and so on tasks. No LSM Tree. No MMap. It's a very simple database with less than 500 LOC. It's good for simple social network or highload system

Disadvantages

• No transaction system. All operations are isolated, but you don't may batching them with automatic rollback.
• Keys function (select/query engine) may be slow. Speed of query may vary from 10ms to 1sec per million keys. Pudge don't use BTree/Skiplist or Adaptive radix tree for store keys in ordered way on every insert. Ordering operation is "lazy" and run only if needed.
• If you need storage or database for hundreds of millions keys - take a look at Sniper or b52. They are optimized for highload (pudge - not).
• No fsync on every insert. Most of database fsync data by the timer too
• Deleted data don't remove from physically (but upsert will try to reuse space). You may shrink database only with backup right now golang pudge.BackupAll("backup")
• Keys automatically convert to binary and ordered with binary comparator. It's simple for use, but ordering will not work correctly for negative numbers for example
• Author of project don't work at Google or Facebook and his name not Howard Chu or Brad Fitzpatrick. But I'm open for issue or contributions.

Motivation

Some databases very well for writing. Some of the databases very well for reading. But pudge is well balanced for both types of operations. It has small cute api, and don't have hidden graveyards. It's just hashmap where values written in files. And you may use one database for in-memory/persistent storage in a stateless stressfree way

Benchmarks

All tests here

Some tests, MacBook Pro (Retina, 13-inch, Early 2015)

Test 1

Number of keys: 1000000 Minimum key size: 16, maximum key size: 64 Minimum value size: 128, maximum value size: 512 Concurrency: 2

	pogreb	goleveldb	bolt	badgerdb	pudge	slowpoke	pudge(mem)
1M (Put+Get), seconds	187	38	126	34	23	23	2
1M Put, ops/sec	5336	34743	8054	33539	47298	46789	439581
1M Get, ops/sec	1782423	98406	499871	220597	499172	445783	1652069
FileSize,Mb	568	357	552	487	358	358	358

Test 4

Number of keys: 10000000 Key size: 8 Value size: 16 Concurrency: 100

	goleveldb	badgerdb	pudge
10M (Put+Get), seconds	165	120	243
10M Put, ops/sec	122933	135709	43843
10M Get, ops/sec	118722	214981	666067
FileSize,Mb	312	1370	381