Popularity
0.7
Growing
Activity
8.8
Growing
6
1
0

Programming language: Go
Tags: Database    

tracedb alternatives and similar packages

Based on the "Database" category

Do you think we are missing an alternative of tracedb or a related project?

Add another 'Database' Package

README

tracedb GoDoc Go Report Card Coverage Status

tracedb: blazing fast timeseries database for IoT and real-time messaging applications

tracedb is blazing fast timeseries database for IoT, realtime messaging applications. Access tracedb with pubsub over tcp or websocket using trace application.

Tracedb can be used for online gaming and mobile apps as it satisfy the requirements for low latency and binary messaging. Tracedb is perfect timeseries data store for applications such as internet of things and internet connected devices.

Key characteristics

  • 100% Go.
  • Optimized for fast lookups and bulk inserts.
  • Can store larger-than-memory data sets.
  • Low memory usage.
  • All DB methods are safe for concurrent use by multiple goroutines.

Planned

  • Database backup and restore. add backup and restore of tracedb data and logs to archive these to external storage systems.
  • Add system topics (read only topics) for systems notifications. For example topic -> "system/errors" to send realtime detailed error messages to client.
  • Documentation - document the technical atchitecture, design principals and advanced usage guides such as optimum configuration guideline to acive maximum throughput for hyper scale writes/reads operations (without bloting memory buffers).

Table of Contents

Quick Start

To build tracedb from source code use go get command.

go get -u github.com/unit-io/tracedb

Usage

Opening a database

To open or create a new database, use the tracedb.Open() function:

package main

import (
    "log"

    "github.com/unit-io/tracedb"
)

func main() {
    db, err := tracedb.Open("tracedb.example", nil)
    if err != nil {
        log.Fatal(err)
        return
    }   
    defer db.Close()
}

Writing to a database

Store a message

Use DB.Put() to store message to a topic or use DB.PutEntry() to store message entry to a topic. DB.PutEntry() allows client to specify ID and Contract parameters. See topic isolation section for more detail.


    err := db.Put([]byte("unit8.b.b1"), []byte("msg.b.b1.1"))

    err := db.PutEntry(tracedb.NewEntry([]byte("unit8.b.b1"),[]byte("msg.b.b1.2")))

Writing to wildcard topics

Tracedb supports wrting to wildcard topics. Use "*" in the topic to write to wildcard topic or use "..." at the end of topic to write to all sub-topics. Writing to following wildcard topics are also supported, "*" or "..."

    b.PutEntry(tracedb.NewEntry([]byte("unit8.*.b11"), []byte("msg.*.b11.1")))
    b.PutEntry(tracedb.NewEntry([]byte("unit8.b.*"), []byte("msg.b.*.1")))
    b.PutEntry(tracedb.NewEntry([]byte("unit8..."), []byte("msg...1")))
    b.PutEntry(tracedb.NewEntry([]byte("*"), []byte("msg.*.1")))
    b.PutEntry(tracedb.NewEntry([]byte("..."), []byte("msg...1")))

Specify ttl

Specify ttl parameter to a topic while storing messages to expire it after specific duration. Note, DB.Get() or DB.Items() function does not fetch expired messages.

    b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b1?ttl=1m"), []byte("msg.b.b1.1m")))
    b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b11?ttl=1h"), []byte("msg.b.b11.1h")))
    b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b111?ttl=1h30m"), []byte("msg.b.b111.1h30m")))

Read messages

Use DB.Get() to read messages from a topic. Use last parameter to specify duration or specify number of recent messages to read from a topic. for example, "last=1h" gets messsages from tracedb stored in last 1 hour, or "last=100" to gets last 100 messages from tracedb. Specify an optional parameter Query.Limit to retrives messages from a topic with a limit.


    msgs, err := db.Get(&tracedb.Query{Topic: []byte("unit8.b.b1?last=10")})
    ....
    msgs, err := db.Get(&tracedb.Query{Topic: []byte("unit8.b.b1?last=10m", Limit: 100}))

Deleting a message

Deleting a message in tracedb is rare and it require additional steps to delete message from a given topic. Generate a unique message ID using DB.NewID() and use this unique message ID while putting message to the tracedb using DB.PutEntry(). To delete message provide message ID to the DB.DeleteEntry() fucntion.


    messageId := db.NewID()
    err := db.PutEntry(&tracedb.Entry{
        ID:       messageId,
        Topic:    []byte("unit8.b.b1"),
        Payload:  []byte("msg.b.b1.deleting"),
    })

    err := db.DeleteEntry(&tracedb.Entry{
        ID:       messageId,
        Topic:    []byte("unit8.b.b1"),
    })

Topic isolation

Topic isolation can be achieved using Contract while putting messages into tracedb or querying messages from a topic. Use DB.NewContract() to generate a new Contract and then specify Contract while putting messages using DB.PutEntry() function. Use Contract in the query to get messages from a topic.

    contract, err := db.NewContract()

    messageId := db.NewID()
    err := db.PutEntry(&tracedb.Entry{
        ID:       messageId,
        Topic:    []byte("unit8.b.b1"),
        Payload:  []byte("msg.b.b1.2"),
        Contract: contract,
    })

    err := db.DeleteEntry(&tracedb.Entry{
        ID:       messageId,
        Topic:    []byte("unit8.b.b1"),
        Contract: contract,
    })
    ....
    msgs, err := db.Get(&tracedb.Query{Topic: []byte("unit8.b.b1?last=10m", Contract: contract, Limit: 100}))

Batch operation

Use batch operation to bulk insert records into tracedb or bulk delete records from tracedb. Batch operation also speeds up reading data if batch operation is used then reading records within short span of time while db is still open. See benchmark examples and run it locally to see performance of runnig batches concurrently.

Writing to a batch

Use Batch.Put() to write to a single topic in a batch. To write to single topic in a batch specify topic in batch options.

    // Writing to single topic in a batch
    err := db.Batch(func(b *tracedb.Batch, completed <-chan struct{}) error {
        opts := tracedb.DefaultBatchOptions
        opts.Topic = []byte("unit8.b.*?ttl=1m")
        b.SetOptions(opts)
        b.Put([]byte("msg.b.*.1"))
        b.Put([]byte("msg.b.*.2"))
        b.Put([]byte("msg.b.*.3"))
        err := b.Write()
        return err
    })

Writing to multiple topics in a batch

Use Batch.PutEntry() function to store messages to multiple topics in a batch.


    // Writing to multiple topics in a batch
    err := db.Batch(func(b *tracedb.Batch, completed <-chan struct{}) error {
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b1"), []byte("msg.b.b11.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b11"), []byte("msg.b.b11.2")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.*"), []byte("msg.b.*.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.*"), []byte("msg.b.*.2")))
        err := b.Write()
        return err
    })

Non-blocking batch operation

All batch operations are non-blocking so client program can decide to wait for completed signal and further execute any additional tasks.

    err := db.Batch(func(b *tracedb.Batch, completed <-chan struct{}) error {
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b1"), []byte("msg.b.b11.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b11"), []byte("msg.b.b11.2")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.*"), []byte("msg.b.*.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.*"), []byte("msg.b.*.2")))
        err := b.Write()
            go func() {
                <-completed // it signals batch has completed and fully committed to db
                print([]byte("unit8.b.b1?last=1m"), db)
                print([]byte("unit8.b.b11?last=1m"), db)
            }()
        return err
    })

Writing batch chunk

Batch operation support writing chunk for large batch. It is safe to use Batch.Write() function multiple times within a batch operation.

As duplicate values are removed in a batch write operation, so specify batch option "AllowDuplicates" to keep duplicate values.

    err := db.Batch(func(b *tracedb.Batch, completed <-chan struct{}) error {
        opts := tracedb.DefaultBatchOptions
        opts.Topic = []byte("unit8.b.*?ttl=1m")
        opts.AllowDuplicates = true
        b.SetOptions(opts)
        t := time.Now()
        for j := 0; j < 250; j++ {
            b.Put(t.MarshalText())
            if j%100 == 0 {
                if err := b.Write(); err != nil {
                    return err
                }
            }
        }
        if err := b.Write(); err != nil {
            return err
        }
        go func() {
            <-completed // it signals batch has completed and fully committed to db
            print([]byte("unit8.b.b1?last=1m"), db)
            print([]byte("unit8.b.b11?last=1m"), db)
        }()
        return nil
    })

Topic isolation in batch operation

Topic isolation can be achieved using Contract while putting messages into tracedb and querying messages from a topic. Use DB.NewContract() to generate a new Contract and then specify Contract while putting messages using Batch.PutEntry() function.

    contract, err := db.NewContract()

    // Writing to single topic in a batch
    err := db.Batch(func(b *tracedb.Batch, completed <-chan struct{}) error {
        opts := tracedb.DefaultBatchOptions
        opts.Topic = []byte("unit8.b.*?ttl=1m")
        opts.Contract = contract
        b.SetOptions(opts)
        b.Put([]byte("msg.b.*.1"))
        b.Put([]byte("msg.b.*.2"))
        b.Put([]byte("msg.b.*.3"))
        err := b.Write()
            go func() {
                <-completed // it signals batch has completed and fully committed to db
                print([]byte("unit8.b.b1?last=1m"), db)
                print([]byte("unit8.b.b11?last=1m"), db)
            }()
        return err
    })

    // Writing to multiple topics in a batch
    err := db.Batch(func(b *tracedb.Batch, completed <-chan struct{}) error {
        opts := tracedb.DefaultBatchOptions
        opts.Contract = contract
        b.SetOptions(opts)
        b.PutEntry(tracedb.NewEntry([]byte("unit8.*.b11"), []byte("msg.*.b11.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.*"), []byte("msg.b.*.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8..."), []byte("msg...")))
        b.PutEntry(tracedb.NewEntry([]byte("*"), []byte("msg.*.1")))
        b.PutEntry(tracedb.NewEntry([]byte("..."), []byte("msg...1")))
        err := b.Write()
        go func() {
            <-completed // it signals batch has completed and fully committed to db
            printWithContract([]byte("unit8.b.b11?last=1m"), contract, db)
        }()
        return err
    })

Message encryption

Set encyrption flag in batch options to encrypt all messages in a batch.

Note, encryption can also be set on entire database using DB.Open() and set encryption flag in options parameter.

    err := db.Batch(func(b *tracedb.Batch, completed <-chan struct{}) error {
        opts := tracedb.DefaultBatchOptions
        opts.Encryption = true
        opts.Topic = []byte("unit8.b.b11?ttl=1m")
        b.SetOptions(opts)
        b.Put([]byte("msg.b.b11.1"))
        b.Put([]byte("msg.b.b11.2"))
        b.Put([]byte("msg.b.b11.3"))
        err := b.Write()
        return err
    })

Batch group

Use BatchGroup.Add() function to group batches and run concurrently without causing write conflict. Use the BatchGroup.Run to run group of batches concurrently. See usage example in below code snippet.

    g := db.NewBatchGroup()
    g.Add(func(b *tracedb.Batch, completed <-chan struct{}) error {
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b1?ttl=2m"), []byte("msg.b.b1.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.c.c1?ttl=1m"), []byte("msg.c.c1.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b1?ttl=3m"), []byte("msg.b.b1.2")))
        err := b.Write()
        go func() {
            <-completed // it signals batch group completion
        }()
        return err
    })

    g.Add(func(b *tracedb.Batch, completed <-chan struct{}) error {
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b11"), []byte("msg.b.b11.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b11"), []byte("msg.b.b11.2")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b1"), []byte("msg.b.b1.3")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.c.c11"), []byte("msg.c.c11.1")))
        err := b.Write()
        go func() {
            <-completed // it signals batch group completion
        }()
        return err
    })

    g.Add(func(b *tracedb.Batch, completed <-chan struct{}) error {
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b111"), []byte("msg.b.b111.1")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b111"), []byte("msg.b.b111.2")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.b.b11"), []byte("msg.b.b11.3")))
        b.PutEntry(tracedb.NewEntry([]byte("unit8.c.c111"), []byte("msg.c.c111")))
        err := b.Write()
        go func() {
            <-completed // it signals batch group completion
        }()
        return err
    })

    err = g.Run()

Iterating over items

Use the DB.Items() function which returns a new instance of ItemIterator. Specify topic to retrives values and use last parameter to specify duration or specify number of recent messages to retreive from the topic. for example, "last=1h" retrieves messsages from tracedb stored in last 1 hour, or "last=100" to retreives last 100 messages from the tracedb:

func print(topic []byte, db *tracedb.DB) {
    // topic -> "unit8.b.b1?last=1m"
    it, err := db.Items(&tracedb.Query{Topic: topic})
    if err != nil {
        log.Fatal(err)
        return
    }
    for it.First(); it.Valid(); it.Next() {
        err := it.Error()
        if err != nil {
            log.Fatal(err)
            return
        }
        log.Printf("%s %s", it.Item().Topic(), it.Item().Value())
    }
}

Statistics

The tracedb keeps a running metrics of internal operations it performs. To get tracedb metrics use DB.Varz() function. The tracedb can perform hyper scale writes, but performance starts degrading due to hardware limitation if writing over 5 millions entries within span of 10 secs. The tracedb will support distribution in upcoming release.


    if varz, err := db.Varz(); err == nil {
        fmt.Printf("%+v\n", varz)
    }

Limitations

The tracedb is not primarily designed for key-value storage. The tracedb mainly designed to store time-series data, messaging data, log data or AI analytics data.

Contributing

If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are welcome.

Licensing

Copyright (c) 2016-2020 Saffat IT Solutions Pvt Ltd. This project is licensed under Affero General Public License v3.


*Note that all licence references and agreements mentioned in the tracedb README section above are relevant to that project's source code only.