remember-go

Cache Slow Database Queries

Programming language: Go

License: GNU General Public License v3.0 or later

Tags: Data Structures    

Latest version: v1.1.1

README

Cache Slow Database Queries

This package is used to cache the results of slow database queries in memory or Redis. It can be used to cache any form of data. A Redis and in-memory storage driver is provided.

See Article for further details including a tutorial.

The package is production ready and the API is stable. A variant of this package has been used in production for over 3 years.

⭐ the project to show your appreciation.

Installation

go get -u github.com/rocketlaunchr/remember-go

Create a Key

Let’s assume the query’s argument is an arbitrary search term and a page number for pagination.

CreateKeyStruct

CreateKeyStruct can generate a JSON based key by providing a struct.

type Key struct {
    Search string
    Page   int `json:"page"`
}

var key string = remember.CreateKeyStruct(Key{"golang", 2})

CreateKey

CreateKey provides more flexibility to generate keys:

// Key will be "search-golang-2"
key :=  remember.CreateKey(false, "-", "search-x-y", "search", "golang", 2)

Initialize the Storage Driver

In-Memory

import "github.com/rocketlaunchr/remember-go/memory"

var ms = memory.NewMemoryStore(10 * time.Minute)

Redis

The Redis storage driver relies on Gary Burd’s excellent Redis client library.

import red "github.com/rocketlaunchr/remember-go/redis"
import "github.com/gomodule/redigo/redis"

var rs = red.NewRedisStore(&redis.Pool{
    Dial: func() (redis.Conn, error) {
        return redis.Dial("tcp", "localhost:6379")
    },
})

Memcached

An experimental (and untested) memcached driver is provided. It relies on Brad Fitzpatrick's memcache driver.

Ristretto

DGraph's Ristretto is a fast, fixed size, in-memory cache with a dual focus on throughput and hit ratio performance.

Create a SlowRetrieve Function

The package initially checks if data exists in the cache. If it doesn’t, then it elegantly fetches the data directly from the database by calling the SlowRetrieve function. It then saves the data into the cache so that next time it doesn’t have to refetch it from the database.

type Result struct {
    Title string
}

slowQuery := func(ctx context.Context) (interface{}, error) {
    results := []Result{}

    stmt := `
        SELECT title
        FROM books WHERE title LIKE ?
        ORDER BY title LIMIT ?, 20
    `

    rows, _ := db.QueryContext(ctx, stmt, search, (page-1)*20)

    for rows.Next() {
        var title string
        rows.Scan(&title)
        results = append(results, Result{title})
    }

    return results, nil
}

Usage

key := remember.CreateKeyStruct(Key{"golang", 2})
exp := 10*time.Minute

results, found, err := remember.Cache(ctx, ms, key, exp, slowQuery, remember.Options{GobRegister: false})

return results.([]Result) // Type assert in order to use

Gob Register Errors

The Redis storage driver stores the data in a gob encoded form. You have to register with the gob package the data type returned by the SlowRetrieve function. It can be done inside a func init(). Alternatively, you can set the GobRegister option to true. This will slightly impact concurrency performance however.

Other useful packages

• dataframe-go - For statistics, machine-learning, and data manipulation/exploration
• dbq - Zero boilerplate database operations for Go
• electron-alert - SweetAlert2 for Electron Applications
• igo - A Go transpiler with cool new syntax such as fordefer (defer for for-loops)
• mysql-go - Properly cancel slow MySQL queries
• react - Build front end applications using Go

Legal Information

The license is a modified MIT license. Refer to LICENSE file for more details.

© 2019-20 PJ Engineering and Business Solutions Pty. Ltd.

Final Notes

Feel free to enhance features by issuing pull-requests.