Popularity
6.5
Stable
Activity
5.3
-
459
8
87
Programming language: Go
License: Apache License 2.0
Tags:
Database
Latest version: v1.10
clickhouse-bulk alternatives and similar packages
Based on the "Database" category.
Alternatively, view clickhouse-bulk alternatives based on common mentions on social networks and blogs.
-
prometheus
The Prometheus monitoring system and time series database. -
Milvus
A cloud-native vector database, storage for next generation AI applications -
influxdb
Scalable datastore for metrics, events, and real-time analytics -
cockroach
CockroachDB - the open source, cloud-native distributed SQL database. -
tidb
TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://tidbcloud.com/free-trial -
vitess
Vitess is a database clustering system for horizontal scaling of MySQL. -
TinyGo
Go compiler for small places. Microcontrollers, WebAssembly (WASM/WASI), and command-line tools. Based on LLVM. -
rqlite
The lightweight, distributed relational database built on SQLite. -
groupcache
groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases. -
VictoriaMetrics
VictoriaMetrics: fast, cost-effective monitoring solution and time series database -
bytebase
The GitLab/GitHub for database DevOps. World's most advanced database DevOps and CI/CD for Developer, DBA and Platform Engineering teams. -
go-cache
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications. -
immudb
immudb - immutable database based on zero trust, SQL/Key-Value/Document model, tamperproof, data change history -
go-mysql-elasticsearch
Sync MySQL data into elasticsearch -
rosedb
Lightweight, fast and reliable key/value storage engine based on Bitcask. -
buntdb
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support -
pREST
PostgreSQL ➕ REST, low-code, simplify and accelerate development, ⚡ instant, realtime, high-performance on any Postgres application, existing or new -
dbmate
:rocket: A lightweight, framework-agnostic database migration tool. -
go-memdb
Golang in-memory database built on immutable radix trees -
xo
Command line tool to generate idiomatic Go code for SQL databases supporting PostgreSQL, MySQL, SQLite, Oracle, and Microsoft SQL Server -
tiedot
A rudimentary implementation of a basic document (NoSQL) database in Go -
nutsdb
A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set. -
LinDB
LinDB is a scalable, high performance, high availability distributed time series database. -
cache2go
Concurrency-safe Go caching library with expiration capabilities and access counters -
GCache
An in-memory cache library for golang. It supports multiple eviction policies: LRU, LFU, ARC -
gocraft/dbr (database records)
Additions to Go's database/sql for super fast performance and convenience. -
lotusdb
Most advanced key-value store written in Go, extremely fast, compatible with LSM tree and B+ tree, optimization of badger and bbolt. -
fastcache
Fast thread-safe inmemory cache for big number of entries in Go. Minimizes GC overhead
InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Promo
www.influxdata.com
Do you think we are missing an alternative of clickhouse-bulk or a related project?
README
ClickHouse-Bulk
Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers.
Installation
Download binary for you platorm
or
or from sources (Go 1.13+):
git clone https://github.com/nikepan/clickhouse-bulk
cd clickhouse-bulk
go build
Features
- Group n requests and send to any of ClickHouse server
- Sending collected data by interval
- Tested with VALUES, TabSeparated formats
- Supports many servers to send
- Supports query in query parameters and in body
- Supports other query parameters like username, password, database
- Supports basic authentication
For example:
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')
INSERT INTO table3 (c1, c2, c3) VALUES ('v4', 'v5', 'v6')
sends as
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')('v4', 'v5', 'v6')
Options
- -config - config file (json); default config.json
Configuration file
{
"listen": ":8124",
"flush_count": 10000, // check by \n char
"flush_interval": 1000, // milliseconds
"clean_interval": 0, // how often cleanup internal tables - e.g. inserts to different temporary tables, or as workaround for query_id etc. milliseconds
"remove_query_id": true, // some drivers sends query_id which prevents inserts to be batched
"dump_check_interval": 300, // interval for try to send dumps (seconds); -1 to disable
"debug": false, // log incoming requests
"dump_dir": "dumps", // directory for dump unsended data (if clickhouse errors)
"clickhouse": {
"down_timeout": 60, // wait if server in down (seconds)
"connect_timeout": 10, // wait for server connect (seconds)
"tls_server_name": "", // override TLS serverName for certificate verification (e.g. in cases you share same "cluster" certificate across multiple nodes)
"insecure_tls_skip_verify": false, // INSECURE - skip certificate verification at all
"servers": [
"http://127.0.0.1:8123"
]
}
}
Environment variables (used for docker image)
CLICKHOUSE_BULK_DEBUG
- enable debug loggingCLICKHOUSE_SERVERS
- comma separated list of serversCLICKHOUSE_FLUSH_COUNT
- count of rows for insertCLICKHOUSE_FLUSH_INTERVAL
- insert intervalCLICKHOUSE_CLEAN_INTERVAL
- internal tables clean intervalDUMP_CHECK_INTERVAL
- interval of resend dumpsCLICKHOUSE_DOWN_TIMEOUT
- wait time if server is downCLICKHOUSE_CONNECT_TIMEOUT
- clickhouse server connect timeoutCLICKHOUSE_TLS_SERVER_NAME
- server name for TLS certificate verificationCLICKHOUSE_INSECURE_TLS_SKIP_VERIFY
- skip certificate verification at all
Quickstart
./clickhouse-bulk
and send queries to :8124
Metrics
manual check main metrics
curl -s http://127.0.0.1:8124/metrics | grep "^ch_"
ch_bad_servers 0
- actual count of bad serversch_dump_count 0
- dumps saved from launchch_queued_dumps 0
- actual dump files id directorych_good_servers 1
- actual good servers countch_received_count 40
- received requests count from launchch_sent_count 1
- sent request count from launch
Tips
For better performance words FORMAT and VALUES must be uppercase.