Popularity
6.5
Stable
Activity
0.0
-
435
8
86
Programming language: Go
License: Apache License 2.0
Tags:
Database
Latest version: v1.10
clickhouse-bulk alternatives and similar packages
Based on the "Database" category.
Alternatively, view clickhouse-bulk alternatives based on common mentions on social networks and blogs.
-
prometheus
The Prometheus monitoring system and time series database. -
cockroach
CockroachDB - the open source, cloud-native distributed SQL database. -
tidb
TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://tidbcloud.com/free-trial -
influxdb
Scalable datastore for metrics, events, and real-time analytics -
Milvus
A cloud-native vector database, storage for next generation AI applications -
vitess
Vitess is a database clustering system for horizontal scaling of MySQL. -
rqlite
The lightweight, distributed relational database built on SQLite -
groupcache
groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases. -
TinyGo
Go compiler for small places. Microcontrollers, WebAssembly (WASM/WASI), and command-line tools. Based on LLVM. -
VictoriaMetrics
VictoriaMetrics: fast, cost-effective monitoring solution and time series database -
immudb
immudb - immutable database based on zero trust, SQL/Key-Value/Document model, tamperproof, data change history -
go-cache
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications. -
bytebase
Database DevOps and CI/CD for Developer, DBA and Platform Engineering team. -
go-mysql-elasticsearch
Sync MySQL data into elasticsearch -
rosedb
Lightweight, fast and reliable key/value storage engine based on Bitcask. -
pREST
PostgreSQL ➕ REST, low-code, simplify and accelerate development, ⚡ instant, realtime, high-performance on any Postgres application, existing or new -
buntdb
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support -
go-memdb
Golang in-memory database built on immutable radix trees -
dbmate
:rocket: A lightweight, framework-agnostic database migration tool. -
xo
Command line tool to generate idiomatic Go code for SQL databases supporting PostgreSQL, MySQL, SQLite, Oracle, and Microsoft SQL Server -
tiedot
A rudimentary implementation of a basic document (NoSQL) database in Go -
nutsdb
A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set. -
LinDB
LinDB is a scalable, high performance, high availability distributed time series database. -
cache2go
Concurrency-safe Go caching library with expiration capabilities and access counters -
GCache
An in-memory cache library for golang. It supports multiple eviction policies: LRU, LFU, ARC -
fastcache
Fast thread-safe inmemory cache for big number of entries in Go. Minimizes GC overhead -
gocraft/dbr (database records)
Additions to Go's database/sql for super fast performance and convenience. -
CovenantSQL
A decentralized, trusted, high performance, SQL database with blockchain features -
lotusdb
Most advanced key-value store written in Go, extremely fast, compatible with LSM tree and B+ tree, optimization of badger and bbolt.
Free Global Payroll designed for tech teams
Building a great tech team takes more than a paycheck. Zero payroll costs, get AI-driven insights to retain best talent, and delight them with amazing local benefits. 100% free and compliant.
Promo
try.revelo.com
Do you think we are missing an alternative of clickhouse-bulk or a related project?
README
ClickHouse-Bulk
Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers.
Installation
Download binary for you platorm
or
or from sources (Go 1.13+):
git clone https://github.com/nikepan/clickhouse-bulk
cd clickhouse-bulk
go build
Features
- Group n requests and send to any of ClickHouse server
- Sending collected data by interval
- Tested with VALUES, TabSeparated formats
- Supports many servers to send
- Supports query in query parameters and in body
- Supports other query parameters like username, password, database
- Supports basic authentication
For example:
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')
INSERT INTO table3 (c1, c2, c3) VALUES ('v4', 'v5', 'v6')
sends as
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')('v4', 'v5', 'v6')
Options
- -config - config file (json); default config.json
Configuration file
{
"listen": ":8124",
"flush_count": 10000, // check by \n char
"flush_interval": 1000, // milliseconds
"clean_interval": 0, // how often cleanup internal tables - e.g. inserts to different temporary tables, or as workaround for query_id etc. milliseconds
"remove_query_id": true, // some drivers sends query_id which prevents inserts to be batched
"dump_check_interval": 300, // interval for try to send dumps (seconds); -1 to disable
"debug": false, // log incoming requests
"dump_dir": "dumps", // directory for dump unsended data (if clickhouse errors)
"clickhouse": {
"down_timeout": 60, // wait if server in down (seconds)
"connect_timeout": 10, // wait for server connect (seconds)
"tls_server_name": "", // override TLS serverName for certificate verification (e.g. in cases you share same "cluster" certificate across multiple nodes)
"insecure_tls_skip_verify": false, // INSECURE - skip certificate verification at all
"servers": [
"http://127.0.0.1:8123"
]
}
}
Environment variables (used for docker image)
CLICKHOUSE_BULK_DEBUG
- enable debug loggingCLICKHOUSE_SERVERS
- comma separated list of serversCLICKHOUSE_FLUSH_COUNT
- count of rows for insertCLICKHOUSE_FLUSH_INTERVAL
- insert intervalCLICKHOUSE_CLEAN_INTERVAL
- internal tables clean intervalDUMP_CHECK_INTERVAL
- interval of resend dumpsCLICKHOUSE_DOWN_TIMEOUT
- wait time if server is downCLICKHOUSE_CONNECT_TIMEOUT
- clickhouse server connect timeoutCLICKHOUSE_TLS_SERVER_NAME
- server name for TLS certificate verificationCLICKHOUSE_INSECURE_TLS_SKIP_VERIFY
- skip certificate verification at all
Quickstart
./clickhouse-bulk
and send queries to :8124
Metrics
manual check main metrics
curl -s http://127.0.0.1:8124/metrics | grep "^ch_"
ch_bad_servers 0
- actual count of bad serversch_dump_count 0
- dumps saved from launchch_queued_dumps 0
- actual dump files id directorych_good_servers 1
- actual good servers countch_received_count 40
- received requests count from launchch_sent_count 1
- sent request count from launch
Tips
For better performance words FORMAT and VALUES must be uppercase.