psql-streamer alternatives and similar packages
Based on the "Server Applications" category.
Alternatively, view psql-streamer alternatives based on common mentions on social networks and blogs.
-
etcd
Distributed reliable key-value store for the most critical data of a distributed system -
Caddy
Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS -
Vault
A tool for secrets management, encryption as a service, and privileged access management -
consul
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure. -
apex
Build, deploy, and manage AWS Lambda functions with ease (with Go support!). -
RoadRunner
🤯 High-performance PHP application server, process manager written in Go and powered with plugins -
SFTPGo
Fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob -
Ponzu
Headless CMS with automatic JSON API. Featuring auto-HTTPS from Let's Encrypt, HTTP/2 Server Push, and flexible server framework written in Go. -
Jocko
Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native) -
Flagr
Flagr is a feature flagging, A/B testing and dynamic configuration microservice -
algernon
Small self-contained pure-Go web server with Lua, Teal, Markdown, HTTP/2, QUIC, Redis and PostgreSQL support -
discovery
A registry for resilient mid-tier load balancing and failover. -
Key Transparency
A transparent and secure way to look up public keys. -
Trickster
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator -
Rendora
dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites -
jackal
💬 Instant messaging server for the Extensible Messaging and Presence Protocol (XMPP). -
Golang API Starter Kit
Go Server/API boilerplate using best practices DDD CQRS ES gRPC -
go-feature-flag
GO Feature Flag is a simple, complete and lightweight self-hosted feature flag solution 100% Open Source. 🎛️ -
SDNS
A high-performance, recursive DNS resolver server with DNSSEC support, focused on preserving privacy. -
Euterpe
Self-hosted music streaming server 🎶 with RESTful API and Web interface. Think of it as your very own Spotify! ☁️🎧 -
Walrus
🔥 Fast, Secure and Reliable System Backup, Set up in Minutes. -
goproxy
🦁 goproxy is a proxy server which can forward http or https requests to remote servers./ goproxy 是一个反向代理服务器,支持转发 http/https 请求。 -
Eru
Eru, a simple, stateless, flexible, production-ready orchestrator designed to easily integrate into existing workflows. Can run any virtualization things in long or short time. -
marathon-consul
Integrates Marathon apps with Consul service discovery. -
cortex-tenant
Prometheus remote write proxy that adds Cortex/Mimir tenant ID based on metric labels -
lets-proxy2
Reverse proxy with automatically obtains TLS certificates from Let's Encrypt -
go-proxy-cache
Simple Reverse Proxy with Caching, written in Go, using Redis. -
Simple CRUD App w/ Gorilla/Mux, MariaDB
Simple CRUD Application with Go, Gorilla/mux, MariaDB, Redis. -
simple-jwt-provider
Simple and lightweight provider which exhibits JWTs, supports login, password-reset (via mail) and user management. -
autobd
autobd is an automated, networked and containerized backup solution -
protoxy
A proxy server than converts JSON request bodies to protocol buffers -
go-fitbit
Fitbit API for Go to fetch, add, update and delete data on Fitbit using REST API -
Gin Healthcheck
Lightweight, customizable healthcheck endpoint for Gin Framework
Updating dependencies is time-consuming.
Do you think we are missing an alternative of psql-streamer or a related project?
README
PSQL-Streamer
This service receives the database events from PostgreSQL using logical replication protocol and feeds them to sinks based on the configuration file settings. Also can receive the events from Kafka generated by e.g. another psql-streamer instance.
Event example
{
"Host": "db1",
"Database": "test",
"Table": "test",
"Action": "insert",
"WALPosition": 418152976,
"Timestamp": "2018-07-03T16:04:27.263625156+02:00",
"Columns": {
"a": 369223,
"b": "a"
}
}
Features
- You can configure a sink to receive events from several different sources (e.g. several Kafka clusters, PostgreSQL databases)
- Basic HTTP API
- Custom sources/sinks can be easily added - they only need to implement a simple interface
- Requires PostgreSQL 10+ (although initial implementation of logical replication came in 9.4 it lacks certain features to be useful)
Sources
PostgreSQL
Receive database events from PostgreSQL using logical replication protocol and transform them into a common event format.
Features
WAL log position persistence: the service persists each log position update it receives in a BoltDB database.
Configurable WAL logs retention on the PostgreSQL side. To allow us to rewind back (in case we need to replay some events) the service implements delayed confirmation of applied logs. This makes PostgreSQL retain the logs for the specified replication slot for some time. See walRetain parameter in the configuration file.
PostgreSQL configuration
- In
postgresql.conf
you need to setwal_level = logical
to make logical replication possible. You also may need to adjustmax_wal_senders
andmax_replication_slots
to match your requirements. - Create a publication in PostgreSQL like this:
CREATE PUBLICATION pub1 FOR ALL TABLES
. This will include in the publication all existing tables and also the ones that will be created in future. If you want only a subset of tables to be replicated - list them specifically. See PostgreSQL documentation for details. - Specify the publication name in the psql-streamer.conf
Kafka
Receive events which were generated by e.g. another psql-streamer instance from one or more Kafka topics. Expects messages in a JSON format conforming to a predefined structure (see Event structure below).
Features
Kafka source works in batching mode with confirmation: if all events in a batch from Kafka are ACKed by sinks then we commit the whole batch or don't commit anything. The batch elements are sent concurrently using goroutines, so it should not be made very large. If several topics are specified then they're worked on in separate goroutines which should provide more parallelism.
Sinks
Kafka
Send events into a Kafka topic based on several configurable rules like:
- Table -> Topic mapping
- Fallback topic messages with no mapping defined
Kafka sink handlers
Incoming events are processed by one or more handlers that generate Kafka messages. They can be used to enrich Events with additional data or encode them in different format.
Currently there's only one handler:
- passthrough: simply marshal an event into JSON and send it to Kafka
Stub
Stub sink is a discard/noop sink that can be used for testing.
HTTP API
All requests are GET.
- /health is a no-op that just returns 200
- /status checks all sinks/sources statuses and outputs them. If something is not OK then it returns 500 instead of 200. Useful for monitoring.
- /stats returns statistics from all sinks/sources
- /debug/[on|off] toggles debugging mode for all sinks/sources
- /metrics gives out Prometheus metrics
Configuration
See psql-streamer.toml for detailed instructions.
Download
Get RPM or DEB packages from Releases page
Build
psql-streamer is written in Go, so you need to install it first.
Then you can run make build
to build a binary.
If you want to build RPM/DEB packages then you need to install FPM and then run make rpm
or make deb