All Versions
Latest Version
Avg Release Cycle
89 days
Latest Release

Changelog History
Page 2

  • v0.5.0-rc3 Changes

    August 15, 2017

    ๐Ÿš€ This is a pre-release of v0.5.0

    โšก๏ธ All vendor dependencies have been updated to the latest version and binaries have been compiled with go 1.9-beta2.

    ๐Ÿš€ The current release notes can be found under

    ๐Ÿ”„ Changes since 0.5.0-rc2

    • ๐Ÿ Windows build works again
    • Fully compatible with 1.9
    • Configs are case sensitive again (#208)
    • Configs errors because of wrong types or parameters now give fix suggestions
    • ๐Ÿ“‡ format.MetadataCopy sub-modulators removed
    • โž• Added a new format.Aggregate
    • ๐Ÿ”Œ Renamed "Aggregate" of the aggregate plugin type to "Plugins"
    • โž• Added a trace method to debug message flow
    • "-t / --trace" renamed to "-pt / --profiletrace", the new "-t / --trace" now refers to message tracing
    • ๐Ÿ“š Documentation updated
  • v0.4.5 Changes

    ๐Ÿš€ This is a patch / minor features release. โšก๏ธ All vendor dependencies have been updated to the latest version and binaries have been compiled with go 1.8.

    ๐Ÿ›  Fixed with 0.4.5

    • producer.Kafka will discard messages returned as "too large" to avoid spooling
    • consumer.HTTP does not truncate messages with WithHeaders:false anymore (thanks @mhils)
    • producer.Websocket now uses gorilla websockets (thanks @glaslos)
    • ๐Ÿณ Dockerfile is now working again
    • It is now possible to (optionally) send nil messages with producer.kafka again
    • โฑ Consumer.Kinesis will renew the iterator object when hitting a timeout
    • Consumer.Kinesis now runs with an offset file set that does not exist
    • Consumer.Kinesis offset file is now written less often (after each batch)
    • ๐Ÿ‘ป Consumer.Kafka does now retry with an "oldest" offset after encountering an OutOfRange exception.
    • ๐Ÿ›  Fixed a crash when using producer.ElasticSearch with date based indexes (thanks @relud)
    • format.Base64Decode now uses data from previous formatters as intended
    • format.JSON arr and obj will now auto create a key if necessary
    • format.JSON now checks for valid state references upon startup
    • format.JSON now properly encodes strings when using "enc"
    • ๐Ÿ›ฐ format.SplitToJSON may now keep JSON payload and is better at escaping string
    • "gollum -tc" will exit with error code 1 upon error
    • "gollum -tc" will now properly display errors during config checking

    ๐Ÿ†• New with 0.4.5

    • โž• Added producer for writing data to Amazon S3 (thanks @relud)
    • โž• Added authentication support to consumer.HTTP (thanks @glaslos)
    • โž• Added authentication support to native.KafkaProducer (thanks @relud)
    • โž• Added authentication support to producer.Kafka (thanks @relud)
    • โž• Added authentication support to consumer.Kafka (thanks @relud)
    • โž• Added consumer group support to consumer.Kafka (thanks @relud)
    • โž• Added a native SystemD consumer (thanks @relud)
    • โž• Added a Statsd producer for counting messages (thanks @relud)
    • โž• Added an option to flatten JSON arrays into single values with format.ProcessJSON (thanks @relud)
    • โž• Added filter.Any to allow "or" style combinations of filters (thanks @relud)
    • โž• Added support for unix timestamp parsing to format.ProcessJSON (thanks @relud)
    • โž• Added filter.Sample to allow processing of every n'th message only (thanks @relud)
    • โž• Added format.TemplateJSON to apply golang templates to JSON payloads (thanks @relud)
    • โž• Added named pipe support to consumer.Console
    • โž• Added "pick" option to format.ProcessJSON to get a single value from an arrays
    • ๐Ÿšš Extended "remove" option pf format.ProcessJSON to remove values from arrays
    • โž• Added "geoip" option to format.ProcessJSON to get GeoIP data from an IP
    • โž• Added index configuration options to producer.ElasticSearch
  • v0.4.4 Changes

    ๐Ÿš€ This is a patch / minor features release. โšก๏ธ All vendor dependencies have been updated to the latest version and binaries have been compiled with go 1.7.

    ๐Ÿ›  Fixed with 0.4.4

    • ๐Ÿ›  Fixed file offset handling in consumer.Kinesis (thanks @relud)
    • ๐Ÿ›  Fixed producer.File RotatePruneAfterHours setting
    • Producer.File symlink switch is now atomic
    • ๐Ÿ›  Fixed panic in producer.Redis when Formatter was not set
    • ๐Ÿ›  Fixed producer.Spooling being stuck for a long time during shutdown
    • ๐Ÿ›  Fixed native.KafkaProducer to map all topics to "default" if no topic mapping was set
    • ๐Ÿ›  Fixed a concurrent map write during initialization in native.KafkaProducer
    • ๐Ÿ›  Fixed consumer.Kafka OffsetFile setting stopping gollum when the offset file was not present
    • consumer.Kafka will retry to connect to a not (yet) existing topic every PersistTimeoutMs
    • Consumer.Kafka now tries to connect every ServerTimeoutSec if initial connect fails
    • 0๏ธโƒฃ Consumer.Kafka MessageBufferCount default value increased to 8192
    • Producer.Kafka and native.KafkaProducer now discard messages with 0-Byte content
    • 0๏ธโƒฃ Producer.Kafka SendRetries set to 1 by default to circumvent a reconnect issue within sarama
    • ๐Ÿ›  Fixed panic in producer.Kafka when shutting down
    • โž• Added manual heartbeat to check a valid broker connection with producer.Kafka
    • Format.Base64Encode now returns the original message if decoding failed
    • socket.producer TCP can be used without ACK
    • ๐Ÿ›  Consumer.Syslogd message handling differences between RFC3164 and RFC5424 / RFC6587 fixed

    ๐Ÿ†• New with 0.4.4

    • ๐Ÿ†• New AWS Firehose producer (thanks @relud)
    • ๐Ÿ†• New format.ProcessTSV for modifying TSV encoded messages (thanks @relud)
    • โž• Added user agent parsing to format.ProcessJSON (thanks @relud)
    • โž• Added support for unix timestamp parsing to format.ProcessJSON (thanks @relud)
    • โž• Added support for new shard detection to consumer.Kinesis (thanks @relud)
    • โž• Added support for mulitple messages per record to producer.Kinesis and consumer.Kinesis (thanks @relud)
    • โž• Added "remove" directive for format.ProcessJSON
    • โž• Added key Formatter support for producer.Redis
    • โž• Added RateLimited- metrics for filter.Rate
    • โž• Added format.Clear to remove message content (e.g. useful for key formatters)
    • โž• Added "KeyFormatterFirst" for producer.Kafka and native.KafkaProducer
    • โž• Added Version support for producer.Kafka and consumer.Kafka
    • โž• Added ClientID support for consumer.Kafka
    • โž• Added folder creation capatibilites to consumer.File when creating offset files
    • โž• Added gollum log messages metrics
    • โž• Added wildcard topic mapping to producer.Kafka and native.KafkaProducer
    • โž• Added FilterAfterFormat to producer.Kafka and native.KafkaProducer
    • Producer.Spooling now continuously looks for new streams to read
    • Producer.Spooling now reacts on SIGHUP to trigger a respooling
    • ๐Ÿ’ป Seperated version information to -r (version, go runtime, modules) and -v (just version) command line flag
    • โž• Added trace commandline flag
  • v0.4.3 Changes

    ๐Ÿš€ This is a patch / minor features release. ๐Ÿ”ง It includes several configuration changes for producer.Kafka that might change the runtime behavior. ๐Ÿ”ง Please check your configuration files.

    ๐Ÿ›  Fixed with 0.4.3

    • ๐Ÿ›  Fixed several race conditions reported by Go 1.6 and go build -race
    • ๐Ÿ›  Fixed the scribe producer to drop unformatted message in case of error
    • ๐Ÿ›  Fixed file.consumer rotation to work on regular files, too
    • ๐Ÿ›  Fixed file.consumer rotation to reset the offset file after a SIGHUP
    • โšก๏ธ Dockerfiles updated
    • ๐ŸŽ Producer.Kafka now sends messages directly to avoid sarama performance bottlenecks
    • 0๏ธโƒฃ consumer.Kafka offset file is properly read on startup if DefaultOffset "oldest" or "newest" is
    • ๐Ÿ‘ Exisiting unix domain socket detection changed to use create instead of stat (better error handling)
    • โšก๏ธ Kafka and Scribe specific metrics are now updated if there are no messages, too
    • ๐Ÿ‘ Scribe producer is now reacting better to server connection errors
    • โœ… Filters and Formatters are now covered with unittests

    ๐Ÿ†• New with 0.4.3

    • ๐Ÿ‘Œ Support for Go1.5 vendor experiment
    • ๐Ÿ†• New producer for librdkafka (not included in standard builds)
    • Metrics added to show memory consumption
    • ๐Ÿ†• New kafka metrics added to show "roundtrip" times for messages
    • producer.Benchmark added to get more meaningful core system profiling results
    • ๐Ÿ†• New filter filter.Rate added to allow limiting streams to a certain number of messages per second
    • โž• Added key support to consumer.Kafka and producer.Kafka
    • โž• Added an "ordered read" config option to consumer.Kafka (round robin reading)
    • โž• Added a new formater format.ExtractJSON to extract a single value from a JSON object
    • ๐Ÿ–จ Go version is now printed with gollum -v
    • Scribe producer now queries scribe server status in regular intervals
    • ๐Ÿ”ง format.Sequence separator character can now be configured
    • ๐Ÿ”ง format.Runlength separator character can now be configured

    Other changes with 0.4.3

    • ๐Ÿ“‡ Renamed producer.Kafka BatchTimeoutSec to BatchTimeoutMs to allow millisecond based values
    • producer.Kafka retry count set to 0
    • 0๏ธโƒฃ producer.Kafka default producer set to RoundRobin
    • 0๏ธโƒฃ producer.Kafka GracePeriodMs default set to 100
    • ๐Ÿ“‡ producer.Kafka MetadataRefreshMs default set to 600000 (10 minutes)
    • 0๏ธโƒฃ producer.Kafka TimeoutMs default set to 10000 (10 seconds)
    • filter.RegExp FilterExpressionNot is evaluated before FilterExpression
    • filter.RegExp FilterExpression is evaluated if FilterExpressionNot passed
  • v0.4.2 Changes

    ๐Ÿš€ This is a patch / minor features release.

    ๐Ÿ›  Fixed with 0.4.2

    • consumer.SysLogD now has more meaningful errormessages
    • ๐Ÿ‘ consumer.File now properly supports file rotation if the file to read is a symlink
    • โšก๏ธ Scribe and Kafka metrics are now only updated upon successful send
    • ๐Ÿ›  Fixed an out of bounds panic when producer.File was rotating logfiles without an extension
    • Compression of files after rotation by produer.File now works (again)
    • producer.Kafka now only reconnects if all topics report an error
    • producer.Spool now properly respools long messages
    • producer.Spool will not delete a file if a message in it could not be processed
    • producer.Spool will try to automatically respool files after a restart
    • producer.Spool will rotate non-empty files even if no new messages come in
    • ๐Ÿšš producer.Spool will recreate folders when removed during runtime
    • producer.Spool will drop messages if rotation failes (not reroute)
    • Messages that are spooled twice now retain their original stream
    • ๐Ÿ‘ Better handling of situations where Sarama (Kafka) writes become blocking
    • ๐Ÿ”Œ Plugins now start up as "initializing" not as "dead" preventing dropped messages during startup

    ๐Ÿ†• New with 0.4.2

    • ๐Ÿ†• New formatter format.SplitToJSON to convert CSV data to JSON
    • ๐Ÿ†• New formatter format.ProcessJSON to modify JSON data
    • producer.File can now set permissions for any folders created
    • RPM spec file added
    • producer.File can now add zero padding to rotated file numbering
    • producer.File can now prune logfiles by file age
    • producer.Spool can now be rate limited
    • Dev version ( is now part of the metrics
    • ๐Ÿ†• New AWS Kinesis producer and consumer (thanks @relud)
  • v0.4.1 Changes

    ๐Ÿš€ This is a patch / minor features release

    ๐Ÿ›  Fixed with 0.4.1

    • ๐Ÿ›  InfluxDB JSON and line protocol fixed
    • shared.WaitGroup.WaitFor with duration 0 falls back to shared.WaitGroup.Wait
    • โœ… proper io.EOF handling for shared.BufferedReader and shared.ByteStream
    • HTTP consumer now responds with 200 instead of 203
    • HTTP consumer properly handles EOF
    • โœ… Increased test coverage

    ๐Ÿ†• New with 0.4.1

    • ๐Ÿ‘Œ Support for InfluxDB line protocol
    • ๐Ÿ†• New setting to enable/disable InfluxDB time based database names
    • Introduction of "fuses" (circuit breaker pattern)
    • โž• Added HTTPs support for HTTP consumer
    • โž• Added POST data support to HTTPRequest producer
  • v0.4.0 Changes

    ๐Ÿš€ This release includes several reliability fixes that prevent messages from being lost during shutdown. During this process the startup/shutdown mechanics were changed which introduced a lot of breaking changes. Also included are improvements on the file, socket and scribe producers. ๐ŸŽ Write performance may show a minor increase for some producers.

    ๐Ÿš€ This release contains breaking changes over version 0.3.x. Custom producers and config files may have to be adjusted.

    ๐Ÿ’ฅ Breaking changes with 0.4.0

    • shared.RuntimeType renamed to TypeRegistry
    • core.StreamTypes renamed to StreamRegistry
    • ๐Ÿšš ?ControlLoop callback parameters for command handling moved to callback members
    • ?ControlLoop renamed to ?Loop, where ? can be a combination of Control (handling of control messages), Message (handling of messages) or Ticker (handling of regular callbacks)
    • ๐Ÿ”Œ PluginControlStop is now splitted into PluginControlStopConsumer and PluginControlStopProducer to allow plugins that are producer and consumers.
    • โฑ Producer.Enqueue now takes care of dropping messages and accepts a timeout overwrite value
    • ๐Ÿ”จ MessageBatch has been refactored to store messages instead of preformatted strings. This allows dropping messages from a batch.
    • ๐Ÿšš Message.Drop has been removed, Message.Route can be used instead
    • ๐Ÿšš The LoopBack consumer has been removed. Producers can now drop messages to any stream using DropToStream.
    • ๐Ÿ”Œ Stream plugins are now allowed to only bind to one stream
    • ๐Ÿ“‡ Renamed producer.HttpReq to producer.HTTPRequest
    • ๐Ÿ“‡ Renamed format.StreamMod to format.StreamRoute
    • ๐Ÿ”ง For format.Envelope postfix and prefix configuration keys have been renamed to EnvelopePostifx and EnvelopePrefix
    • Base64Encode and Base64Decode formatter parameters have been renamed to "Base64*"
    • โœ‚ Removed the MessagesPerSecAvg metric
    • Two functions were added to the MessageSource interface to allow blocked/active state query
    • ๐Ÿšš The low resolution timer has been removed

    ๐Ÿ›  Fixed with 0.4.0

    • Messages stored in channels or MessageBatches can now be flushed properly during shutdown
    • Several producers now properly block when their queue is full (messages could be lost before)
    • Producer control commands now have priority over processing messages
    • โœ… Switched to sarama trunk version to get the latest broker connection fixes
    • ๐Ÿ›  Fixed various message loss scenarios in file, kafka and http request producer
    • Kafka producer is now reconnecting upon every error (intermediate fix)
    • StreamRoute formatter now properly works when the separator is a space
    • ๐Ÿ”Œ File, Kafka and HTTPRequest plugins don't hava mandatory values anymore
    • Socket consumer can now reopen a dropped connection
    • Socket consumer can now change access rights on unix domain sockets
    • Socket consumer now closes non-udp connections upon any error
    • ๐Ÿšš Socket consumer can now remove an existing UDS file with the same name if necessary
    • Socket consumer now uses proper connection timeouts
    • Socket consumer now sends special acks on error
    • All net.Dial commands were replaced with net.DialTimeout
    • The makfile now correctly includes the config folder
    • Thie file producer now behaves correctly when directory creation fails
    • ๐Ÿ“Œ Spinning loops are now more CPU friendly
    • ๐Ÿ”Œ Plugins can now be addressed by longer paths, too, e.g. ""
    • ๐ŸŒฒ Log messages that appear during startup are now written to the set log producer, too
    • ๐Ÿ›  Fixed a problem where control messages could cause shutdown to get stucked
    • ๐Ÿ‘ The Kafka producer has been rewritten for better error handling
    • The scribe producer now dynamically modifies the batch size on error
    • The metric server tries to reopen connection every 5 seconds
    • Float metrics are now properly rounded
    • Ticker functions are now restarted after the function is done, preventing double calls
    • No empty messages will be sent during shutdown

    ๐Ÿ†• New with 0.4.0

    • โž• Added a new stream plugin to route messages to one or more other streams
    • The file producer can now delete old files upon rotate (pruning)
    • The file producer can now overwrite files and set file permissions
    • โž• Added metrics for dropped, discarded, filtered and unroutable messages
    • Streams can now overwrite a producer's ChannelTimeoutMs setting (only for this stream)
    • Producers are now shut down in order based on DropStream dependencies
    • Messages now keep a one-step history of their StreamID
    • โž• Added format.StreamRevert to go back to the last used stream (e.g. after a drop)
    • โž• Added producer.Spooling that temporary stores messages to disk before trying them again (e.g. useful for disconnect scenarios)
    • โž• Added a new formatter to prepend stream names
    • โž• Added a new formatter to serialize messages
    • โž• Added a new formatter to convert collectd to InfluxDB (0.8.x and 0.9.x)
    • It is now possible to add a custom string after the version number
    • ๐Ÿ”Œ Plugins compiled from the contrib folder are now listed in the version string
    • All producers can now define a filter applied before formatting
    • โž• Added unittests to check all bundled producer, consumer, format, filter and stream for interface compatibility
    • ๐Ÿ”Œ Plugins can now be registered and queried by a string based ID via core.PluginRegistry
    • โž• Added producer for InfluxDB data (0.8.x and 0.9.x)
    • Kafka, scribe and elastic search producers now have distinct metrics per topic/category/index
    • ๐Ÿ”– Version number is now added to the metrics as in "MMmmpp" (M)ajor (m)inor (p)atch