All Versions
287
Latest Version
Avg Release Cycle
16 days
Latest Release
548 days ago
Changelog History
Page 2
Changelog History
Page 2
-
v4.3.0 Changes
June 23, 2022➕ Added
- Timestamp Bloblang methods are now able to emit and process
time.Time
values. - 🆕 New
ts_tz
method for switching the timezone of timestamp values. - 👍 The
elasticsearch
output fieldtype
now supports interpolation functions. - 🗄 The
redis
processor has been reworked to be more generally useful, the oldoperator
andkey
fields are now deprecated in favour of newcommand
andargs_mapping
fields. - Go API: Added component bundle
./public/components/aws
for all AWS components, including aRunLambda
function. - 🆕 New
cached
processor. - 🔌 Go API: New APIs for registering both metrics exporters and open telemetry tracer plugins.
- 🏗 Go API: The stream builder API now supports configuring a tracer, and tracer configuration is now isolated to the stream being executed.
- 🔌 Go API: Plugin components can now access input and output resources.
- 👍 The
redis_streams
output fieldstream
field now supports interpolation functions. - 👍 The
kafka_franz
input and outputs now supportAWS_MSK_IAM
as a SASL mechanism. - 🆕 New
pusher
output. - ✅ Field
input_batches
added to config unit tests for injecting a series of message batches.
🛠 Fixed
- Corrected an issue where Prometheus metrics from batching at the buffer level would be skipped when combined with input/output level batching.
- 📦 Go API: Fixed an issue where running the CLI API without importing a component package would result in template init crashing.
- 🔧 The
http
processor andhttp_client
input and output no longer have default headers as part of their configuration. AContent-Type
header will be added to requests with a default value ofapplication/octet-stream
when a message body is being sent and the configuration has not added one explicitly. - 🌲 Logging in
logfmt
mode withadd_timestamp
enabled now works.
- Timestamp Bloblang methods are now able to emit and process
-
v4.2.0 Changes
June 03, 2022➕ Added
- Field
credentials.from_ec2_role
added to all AWS based components. - 👍 The
mongodb
input now supports aggregation filters by setting the newoperation
field. - 🆕 New
gcp_cloudtrace
tracer. - 🆕 New
slug
bloblang string method. - 👍 The
elasticsearch
output now supports thecreate
action. - Field
tls.root_cas_file
added to thepulsar
input and output. - 📇 The
fallback
output now adds a metadata fieldfallback_error
to messages when shifted. - New bloblang methods
ts_round
,ts_parse
,ts_format
,ts_strptime
,ts_strftime
,ts_unix
andts_unix_nano
. Most are aliases of (now deprecated) time methods withtimestamp_
prefixes. - 🔊 Ability to write logs to a file (with optional rotation) instead of stdout.
🛠 Fixed
- 🐳 The default docker image no longer throws configuration errors when running streams mode without an explicit general config.
- The field
metrics.mapping
now allows environment functions such ashostname
andenv
. - Fixed a lock-up in the
amqp_0_9
output caused when messages sent with theimmediate
ormandatory
flags were rejected. - 🛠 Fixed a race condition upon creating dynamic streams that self-terminate, this was causing panics in cases where the stream finishes immediately.
- Field
-
v4.1.0 Changes
May 11, 2022➕ Added
- 📇 The
nats_jetstream
input now adds headers to messages as metadata. - Field
headers
added to thenats_jetstream
output. - Field
lazy_quotes
added to the CSV input.
🛠 Fixed
- 🛠 Fixed an issue where resource and stream configs imported via wildcard pattern could not be live-reloaded with the watcher (
-w
) flag. - Bloblang comparisons between numerical values (including
match
expression patterns) no longer require coercion into explicit types. - Reintroduced basic metrics from the
twitter
anddiscord
template based inputs. - Prevented a metrics label mismatch when running in streams mode with resources and
prometheus
metrics. - 🌲 Label mismatches with the
prometheus
metric type now log errors and skip the metric without stopping the service. - 🛠 Fixed a case where empty files consumed by the
aws_s3
input would trigger early graceful termination.
- 📇 The
-
v4.0.0 Changes
April 20, 2022🚀 This is a major version release, for more information and guidance on how to migrate please refer to https://benthos.dev/docs/guides/migration/v4.
➕ Added
- In Bloblang it is now possible to reference the
root
of the document being created within a mapping query. - 👍 The
nats_jetstream
input now supports pull consumers. - Field
max_number_of_messages
added to theaws_sqs
input. - Field
file_output_path
added to theprometheus
metrics type. - ✅ Unit test definitions can now specify a label as a
target_processors
value. - 🆕 New connection settings for all sql components.
- 🆕 New experimental
snowflake_put
output. - New experimental
gcp_cloud_storage
cache. - Field
regexp_topics
added to thekafka_franz
input. - 👍 The
hdfs
outputdirectory
field now supports interpolation functions. - 👍 The cli
list
subcommand now supports acue
format. - Field
jwt.headers
added to all HTTP client components. - Output condition
file_json_equals
added to config unit test definitions.
🛠 Fixed
- The
sftp
output no longer opens files in both read and write mode. - The
aws_sqs
input withreset_visibility
set tofalse
will no longer reset timeouts on pending messages during gracefully shutdown. - The
schema_registry_decode
processor now handles AVRO logical types correctly. Details in #1198 and #1161 and also in https://github.com/linkedin/goavro/issues/242.
🔄 Changed
- 🔧 All components, features and configuration fields that were marked as deprecated have been removed.
- 🏗 The
pulsar
input and output are no longer included in the default Benthos builds. - 0️⃣ The field
pipeline.threads
field now defaults to-1
, which automatically matches the host machine CPU count. - 💅 Old style interpolation functions (
${!json:foo,1}
) are removed in favour of the newer Bloblang syntax (${! json("foo") }
). - The Bloblang functions
meta
,root_meta
,error
andenv
now returnnull
when the target value does not exist. - ⚡️ The
clickhouse
SQL driver Data Source Name format parameters have been changed due to a client library update. This also means placeholders insql_raw
components should use dollar syntax. - 🐳 Docker images no longer come with a default config that contains generated environment variables, use
-s
flag arguments instead. - All cache components have had their retry/backoff fields modified for consistency.
- 0️⃣ All cache components that support a general default TTL now have a field
default_ttl
with a duration string, replacing the previous field. - 0️⃣ The
http
processor andhttp_client
output now execute message batch requests as individual requests by default. This behaviour can be disabled by explicitly settingbatch_as_multipart
totrue
. - 0️⃣ Outputs that traditionally wrote empty newlines at the end of batches with >1 message when using the
lines
codec (socket
,stdout
,file
,sftp
) no longer do this by default. - The
switch
output fieldretry_until_success
now defaults tofalse
. - 0️⃣ All AWS components now have a default
region
field that is empty, allowing environment variables or profile values to be used by default. - 0️⃣ Serverless distributions of Benthos (AWS lambda, etc) have had the default output config changed to reject messages when the processing fails, this should make it easier to handle errors from invocation.
- 📄 The standard metrics emitted by Benthos have been largely simplified and improved, for more information check out the metrics page.
- 0️⃣ The default metrics type is now
prometheus
. - The
http_server
metrics type has been renamed tojson_api
. - The
stdout
metrics type has been renamed tologger
. - 🔧 The
logger
configuration section has been simplified, withlogfmt
being the new default format. - 0️⃣ The
logger
fieldadd_timestamp
is nowfalse
by default. - 🚚 Field
parts
has been removed from all processors. - Field
max_in_flight
has been removed from a range of output brokers as it no longer required. - 🚚 The
dedupe
processor now acts upon individual messages by default, and thehash
field has been removed. - 🌲 The
log
processor now executes for each individual message of a batch. - The
sleep
processor now executes for each individual message of a batch. - ✅ The
benthos test
subcommand no longer walks when targetting a directory, instead use triple-dot syntax (./dir/...
) or wildcard patterns. - Go API: Module name has changed to
github.com/benthosdev/benthos/v4
. - 📦 Go API: All packages within the
lib
directory have been removed in favour of the newer APIs withinpublic
. - Go API: Distributed tracing is now via the Open Telemetry client library.
- In Bloblang it is now possible to reference the
-
v3.65.0 Changes
March 07, 2022➕ Added
- 🆕 New
sql_raw
processor and output.
🛠 Fixed
- Corrected a case where nested
parallel
processors that result in emptied batches (all messages filtered) would propagate an unack rather than an acknowledgement.
🔄 Changed
- 🚚 The
sql
processor and output are no longer marked as deprecated and will therefore not be removed in V4. This change was made in order to provide more time to migrate to the newsql_raw
processor and output.
- 🆕 New
-
v3.64.0 Changes
February 23, 2022➕ Added
- Field
nack_reject_patterns
added to theamqp_0_9
input. - 🆕 New experimental
mongodb
input. - 📜 Field
cast
added to thexml
processor andparse_xml
bloblang method. - New experimental
gcp_bigquery_select
processor. - 🆕 New
assign
bloblang method. - 👍 The
protobuf
processor now supportsAny
fields in protobuf definitions. - The
azure_queue_storage
input fieldqueue_name
now supports interpolation functions.
🛠 Fixed
- 🛠 Fixed an issue where manually clearing errors within a
catch
processor would result in subsequent processors in the block being skipped. - The
cassandra
output should now automatically matchfloat
columns. - 🛠 Fixed an issue where the
elasticsearch
output would collapse batched messages of matching ID rather than send as individual items. - 🚚 Running streams mode with
--no-api
no longer removes the/ready
endpoint.
🔄 Changed
- 🗄 The
throttle
processor has now been marked as deprecated.
- Field
-
v3.63.0 Changes
February 08, 2022➕ Added
- 👍 Field
cors
added to thehttp_server
input and output, for supporting CORS requests when custom servers are used. - Field
server_side_encryption
added to theaws_s3
output. - Field
use_histogram_timing
andhistogram_buckets
added to theprometheus
metrics exporter. - 🆕 New duration string and back off field types added to plugin config builders.
- Experimental field
multipart
added to thehttp_client
output. - Codec
regex
added to inputs. - ⏱ Field
timeout
added to thecassandra
output. - New experimental
gcp_bigquery_select
input. - Field
ack_wait
added to thenats_jetstream
input.
🔄 Changed
- 💅 The old map-style resource config fields (
resources.processors.<name>
, etc) are now marked as deprecated. Use the newer list based fields (processor_resources
, etc) instead.
🛠 Fixed
- 👍 The
generate
input now supports zeroed duration strings (0s
, etc) for unbounded document creation. - The
aws_dynamodb_partiql
processor no longer ignores theendpoint
field. - Corrected duplicate detection for custom cache implementations.
- 🛠 Fixed panic caused by invalid bounds in the
range
function. - ✅ Resource config files imported now allow (and ignore) a
tests
field. - 🛠 Fixed an issue where the
aws_kinesis
input would fail to back off during unyielding read attempts. - 🛠 Fixed a linting error with
zmq4
input/outputurls
fields that was incorrectly expecting a string.
- 👍 Field
-
v3.62.0 Changes
January 21, 2022➕ Added
- 🔀 Field
sync
added to thegcp_pubsub
input. - 🆕 New input, processor, and output config field types added to the plugin APIs.
- ➕ Added new experimental
parquet
processor. - 🆕 New Bloblang method
format_json
. - 👍 Field
collection
inmongodb
processor and output now supports interpolation functions. - Field
output_raw
added to thejq
processor. - The lambda distribution now supports a
BENTHOS_CONFIG_PATH
environment variable for specifying a custom config path. - 📇 Field
metadata
added tohttp
andhttp_client
components. - Field
ordering_key
added to thegcp_pubsub
output. - A suite of new experimental
geoip_
methods have been added. - ➕ Added flag
--deprecated
to thebenthos lint
subcommand for detecting deprecated fields.
🔄 Changed
- The
sql
processor and output have been marked deprecated in favour of the newersql_insert
,sql_select
alternatives.
🛠 Fixed
- The input codec
chunked
is no longer capped by the packet size of the incoming streams. - The
schema_registry_decode
andschema_registry_encode
processors now honour trailing slashes in theurl
field. - 🔧 Processors configured within
pipeline.processors
now share processors across threads rather than clone them. - 🔌 Go API: Errors returned from input/output plugin
Close
methods no longer cause shutdown to block. - 🔧 The
pulsar
output should now follow authentication configuration. - 🛠 Fixed an issue where the
aws_sqs
output might occasionally retry a failed message send with an invalid empty message body.
- 🔀 Field
-
v3.61.0 Changes
December 28, 2021➕ Added
- Field
json_marshal_mode
added to the MongoDB processor. - Fields
extract_headers.include_prefixes
andextract_headers.include_patterns
added to thehttp_client
input and output and to thehttp
processor. - 📇 Fields
sync_response.metadata_headers.include_prefixes
andsync_response.metadata_headers.include_patterns
added to thehttp_server
input. - The
http_client
input and output and thehttp
processor fieldcopy_response_headers
has been deprecated in favour of theextract_headers
functionality. - ➕ Added new cli flag
--no-api
for thestreams
subcommand to disable the REST API. - 🆕 New experimental
kafka_franz
input and output. - ➕ Added new Bloblang function
ksuid
. - 👍 All
codec
input fields now support custom csv delimiters.
🛠 Fixed
- Streams mode paths now resolve glob patterns in all cases.
- 🌲 Prevented the
nats
input from error logging when acknowledgments can't be fulfilled due to the lack of message replies. - 🛠 Fixed an issue where GCP inputs and outputs could terminate requests early due to a cancelled client context.
- 💅 Prevented more parsing errors in Bloblang mappings with windows style line endings.
- Field
-
v3.60.1 Changes
December 03, 2021🛠 Fixed
- 🛠 Fixed an issue where the
mongodb
output would incorrectly report upsert not allowed on valid operators.
- 🛠 Fixed an issue where the