Benthos changelog

Changelog History

Page 2

• v4.3.0 Changes

  June 23, 2022

  ➕ Added

  • Timestamp Bloblang methods are now able to emit and process time.Time values.
  • 🆕 New ts_tz method for switching the timezone of timestamp values.
  • 👍 The elasticsearch output field type now supports interpolation functions.
  • 🗄 The redis processor has been reworked to be more generally useful, the old operator and key fields are now deprecated in favour of new command and args_mapping fields.
  • Go API: Added component bundle ./public/components/aws for all AWS components, including a RunLambda function.
  • 🆕 New cached processor.
  • 🔌 Go API: New APIs for registering both metrics exporters and open telemetry tracer plugins.
  • 🏗 Go API: The stream builder API now supports configuring a tracer, and tracer configuration is now isolated to the stream being executed.
  • 🔌 Go API: Plugin components can now access input and output resources.
  • 👍 The redis_streams output field stream field now supports interpolation functions.
  • 👍 The kafka_franz input and outputs now support AWS_MSK_IAM as a SASL mechanism.
  • 🆕 New pusher output.
  • ✅ Field input_batches added to config unit tests for injecting a series of message batches.

  🛠 Fixed

  • Corrected an issue where Prometheus metrics from batching at the buffer level would be skipped when combined with input/output level batching.
  • 📦 Go API: Fixed an issue where running the CLI API without importing a component package would result in template init crashing.
  • 🔧 The http processor and http_client input and output no longer have default headers as part of their configuration. A Content-Type header will be added to requests with a default value of application/octet-stream when a message body is being sent and the configuration has not added one explicitly.
  • 🌲 Logging in logfmt mode with add_timestamp enabled now works.

• v4.2.0 Changes

  June 03, 2022

  ➕ Added

  • Field credentials.from_ec2_role added to all AWS based components.
  • 👍 The mongodb input now supports aggregation filters by setting the new operation field.
  • 🆕 New gcp_cloudtrace tracer.
  • 🆕 New slug bloblang string method.
  • 👍 The elasticsearch output now supports the create action.
  • Field tls.root_cas_file added to the pulsar input and output.
  • 📇 The fallback output now adds a metadata field fallback_error to messages when shifted.
  • New bloblang methods ts_round, ts_parse, ts_format, ts_strptime, ts_strftime, ts_unix and ts_unix_nano. Most are aliases of (now deprecated) time methods with timestamp_ prefixes.
  • 🔊 Ability to write logs to a file (with optional rotation) instead of stdout.

  🛠 Fixed

  • 🐳 The default docker image no longer throws configuration errors when running streams mode without an explicit general config.
  • The field metrics.mapping now allows environment functions such as hostname and env.
  • Fixed a lock-up in the amqp_0_9 output caused when messages sent with the immediate or mandatory flags were rejected.
  • 🛠 Fixed a race condition upon creating dynamic streams that self-terminate, this was causing panics in cases where the stream finishes immediately.

• v4.1.0 Changes

  May 11, 2022

  ➕ Added

  • 📇 The nats_jetstream input now adds headers to messages as metadata.
  • Field headers added to the nats_jetstream output.
  • Field lazy_quotes added to the CSV input.

  🛠 Fixed

  • 🛠 Fixed an issue where resource and stream configs imported via wildcard pattern could not be live-reloaded with the watcher (-w) flag.
  • Bloblang comparisons between numerical values (including match expression patterns) no longer require coercion into explicit types.
  • Reintroduced basic metrics from the twitter and discord template based inputs.
  • Prevented a metrics label mismatch when running in streams mode with resources and prometheus metrics.
  • 🌲 Label mismatches with the prometheus metric type now log errors and skip the metric without stopping the service.
  • 🛠 Fixed a case where empty files consumed by the aws_s3 input would trigger early graceful termination.

• v4.0.0 Changes

  April 20, 2022

  🚀 This is a major version release, for more information and guidance on how to migrate please refer to https://benthos.dev/docs/guides/migration/v4.

  ➕ Added

  • In Bloblang it is now possible to reference the root of the document being created within a mapping query.
  • 👍 The nats_jetstream input now supports pull consumers.
  • Field max_number_of_messages added to the aws_sqs input.
  • Field file_output_path added to the prometheus metrics type.
  • ✅ Unit test definitions can now specify a label as a target_processors value.
  • 🆕 New connection settings for all sql components.
  • 🆕 New experimental snowflake_put output.
  • New experimental gcp_cloud_storage cache.
  • Field regexp_topics added to the kafka_franz input.
  • 👍 The hdfs output directory field now supports interpolation functions.
  • 👍 The cli list subcommand now supports a cue format.
  • Field jwt.headers added to all HTTP client components.
  • Output condition file_json_equals added to config unit test definitions.

  🛠 Fixed

  • The sftp output no longer opens files in both read and write mode.
  • The aws_sqs input with reset_visibility set to false will no longer reset timeouts on pending messages during gracefully shutdown.
  • The schema_registry_decode processor now handles AVRO logical types correctly. Details in #1198 and #1161 and also in https://github.com/linkedin/goavro/issues/242.

  🔄 Changed

  • 🔧 All components, features and configuration fields that were marked as deprecated have been removed.
  • 🏗 The pulsar input and output are no longer included in the default Benthos builds.
  • 0️⃣ The field pipeline.threads field now defaults to -1, which automatically matches the host machine CPU count.
  • 💅 Old style interpolation functions (${!json:foo,1}) are removed in favour of the newer Bloblang syntax (${! json("foo") }).
  • The Bloblang functions meta, root_meta, error and env now return null when the target value does not exist.
  • ⚡️ The clickhouse SQL driver Data Source Name format parameters have been changed due to a client library update. This also means placeholders in sql_raw components should use dollar syntax.
  • 🐳 Docker images no longer come with a default config that contains generated environment variables, use -s flag arguments instead.
  • All cache components have had their retry/backoff fields modified for consistency.
  • 0️⃣ All cache components that support a general default TTL now have a field default_ttl with a duration string, replacing the previous field.
  • 0️⃣ The http processor and http_client output now execute message batch requests as individual requests by default. This behaviour can be disabled by explicitly setting batch_as_multipart to true.
  • 0️⃣ Outputs that traditionally wrote empty newlines at the end of batches with >1 message when using the lines codec (socket, stdout, file, sftp) no longer do this by default.
  • The switch output field retry_until_success now defaults to false.
  • 0️⃣ All AWS components now have a default region field that is empty, allowing environment variables or profile values to be used by default.
  • 0️⃣ Serverless distributions of Benthos (AWS lambda, etc) have had the default output config changed to reject messages when the processing fails, this should make it easier to handle errors from invocation.
  • 📄 The standard metrics emitted by Benthos have been largely simplified and improved, for more information check out the metrics page.
  • 0️⃣ The default metrics type is now prometheus.
  • The http_server metrics type has been renamed to json_api.
  • The stdout metrics type has been renamed to logger.
  • 🔧 The logger configuration section has been simplified, with logfmt being the new default format.
  • 0️⃣ The logger field add_timestamp is now false by default.
  • 🚚 Field parts has been removed from all processors.
  • Field max_in_flight has been removed from a range of output brokers as it no longer required.
  • 🚚 The dedupe processor now acts upon individual messages by default, and the hash field has been removed.
  • 🌲 The log processor now executes for each individual message of a batch.
  • The sleep processor now executes for each individual message of a batch.
  • ✅ The benthos test subcommand no longer walks when targetting a directory, instead use triple-dot syntax (./dir/...) or wildcard patterns.
  • Go API: Module name has changed to github.com/benthosdev/benthos/v4.
  • 📦 Go API: All packages within the lib directory have been removed in favour of the newer APIs within public.
  • Go API: Distributed tracing is now via the Open Telemetry client library.

• v3.65.0 Changes

  March 07, 2022

  ➕ Added

  • 🆕 New sql_raw processor and output.

  🛠 Fixed

  • Corrected a case where nested parallel processors that result in emptied batches (all messages filtered) would propagate an unack rather than an acknowledgement.

  🔄 Changed

  • 🚚 The sql processor and output are no longer marked as deprecated and will therefore not be removed in V4. This change was made in order to provide more time to migrate to the new sql_raw processor and output.

• v3.64.0 Changes

  February 23, 2022

  ➕ Added

  • Field nack_reject_patterns added to the amqp_0_9 input.
  • 🆕 New experimental mongodb input.
  • 📜 Field cast added to the xml processor and parse_xml bloblang method.
  • New experimental gcp_bigquery_select processor.
  • 🆕 New assign bloblang method.
  • 👍 The protobuf processor now supports Any fields in protobuf definitions.
  • The azure_queue_storage input field queue_name now supports interpolation functions.

  🛠 Fixed

  • 🛠 Fixed an issue where manually clearing errors within a catch processor would result in subsequent processors in the block being skipped.
  • The cassandra output should now automatically match float columns.
  • 🛠 Fixed an issue where the elasticsearch output would collapse batched messages of matching ID rather than send as individual items.
  • 🚚 Running streams mode with --no-api no longer removes the /ready endpoint.

  🔄 Changed

  • 🗄 The throttle processor has now been marked as deprecated.

• v3.63.0 Changes

  February 08, 2022

  ➕ Added

  • 👍 Field cors added to the http_server input and output, for supporting CORS requests when custom servers are used.
  • Field server_side_encryption added to the aws_s3 output.
  • Field use_histogram_timing and histogram_buckets added to the prometheus metrics exporter.
  • 🆕 New duration string and back off field types added to plugin config builders.
  • Experimental field multipart added to the http_client output.
  • Codec regex added to inputs.
  • ⏱ Field timeout added to the cassandra output.
  • New experimental gcp_bigquery_select input.
  • Field ack_wait added to the nats_jetstream input.

  🔄 Changed

  • 💅 The old map-style resource config fields (resources.processors.<name>, etc) are now marked as deprecated. Use the newer list based fields (processor_resources, etc) instead.

  🛠 Fixed

  • 👍 The generate input now supports zeroed duration strings (0s, etc) for unbounded document creation.
  • The aws_dynamodb_partiql processor no longer ignores the endpoint field.
  • Corrected duplicate detection for custom cache implementations.
  • 🛠 Fixed panic caused by invalid bounds in the range function.
  • ✅ Resource config files imported now allow (and ignore) a tests field.
  • 🛠 Fixed an issue where the aws_kinesis input would fail to back off during unyielding read attempts.
  • 🛠 Fixed a linting error with zmq4 input/output urls fields that was incorrectly expecting a string.

• v3.62.0 Changes

  January 21, 2022

  ➕ Added

  • 🔀 Field sync added to the gcp_pubsub input.
  • 🆕 New input, processor, and output config field types added to the plugin APIs.
  • ➕ Added new experimental parquet processor.
  • 🆕 New Bloblang method format_json.
  • 👍 Field collection in mongodb processor and output now supports interpolation functions.
  • Field output_raw added to the jq processor.
  • The lambda distribution now supports a BENTHOS_CONFIG_PATH environment variable for specifying a custom config path.
  • 📇 Field metadata added to http and http_client components.
  • Field ordering_key added to the gcp_pubsub output.
  • A suite of new experimental geoip_ methods have been added.
  • ➕ Added flag --deprecated to the benthos lint subcommand for detecting deprecated fields.

  🔄 Changed

  • The sql processor and output have been marked deprecated in favour of the newer sql_insert, sql_select alternatives.

  🛠 Fixed

  • The input codec chunked is no longer capped by the packet size of the incoming streams.
  • The schema_registry_decode and schema_registry_encode processors now honour trailing slashes in the url field.
  • 🔧 Processors configured within pipeline.processors now share processors across threads rather than clone them.
  • 🔌 Go API: Errors returned from input/output plugin Close methods no longer cause shutdown to block.
  • 🔧 The pulsar output should now follow authentication configuration.
  • 🛠 Fixed an issue where the aws_sqs output might occasionally retry a failed message send with an invalid empty message body.

• v3.61.0 Changes

  December 28, 2021

  ➕ Added

  • Field json_marshal_mode added to the MongoDB processor.
  • Fields extract_headers.include_prefixes and extract_headers.include_patterns added to the http_client input and output and to the http processor.
  • 📇 Fields sync_response.metadata_headers.include_prefixes and sync_response.metadata_headers.include_patterns added to the http_server input.
  • The http_client input and output and the http processor field copy_response_headers has been deprecated in favour of the extract_headers functionality.
  • ➕ Added new cli flag --no-api for the streams subcommand to disable the REST API.
  • 🆕 New experimental kafka_franz input and output.
  • ➕ Added new Bloblang function ksuid.
  • 👍 All codec input fields now support custom csv delimiters.

  🛠 Fixed

  • Streams mode paths now resolve glob patterns in all cases.
  • 🌲 Prevented the nats input from error logging when acknowledgments can't be fulfilled due to the lack of message replies.
  • 🛠 Fixed an issue where GCP inputs and outputs could terminate requests early due to a cancelled client context.
  • 💅 Prevented more parsing errors in Bloblang mappings with windows style line endings.

• v3.60.1 Changes

  December 03, 2021

  🛠 Fixed

  • 🛠 Fixed an issue where the mongodb output would incorrectly report upsert not allowed on valid operators.

• « First
• ‹ Prev
• 1
• 2
• 3
• 4
• 5
• 6
• Next ›
• Last »