Skip to main content

Migrating to Version 4

Benthos has been at major version 3 for more than two years, during which time it has gained a huge amount of functionality without introducing any breaking changes. However, the number of components, APIs and features that have been deprecated in favour of better solutions has grown steadily and the time has finally come to purge them. There are also some areas of functionality that have been improved with breaking changes.

This document outlines the changes made to Benthos since V3 and tips for how to migrate to V4 in places where those changes are significant.

Deprecated Components Removed​

All components, features and configuration fields that were marked as deprecated in the latest release of V3 have been removed in V4. In order to detect deprecated components or fields within your existing configuration files you can run the linter from a later release of V3 Benthos with the --deprecated flag:

benthos lint --deprecated ./configs/*.yaml

This should report all remaining deprecated components. All deprecated components have favoured alternative solutions in V3, so it should be possible to slowly eliminate deprecated aspects of your config using V3 before upgrading.

New Go Module Name​

For users of the Go plugin APIs the import path of this module needs to be updated to github.com/benthosdev/benthos/v4, like so:

import "github.com/benthosdev/benthos/v4/public/service"

It should be pretty quick to update your imports, either using a tool or just running something like:

grep "Jeffail/benthos/v3" . -Rl | xargs -I{} sed -i 's/Jeffail\/benthos\/v3/benthosdev\/benthos\/v4/g' {}

Pipeline Threads Behaviour Change​

https://github.com/Jeffail/benthos/issues/399

In V3 the pipeline.threads field defaults to 1. If this field is explicitly set to 0 it will automatically match the number of CPUs on the host machine. In V4 this will change so that the default value of pipeline.threads is -1, where this value indicates we should match the number of host CPUs. An explicit value of 0 is still considered valid and functionally equivalent to -1.

Old Style Interpolation Functions Removed​

The original style of interpolation functions, where you specify a function name followed by a colon and then any arguments (${!json:foo,1}) has been deprecated (and undocumented) for a while now. What we've had instead is a subset of Bloblang allowing you to use functions directly (${! json("foo").from(1) }), but with the old style still supported for backwards compatibility.

However, supporting the old style means our parsing capabilities are weakened and so it is now removed in order to allow more powerful interpolations in the future.

Bloblang Changes​

https://github.com/Jeffail/benthos/issues/571

The functions meta, root_meta, error and env now return null when the target value does not exist. This is in order to improve consistency across different functions and query types. In cases where a default empty string is preferred you can add .or("") onto the function. In cases where you want to throw an error when the value does not exist you can add .not_null() onto the function.

Root referencing​

It is now possible to reference the root of the document being created within a mapping query, i.e. root.hash = root.string().hash("xxhash64").

Env Var Docker Configuration​

Docker builds will no longer come with a default config that contains generated environment variables. This system doesn't scale at all for complex configuration files and was becoming a challenge to maintain (and also huge). Instead, the new -s flag has been the preferred way to configure Benthos through arguments and will need to be used exclusively in V4.

It's worth noting that this does not prevent you from defining your own env var based configuration and adding that to your docker image. It's entirely possible to copy the config from V3 and have that work, it just won't be present by default any more.

In order to migrate to the -s flag use the path of the fields you're setting instead of the generated environment variables, so:

docker run --rm -p 4195:4195 jeffail/benthos \
    -e "INPUT_TYPE=http_server" \
    -e "OUTPUT_TYPE=kafka" \
    -e "OUTPUT_KAFKA_ADDRESSES=kafka-server:9092" \
    -e "OUTPUT_KAFKA_TOPIC=benthos_topic"

Becomes:

docker run --rm -p 4195:4195 jeffail/benthos \
  -s "input.type=http_server" \
  -s "output.type=kafka" \
  -s "output.kafka.addresses=kafka-server:9092" \
  -s "output.kafka.topic=benthos_topic"

Old Plugin APIs Removed​

Any packages from within the lib directory have been removed. Please use only the APIs within the public directory, the API docs count be found on pkg.go.dev, and examples can be found in the benthos-plugin-example repository. These new APIs can be found in V3 so if you have many components you can migrate them incrementally by sticking with V3 until completion.

Many of the old packages within lib can also still be found within internal, if you're in a pickle you can find some of those APIs and copy/paste them into your own repository.

Caches​

All caches that support retries have had their retry/backoff configuration fields modified in order to be more consistent. The new common format is:

retries:
  initial_interval: 1s
  max_interval: 5s
  max_elapsed_time: 30s

In cases where it might be desirable to disable retries altogether (the ristretto cache) there is also an enabled field.

TTL changes​

Caches that support TTLs have had their ttl fields renamed to default_ttl in order to make it clearer that their purpose is to provide a fallback. All of these values are now duration string types, i.e. a cache with an integer seconds based field with a previous value of 60 should now be defined as 60s.

Field Default Changes​

https://github.com/Jeffail/benthos/issues/392

Lots of fields have had default values removed in cases where they were deemed unlikely to be useful and likely to cause frustration. This specifically applies to any url, urls, address or addresses fields that may have once had a default value containing a common example for the particular service. In most cases this should cause minimal disruption as the field is non-optional and therefore not specifying it explicitly will result in config errors.

However, there are the following exceptions that are worth noting:

The http processor and http_client output no longer create multipart requests by default​

The http processor and http_client output now execute message batch requests as individual requests by default. This behaviour can be disabled by explicitly setting batch_as_multipart to true.

The switch output retry_until_success​

By default the switch output continues retrying switch case outputs until success. This default was sensible at the time as we didn't have a concept of intentionally nacking messages, and therefore a nacked message was likely a recoverable problem and retrying internally means that messages matching multiple cases wouldn't produce duplicates.

However, since then Benthos has evolved and a very common pattern with the switch output is to reject messages that failed during processing using the reject output. But because of the default value of retry_until_success many users end up in a confusing situation where using a reject output results in the pipeline blocking indefinitely until they discover this field.

Therefore the default value of retry_until_success will now be false, which means users that aren't using a reject flow in one of their switch cases, and have a configuration where messages could match multiple cases, should explicitly set this field to true in order to avoid potential duplicates during downstream outages.

AWS region fields​

https://github.com/Jeffail/benthos/issues/696

Any configuration sections containing AWS fields no longer have a default region of eu-west-1. Instead, the field will be empty by default, where unless explicitly set the environment variable AWS_REGION will be used. This will cause problems for users where they expect the region eu-west-1 to be targeted when neither the field nor the environment variable AWS_REGION are set.

Serverless Default Output​

The default output of the serverless distribution of Benthos is now the following config:

output:
  switch:
    retry_until_success: false
    cases:
      - check: errored()
        output:
          reject: "processing failed due to: ${! error() }"
      - output:
          sync_response: {}

This change was made in order to return processing errors directly to the invoker by default.

Metrics Changes​

https://github.com/Jeffail/benthos/issues/1066

The metrics produced by a Benthos stream have been greatly simplified and now make better use of labels/tags in order to provide component-specific insights. The configuration and behaviour of metrics types has also been made more consistent, with metric names being the same throughout and mapping now being a general top-level field.

For a full overview of the new system check out the metrics about page.

The http_server type renamed to json_api​

The name given to the generic JSON API metrics type was http_server, which was confusing as it isn't the only metrics output type that presents as an HTTP server endpoint. This type was also only originally intended for local debugging, which the prometheus type is also good for.

In order to distinguish this metrics type by its unique feature, which is that it exposes metrics as a JSON object, it has been renamed to json_api.

The stdout type renamed to logger​

The stdout metrics type now emits metrics using the Benthos logger, and therefore also matches the logger format. As such, it has been renamed to logger in order to reflect that.

No more dots​

In V3 metrics names contained dots in order to represent pseudo-paths of the source component. In V4 all metric names produced by Benthos have been changed to contain only alpha-numeric characters and underscores. It is recommended that any custom metric names produced by your metric processors and custom plugins should match this new format for consistency.

Since dots were invalid characters in Prometheus metric names, in V3 the prometheus metrics type made some automatic modifications to all names before registering them. This rewrite first replaced all - and _ characters to a double underscore (__), and then replaced all . characters with _. This was an ugly work around and has been removed in V4, but means in previous cases where custom metrics containing dots were automatically converted you will instead see error logs reporting that the names were invalid and therefore ignored.

If you wish to retain the old rewrite behaviour you can reproduce it with the new mapping field:

metrics:
  mapping: 'root = this.replace("_", "__").replace("-", "__").replace(".", "_")'
  prometheus: {}

However, it's recommended to change your metric names instead.

Tracing Changes​

https://github.com/Jeffail/benthos/issues/872

Distributed tracing within Benthos is now done via the Open Telemetry client library. Unfortunately, this client library does not support the full breadth of options as we had before. As such, the jaeger tracing type now only supports the const sampling type, and the field service_name has been removed.

This will likely mean tracing output will appear different in this release, and if you were relying on code that extracts and interacts with spans from messages in your custom plugins then it will need to be converted to use the official Open Telemetry APIs.

Logging Changes​

https://github.com/Jeffail/benthos/issues/589

The logger config section has been simplified, the new default set to logfmt, and the classic format removed. The default value of add_timestamp has also been changed to false.

Processor Batch Behaviour Changes​

https://github.com/Jeffail/benthos/issues/408

Some processors that once executed only once per batch have been updated to execute upon each message individually by default. This change has been made because it was felt the individual message case was considerably more common (and intuitive) and that it is possible to satisfy the batch-wide behaviour in other ways that are opt-in, such as by placing the processors within a branch and having your request_map explicit for a single batch_index (i.e. request_map: root = if batch_index() != 0 { deleted() }).

Processor parts field removed​

Many processors previously had a parts field, which allowed you to explicitly list the indexes of a batch to apply the processor to. This field had confusing naming and was rarely used (or even known about). Since that same behaviour can be reproduced by placing the processor within a branch (or switch) all parts fields have been removed.

dedupe​

The dedupe processor has been reworked so that it now acts upon individual messages by default. It's now mandatory to specify a key, and the parts and hash fields have been removed. Instead, specify full-content hashing with interpolation functions in the key field, e.g. ${! content().hash("xxhash64") }.

In order to deduplicate an entire batch it is likely easier to use a cache processor with the add operator:

pipeline:
  processors:
    # Try and add one message to a cache that identifies the whole batch
    - branch:
        request_map: |
          root = if batch_index() == 0 {
            this.id
          } else { deleted() }
        processors:
          - cache:
              operator: add
              key: ${! content() }
              value: t
    # Delete all messages if we failed
    - bloblang: |
        root = if errored().from(0) {
          deleted()
        }

log​

The log processor now executes for every message of batches by default.

sleep​

The sleep processor now executes for every message of batches by default.

Broker Ditto Macro Gone​

The hidden macro ditto for broker configs is now removed. Use the copies field instead. For some edge cases where copies does not satisfy your requirements you may be better served using configuration templates. If all else fails then please reach out and we can look into other solutions.