Skip to main content

Components

A good ninja gets clued up on its gear.

Core Components​

Every Benthos pipeline has at least one input, an optional buffer, an output and any number of processors:

input:
  kafka:
    addresses: [ TODO ]
    topics: [ foo, bar ]
    consumer_group: foogroup


buffer:
  type: none


pipeline:
  processors:
  - bloblang: |
      message = this
      meta.link_count = links.length()


output:
  aws_s3:
    bucket: TODO
    path: '${! meta("kafka_topic") }/${! json("message.id") }.json'

These are the main components within Benthos and they provide the majority of useful behaviour.

Observability Components​

There are also the observability components http, logger, metrics, and tracing, which allow you to specify how Benthos exposes observability data:

http:
  address: 0.0.0.0:4195
  enabled: true
  debug_endpoints: false


logger:
  prefix: benthos
  level: WARN
  json_format: true


metrics:
  statsd:
    prefix: foo
    address: localhost:8125
    flush_period: 100ms


tracer:
  jaeger:
    agent_address: localhost:6831
    service_name: benthos

Resource Components​

Finally, there are caches and rate limits. These are components that are referenced by core components and can be shared.

input:
  http_client: # This is an input
    url: TODO
    rate_limit: foo_ratelimit # This is a reference to a rate limit


pipeline:
  processors:
    - cache: # This is a processor
        resource: baz_cache # This is a reference to a cache
        operator: add
        key: '${! json("id") }'
        value: "x"
    - bloblang: root = if errored() { deleted() }


rate_limit_resources:
  - label: foo_ratelimit
    local:
      count: 500
      interval: 1s


cache_resources:
  - label: baz_cache
    memcached:
      addresses: [ localhost:11211 ]
      ttl: 60

It's also possible to configure inputs, outputs and processors as resources which allows them to be reused throughout a configuration with the resource input, resource output and resource processor respectively.

For more information about any of these component types check out their sections: