Skip to main content

process_map

DEPRECATED

This component is deprecated and will be removed in the next major version release. Please consider moving onto alternative components.

A processor that extracts and maps fields identified via dot path from the original payload into a new object, applies a list of processors to the newly constructed object, and finally maps the result back into the original payload.

# Common config fields, showing default values
label: ""
process_map:
  conditions: []
  premap: {}
  premap_optional: {}
  processors: []
  postmap: {}
  postmap_optional: {}

Alternatives​

All functionality of this processor has been superseded by the branch processor.

This processor is useful for performing processors on subsections of a payload. For example, you could extract sections of a JSON object in order to construct a reduced request object for an http processor, then map the result back into a field within the original object.

The order of stages of this processor are as follows:

  • Conditions are tested (if specified) against each message, messages that do not pass will not be processed.
  • Messages that are flagged for processing are mapped according to the premap fields, creating a new object. If the premap stage fails (targets are not found) the message will not be processed.
  • Messages that are mapped are processed as a batch.
  • After all child processors are applied to the mapped messages they are mapped back into the original messages they originated from following the postmap fields. If the postmap stage fails the mapping is skipped and the message payload remains as it started.

If the premap is empty then the full payload is sent to the processors, if the postmap is empty then the processed result replaces the original contents entirely.

Batch Ordering​

This processor supports batched messages, but the list of processors to apply must NOT change the ordering (or count) of the messages (do not use a group_by processor, for example).

Error Handling​

When premap, processing or postmap stages fail the underlying message will remain unchanged, the errors are logged, and the message is flagged as having failed, allowing you to use standard processor error handling patterns for recovery.

Fields​

conditions​

A list of conditions to test against messages. If any condition fails then the message will not be mapped and processed.

Type: array
Default: [] 

# Examples


conditions:
  - bloblang: document.urls.length() > 0

premap​

A map of source to destination paths used to create a new object from the original. An empty (or dot .) path indicates the root of the object. If a map source is not found then the message will not be processed, for optional sources use the field premap_optional.

Type: object
Default: {} 

# Examples


premap:
  .: field.from.document


premap:
  bar.baz: root.extra.baz
  foo: root.body.foo

premap_optional​

A map of optional source to destination paths used to create a new object from the original.

Type: object
Default: {}

processors​

A list of processors to apply to mapped payloads.

Type: array
Default: []

postmap​

A map of destination to source paths used to map results from processing back into the original payload. An empty (or dot .) path indicates the root of the object. If a source is not found then the mapping is abandoned, for optional sources use the postmap_optional field.

Type: object
Default: {} 

# Examples


postmap:
  results.foo: .

postmap_optional​

A map of optional destination to source paths used to map results from processing back into the original payload.

Type: object
Default: {}

parts​

An optional array of message indexes of a batch that the processor should apply to. If left empty all messages are processed. This field is only applicable when batching messages at the input level.

Indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1.

Type: array
Default: []

Examples​

Given a message payload of:

{
  "doc": {
    "id": "foo",
    "title": "foo bar baz",
    "description": "here's a thing",
    "content": "this is a body"
  }
}

We might wish to perform language detection on the doc.content field by sending it to a hypothetical HTTP service. We do not wish to overwrite the original document with the result, and instead want to place it within the path doc.language, and so this is a good use case for process_map:

pipeline:
  processors:
    - process_map:
        premap:
          content: doc.content
        processors:
          - http:
              url: http://localhost:1234
        postmap:
          doc.language: .

With the above config we would send our target HTTP service the payload {"content":"this is a body"}, and whatever the service returns will get mapped into our original document:

{
  "doc": {
    "id": "foo",
    "title": "foo bar baz",
    "description": "here's a thing",
    "content": "this is a body",
    "language": {
      "code": "en",
      "certainty": 0.2
    }
  }
}