archive

Archives all the messages of a batch into a single message according to the selected archive format.

# Config fields, showing default values
label: ""
archive:
  format: binary
  path: ${!count("files")}-${!timestamp_unix_nano()}.txt

Some archive formats (such as tar, zip) treat each archive item (message part) as a file with a path. Since message parts only contain raw data a unique path must be generated for each part. This can be done by using function interpolations on the 'path' field as described here. For types that aren't file based (such as binary) the file field is ignored.

The resulting archived message adopts the metadata of the first message part of the batch.

The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching in this doc.

Fields

format

The archiving format to apply.

Type: string
Default: "binary"
Options: tar, zip, binary, lines, json_array, concatenate.

path

The path to set for each message in the archive (when applicable). This field supports interpolation functions.

Type: string
Default: "${!count(\"files\")}-${!timestamp_unix_nano()}.txt"

# Examples

path: ${!count("files")}-${!timestamp_unix_nano()}.txt

path: ${!meta("kafka_key")}-${!json("id")}.json

Formats

concatenate

Join the raw contents of each message into a single binary message.

tar

Archive messages to a unix standard tape archive.

zip

Archive messages to a zip file.

binary

Archive messages to a binary blob format consisting of:

• Four bytes containing number of messages in the batch (in big endian)
• For each message part:
  • Four bytes containing the length of the message (in big endian)
  • The content of message

lines

Join the raw contents of each message and insert a line break between each one.

json_array

Attempt to parse each message as a JSON document and append the result to an array, which becomes the contents of the resulting message.

Examples

If we had JSON messages in a batch each of the form:

{"doc":{"id":"foo","body":"hello world 1"}}

And we wished to tar archive them, setting their filenames to their respective unique IDs (with the extension .json), our config might look like this:

pipeline:
  processors:
    - archive:
        format: tar
        path: ${!json("doc.id")}.json