cache
Performs operations against a cache resource for each message, allowing you to store or retrieve data within message payloads.
- Common
- Advanced
# Common config fields, showing default valueslabel: ""cache:resource: ""operator: setkey: ""value: ""
# All config fields, showing default valueslabel: ""cache:resource: ""operator: setkey: ""value: ""ttl: ""parts: []
This processor will interpolate functions within the key
and value
fields individually for each message. This allows you to specify dynamic keys and values based on the contents of the message payloads and metadata. You can find a list of functions here.
Examples​
- Deduplication
- Hydration
Deduplication can be done using the add operator with a key extracted from the
message payload, since it fails when a key already exists we can remove the
duplicates using a
bloblang
processor:
pipeline:processors:- cache:resource: foocacheoperator: addkey: '${! json("message.id") }'value: "storeme"- bloblang: root = if errored() { deleted() }cache_resources:- label: foocacheredis:url: tcp://TODO:6379
It's possible to enrich payloads with content previously stored in a cache by
using the branch
processor:
pipeline:processors:- branch:processors:- cache:resource: foocacheoperator: getkey: '${! json("message.document_id") }'result_map: 'root.message.document = this'# NOTE: If the data stored in the cache is not valid JSON then use# something like this instead:# result_map: 'root.message.document = content().string()'cache_resources:- label: foocachememcached:addresses: [ "TODO:11211" ]
Fields​
resource
​
The cache
resource to target with this processor.
Type: string
Default: ""
operator
​
The operation to perform with the cache.
Type: string
Default: "set"
Options: set
, add
, get
, delete
.
key
​
A key to use with the cache. This field supports interpolation functions.
Type: string
Default: ""
value
​
A value to use with the cache (when applicable). This field supports interpolation functions.
Type: string
Default: ""
ttl
​
The TTL of each individual item as a duration string. After this period an item will be eligible for removal during the next compaction. Not all caches support per-key TTLs, and those that do not will fall back to their generally configured TTL setting. This field supports interpolation functions.
Type: string
Default: ""
Requires version 3.33.0 or newer
# Examplesttl: 60sttl: 5mttl: 36h
parts
​
An optional array of message indexes of a batch that the processor should apply to. If left empty all messages are processed. This field is only applicable when batching messages at the input level.
Indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1.
Type: array
Default: []
Operators​
set
​
Set a key in the cache to a value. If the key already exists the contents are overridden.
add
​
Set a key in the cache to a value. If the key already exists the action fails with a 'key already exists' error, which can be detected with processor error handling.
get
​
Retrieve the contents of a cached key and replace the original message payload with the result. If the key does not exist the action fails with an error, which can be detected with processor error handling.
delete
​
Delete a key and its contents from the cache. If the key does not exist the action is a no-op and will not fail with an error.