Skip to main content

grok

Parses messages into a structured format by attempting to apply a list of Grok expressions, the first expression to result in at least one value replaces the original message with a JSON object containing the values.

# Common config fields, showing default values
label: ""
grok:
  expressions: []
  pattern_definitions: {}
  pattern_paths: []

Type hints within patterns are respected, therefore with the pattern %{WORD:first},%{INT:second:int} and a payload of foo,1 the resulting payload would be {"first":"foo","second":1}.

Performance​

This processor currently uses the Go RE2 regular expression engine, which is guaranteed to run in time linear to the size of the input. However, this property often makes it less performant than PCRE based implementations of grok. For more information see https://swtch.com/~rsc/regexp/regexp1.html.

Examples​

Grok can be used to parse unstructured logs such as VPC flow logs that look like this:

2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK

Into structured objects that look like this:

{"accountid":"123456789010","action":"ACCEPT","bytes":4249,"dstaddr":"172.31.16.21","dstport":22,"end":1418530070,"interfaceid":"eni-1235b8ca123456789","logstatus":"OK","packets":20,"protocol":6,"srcaddr":"172.31.16.139","srcport":20641,"start":1418530010,"version":2}

With the following config:

pipeline:
  processors:
    - grok:
        expressions:
          - '%{VPCFLOWLOG}'
        pattern_definitions:
          VPCFLOWLOG: '%{NUMBER:version:int} %{NUMBER:accountid} %{NOTSPACE:interfaceid} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:logstatus}'

Fields​

expressions​

One or more Grok expressions to attempt against incoming messages. The first expression to match at least one value will be used to form a result.

Type: array
Default: []

pattern_definitions​

A map of pattern definitions that can be referenced within patterns.

Type: object
Default: {}

pattern_paths​

A list of paths to load Grok patterns from. This field supports wildcards, including super globs (double star).

Type: array
Default: []

named_captures_only​

Whether to only capture values from named patterns.

Type: bool
Default: true

use_default_patterns​

Whether to use a default set of patterns.

Type: bool
Default: true

remove_empty_values​

Whether to remove values that are empty from the resulting structure.

Type: bool
Default: true

parts​

An optional array of message indexes of a batch that the processor should apply to. If left empty all messages are processed. This field is only applicable when batching messages at the input level.

Indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1.

Type: array
Default: []

Default Patterns​

A summary of the default patterns on offer can be found here.