OpenSearch Piped Processing Language (PPL)

fields

fields field1, field2

Displays a set of fields which needs projection.

join

LEFT JOIN left=l, right=r on l.id = r.id `join_right_lg` | fields l.field_1, r.field_2

Joins two datasets together.

where

where field1="success" | where field2 != "i-023fe0a90929d8822" | fields field3, field4, field5,field6 | head 1000

Filters the data based on the conditions that you specify.

aws:fieldIndex

source = [`aws:fieldIndex`="region", `region` = "us-west-2"] | where status = 200 | head 10

Returns indexed data only, by forcing a query to scan only log groups that are indexed on a field that you specify in the query.

stats

stats count(), count(field1), min(field1), max(field1), avg(field1) by field2 | head 1000

Performs aggregations and calculations

parse

parse field1 ".*/(?<field2>[^/]+$)" | where field2 = "requestId" | fields field1, field2 | head 1000

Extracts a regular expression (regex) pattern from a string and displays the extracted pattern. The extracted pattern can be further used to create new fields or filter data.

sort

stats count(), count(field1), min(field1) as field1Alias, max(`field1`), avg(`field1`) by field2 | sort -field1Alias | head 1000

Sort the displayed results by a field name. Use sort -FieldName to sort in descending order.

eval

eval field2 = field1 * 2 | fields field1, field2 | head 20

Modifies or processes the value of a field and stores it in a different field. This is useful to mathematically modify a column, apply string functions to a column, or apply date functions to a column.

rename

rename field2 as field1 | fields field1;

Renames one or more fields in the search result.

head

fields `@message` | head 20

Limits the displayed query results to the first N rows.

top

top 2 field1 by field2

Finds the most frequent values for a field.

dedup

dedup field1 | fields field1, field2, field3

Removes duplicate entries based on the fields that you specify.

rare

rare field1 by field2

Finds the least frequent values of all fields in the field list.

subquery

where field_1 IN [ search source= `subquery_lg` | fields field_2 ] | fields id, field_1

Performs complex, nested queries within your PPL statements.

trendline

trendline sma(2, field1) as field1Alias

Calculates the moving averages of fields.

eventStats

eventstats sum(field1) by field2

Enriches your event data with calculated summary statistics. It analyzes specified fields within your events, computes various statistical measures, and then appends these results to each original event as new fields.

expand

eval tags_array_string = json_extract(`@message`, '$.tags')| eval tags_array = json_array(json_extract(tags_string, '$[0]'), json_extract(tags_string, '$[1]'))| expand tags_array as color_tags

Breaks down a field containing multiple values into separate rows, creating a new row for each value in the specified field.

fillnull

fields `@timestamp`, error_code, status_code | fillnull using status_code = "UNKNOWN", error_code = "UNKNOWN"

Fills null fields with the value that you provide. It can be used in one or more fields.

flatten

eval metadata_struct = json_object('size', json_extract(metadata_string, '$.size'), 'color', json_extract(metadata_string, '$.color')) | flatten metadata_struct as (meta_size, meta_color)

Flattens a field. The field must be of this type: struct<?,?> or array<struct<?,?>>.

cidrmatch

where cidrmatch(ip, '2003:db8::/32') | fields ip

Checks if the specified IP address is within the given CIDR range.

fieldsummary

where field1 != 200 | fieldsummary includefields= field1 nulls=true

Calculates basic statistics for each field (count, distinct count, min, max, avg, stddev, and mean).

grok

grok email '.+@%{HOSTNAME:host}' | fields email, host

Parses a text field with a grok pattern and appends the results to the search result.

String functions

eval field1Len = LENGTH(field1) | fields field1Len

Built-in functions in PPL that can manipulate and transform string and text data within PPL queries. For example, converting case, combining strings, extracting parts, and cleaning text.

Date-Time functions

eval newDate = ADDDATE(DATE('2020-08-26'), 1) | fields newDate

Built-in functions for handling and transforming date and timestamp data in PPL queries. For example, date_add, date_format, datediff, date-sub, timestampadd, timestampdiff, current_timezone, utc_timestamp, and current_date.

Condition functions

eval field2 = isnull(field1) | fields field2, field1, field3

Built-in functions that check for specific field conditions, and evaluate expressions conditionally. For example, if field1 is null, return field2.

Math functions

eval field2 = ACOS(field1) | fields field1

Built-in functions for performing mathematical calculations and transformations in PPL queries. For example, abs (absolute value), round (rounds numbers), sqrt (square root), pow (power calculation), and ceil (rounds up to nearest integer).

CryptoGraphic functions

eval crypto = MD5(field)| head 1000

To calculate the hash of given field

JSON functions

eval valid_json = json('[1,2,3,{"f1":1,"f2":[5,6]},4]') | fields valid_json

Built-in functions for handling JSON including arrays, extracting, and validation. For example, json_object, json_array, to_json_string, json_array_length, json_extract, json_keys, and json_valid.