filter

Use filter to get log events that match one or more conditions.

Example: Filter log events using one condition

The code snippet shows an example of a query that returns all log events where the value for range is greater than 3000. The query limits the results to 20 log events and sorts the logs events by @timestamp and in descending order.

fields @timestamp, @message
| filter (range>3000)
| sort @timestamp desc
| limit 20 

Example: Filter log events using more than one condition

You can use the keywords and and or to combine more than one condition.

The code snippet shows an example of a query that returns log events where the value for range is greater than 3000 and value for accountId is equal to 123456789012. The query limits the results to 20 log events and sorts the logs events by @timestamp and in descending order.

fields @timestamp, @message
| filter (range>3000 and accountId=123456789012)
| sort @timestamp desc
| limit 20

Indexed fields and the filter command

If you have created field indexes for a log group, you can leverage those field indexes to make your filter queries more efficient and reduce scanned volume. For example, suppose you have created a field index for requestId. Then, any CloudWatch Logs Insights query on that log group that includes filter requestId = value or filter requestId IN [value, value, ...] will attempt to skip processing log events that are known not to include the indexed field. By attempting to scan only the log events that are known to contain that indexed field, scan volume can be reduced and the query is faster.

For more information about field indexes and how to create them, see Create field indexes to improve query performance and reduce scan volume.

Example: Find log events that are related to a certain request ID, using indexes

This example assumes that you have created a field index on requestId. For log groups that use this field index, the query will leverage field indexes to attempt to scan the least amount of log events to find events with requestId with a value of 123456

fields @timestamp, @message
| filter requestId = "1234656"
| limit 20

Matches and regular expressions in the filter command

The filter command supports the use of regular expressions. You can use the following comparison operators (=, !=, <, <=, >, >=) and Boolean operators (and, or, and not).

You can use the keyword in to test for set membership and check for elements in an array. To check for elements in an array, put the array after in. You can use the Boolean operator not with in. You can create queries that use in to return log events where fields are string matches. The fields must be complete strings. For example, the following code snippet shows a query that uses in to return log events where the field logGroup is the complete string example_group.

fields @timestamp, @message
| filter logGroup in ["example_group"]

You can use the keyword phrases like and not like to match substrings. You can use the regular expression operator =~ to match substrings. To match a substring with like and not like, enclose the substring that you want to match in single or double quotation marks. You can use regular expression patterns with like and not like. To match a substring with the regular expression operator, enclose the substring that you want to match in forward slashes. The following examples contain code snippets that show how you can match substrings using the filter command.

Examples: Match substrings

The following examples return log events where f1 contains the word Exception. All three examples are case sensitive.

The first example matches a substring with like.

fields f1, f2, f3 
| filter f1 like "Exception"

The second example matches a substring with like and a regular expression pattern.

fields f1, f2, f3 
| filter f1 like /Exception/

The third example matches a substring with a regular expression.

fields f1, f2, f3 
| filter f1 =~ /Exception/

Example: Match substrings with wildcards

You can use the period symbol (.) as a wildcard in regular expressions to match substrings. In the following example, the query returns matches where the value for f1 begins with the string ServiceLog.

fields f1, f2, f3
| filter f1 like /ServiceLog./

You can place the asterisk symbol after the period symbol (.*) to create a greedy quantifier that returns as many matches as possible. For example, the following query returns matches where the value for f1 not only begins with the string ServiceLog, but also includes the string ServiceLog.

fields f1, f2, f3
| filter f1 like /ServiceLog.*/

Possible matches can be formatted like the following:

Example: Exclude substrings from matches

The following example shows a query that returns log events where f1 doesn't contain the word Exception. The example is case senstive.

fields f1, f2, f3 
| filter f1 not like "Exception"

Example: Match substrings with case-insensitive patterns

You can match substrings that are case insensitive with like and regular expressions. Place the following parameter (?i) before the substring you want to match. The following example shows a query that returns log events where f1 contains the word Exception or exception.

fields f1, f2, f3 
| filter f1 like /(?i)Exception/