Understanding scheduled queries concepts

Before creating scheduled queries, understand these key concepts that affect how your queries run and where results are delivered.

IAM role separation

Scheduled queries require two separate IAM roles: one for executing queries and another for delivering results to destinations such as Amazon S3 buckets, Amazon EventBridge event buses, or lookup tables. Understanding why this separation exists helps you configure permissions correctly and use the security and operational benefits it provides.

The two-role architecture divides responsibilities between data access and data delivery. The query execution role accesses your log data and runs queries, while the destination delivery role writes results to your chosen destination. This separation follows the principle of least privilege—each role has only the permissions it needs for its specific function.

This separation provides practical benefits for your operations. From a security perspective, if you need to change where results are delivered, you only modify the destination delivery role without changing the query execution permissions. For compliance and auditing, you can clearly track which role accesses sensitive log data and which role writes to external systems. This makes it easier to demonstrate that your log analysis infrastructure follows security best practices.

Cross-region and cross-account usage

A scheduled query is created in a specific region and runs in that region. However, you can query log groups and deliver results across regions and accounts. You need to set up one or more AWS accounts as monitoring accounts and link them with multiple source accounts. A monitoring account is a central AWS account that can view and interact with observability data generated from source accounts. A source account is an individual AWS account that generates observability data for the resources that reside in it. Source accounts share their observability data with the monitoring account. So you can setup scheduled queries from the monitoring account using the log groups of all linked accounts.

Alternative approach: Use CloudWatch Logs centralization to replicate log data from multiple accounts and regions into a central monitoring account. This allows you to create scheduled queries in a single region that access all your centralized logs, avoiding cross-region queries and simplifying IAM permissions management.

Schedule expressions and timezone handling

The schedule you define determines when your query runs and how often it executes. Choosing the right schedule expression affects when you receive results and how much data you query. Understanding the expression types helps you choose between simplicity and precision.

Cron expressions provide precise control over timing, allowing you to specify exact times, days of the week, or days of the month. Use cron expressions when you need queries to run at specific business hours or align with operational schedules. In the console you can also scheduled queries using easy calendar options.

All scheduled queries run in UTC, regardless of your local timezone or where your AWS resources are located. This is particularly important when you schedule queries for business hours or time-sensitive analysis. For example, if your business operates in US Eastern Time and you want a daily report at 9 AM ET, you need to account for the UTC offset (14:00 UTC during daylight saving time, 13:00 UTC otherwise). Plan your schedule expressions with UTC in mind to ensure queries run at the intended times.

Choosing a query language

Scheduled queries support three different query languages, and your choice affects both how you write queries and how easily your team can maintain them. The right language depends on your analysis requirements and your team's existing skills.

If you are primarily filtering and aggregating log data, CloudWatch Logs Insights Query Language offers the most straightforward syntax. For complex data transformations where you need to reshape or enrich data through multiple steps, PPL's pipeline approach makes the logic easier to follow. When you need to perform joins or complex aggregations similar to database operations, SQL provides familiar syntax that database-experienced teams can adopt quickly.

Destination selection and use cases

Where you send query results determines what you can do with them. This choice shapes your entire downstream workflow—whether you are building long-term analytics, triggering automated responses, or both. Understanding the strengths of each destination type helps you design the right architecture for your use case.

Amazon S3 destinations are optimized for storage and batch processing. When you need to keep query results for months or years, analyze trends over time, or feed data into analytics platforms, Amazon S3 provides cost-effective storage with unlimited retention. EventBridge destinations are optimized for real-time automation. When query results should trigger immediate actions—like sending alerts, starting workflows, or updating systems—EventBridge delivers results as events that your applications can respond to instantly. By default all query completion events are automatically sent as events to the default event bus, enabling integration with downstream processing systems, Lambda functions, or other event-driven architectures. Results are only published to destinations when query is executed successfully. Lookup table destinations are optimized for keeping reference data current. A lookup table destination automatically populates or refreshes the specified lookup table with the query results on each scheduled execution, so other queries can reference the latest data with the lookup command.

Query result format and structure

Scheduled queries deliver results in JSON format, but each destination type receives a different payload. For a lookup table destination, the query results become the content of the lookup table, and each run replaces that content. For more information, see Configuring lookup table destinations for scheduled queries.

Amazon S3 destinations receive the result rows of the query. Each object contains a JSON array with one entry for each row in the result set, and each entry maps the output field names of the query to their values. The object does not contain query metadata or query statistics, and it omits the @ptr field even if the query requests it, because that field is usable only in the console.

EventBridge destinations receive query metadata, including the query statistics, but no result rows. To retrieve the rows of a completed query, call GetQueryResults with the value of queryId from the event.

The following example shows the event that CloudWatch Logs publishes to EventBridge when a scheduled query completes.

{
    "version": "0",
    "id": "be72061b-eca2-e068-a7e1-83e01d6fe807",
    "detail-type": "Scheduled Query Completed",
    "source": "aws.logs",
    "account": "123456789012",
    "time": "2025-11-18T11:31:48Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:logs:us-east-1:123456789012:scheduled-query:477b4380-b098-474e-9c5e-e10a8cc2e6e7"
    ],
    "detail": {
        "queryId": "2038fd57-ab4f-4018-bb2f-61d363f4a004",
        "queryString": "fields @timestamp, @message, @logStream\n| filter @message like /ERROR/\n| sort @timestamp desc\n| limit 10000",
        "logGroupIdentifiers": [
            "/aws/lambda/my-function"
        ],
        "status": "Complete",
        "startTime": 1763465460,
        "statistics": {
            "recordsMatched": 1842,
            "recordsScanned": 48325,
            "estimatedRecordsSkipped": 0,
            "bytesScanned": 12081250,
            "estimatedBytesSkipped": 0,
            "logGroupsScanned": 1,
            "resultCount": 1842
        }
    }
}

This query does not aggregate, and it returned fewer rows than its limit of 10,000, so each of the 1,842 matching log events became one output row and recordsMatched and resultCount are equal. A query that aggregates, or one whose result set is truncated by limit, produces a resultCount lower than recordsMatched. For more information, see Understanding recordsScanned, recordsMatched, and resultCount.

Key elements include:

The following table describes each field in the statistics object. For the API definitions of these fields, see QueryStatistics.

Understanding recordsScanned, recordsMatched, and resultCount

Three of the query statistics count different things, and comparing them directly can be misleading. Each one measures a different stage of query processing:

Each stage narrows the data. A command such as stats combines many log events into a single output row, so resultCount can be much smaller than recordsMatched. A large recordsMatched with a small resultCount does not mean that rows are missing from the result set.

Example – Aggregation

The following query counts the error messages in each log stream of one log group over a one-hour period.

filter @message like /ERROR/
| stats count(*) as errorCount by @logStream

If the query reads 1,500,000 log events, 24,318 of them contain ERROR, and those matching log events come from 12 log streams, then the query completes with the following statistics.

"statistics": {
    "recordsMatched": 24318,
    "recordsScanned": 1500000,
    "estimatedRecordsSkipped": 0,
    "bytesScanned": 450000000,
    "estimatedBytesSkipped": 0,
    "logGroupsScanned": 1,
    "resultCount": 12
}

stats produces one row for each log stream, so resultCount is 12 while recordsMatched is 24,318. The 12 values of errorCount add up to 24,318.

Example – Post-aggregation filter

The following query keeps only the log streams that produced more than 1,000 errors.

filter @message like /ERROR/
| stats count(*) as errorCount by @logStream
| filter errorCount > 1000

The final filter command runs after grouping, so it removes rows from the result set instead of log events from the scan. If 3 of the 12 log streams have an errorCount greater than 1,000, then resultCount is 3 instead of 12. recordsScanned and bytesScanned do not change, because the query reads the same log data either way.

Example – Limit

A limit command reduces resultCount without any aggregation. The following query returns the 100 most recent error messages.

filter @message like /ERROR/
| sort @timestamp desc
| limit 100

If the query scans the same 1,500,000 log events and matches the same 24,318, then resultCount is 100, because limit caps the result set at 100 rows. In the console, you see this relationship as Showing 100 of 24,318 records matched.

Each statistic answers a different question.