Sending logs to CloudWatch Logs using HLC endpoint

Amazon CloudWatch Logs supports an HTTP Log Collector (HLC) endpoint that allows you to send logs directly to CloudWatch Logs using a simple HTTP-based protocol. This feature simplifies log ingestion for applications and services without requiring AWS SDK integration.

The HLC endpoint feature enables you to:

Prerequisites

Before you can send logs using the HLC endpoint, you need to:

Option 1: Simplified setup using the AWS console (recommended)

The AWS Management Console provides a streamlined workflow to generate API keys for HLC endpoint access.

Permissions included: The automatically created IAM user will have the following permissions:

Option 2: Manual setup

If you prefer more control over the IAM configuration or need to customize permissions, you can set up the HLC endpoint manually.

Step 1: Create an IAM user

Create an IAM user that will be used for log ingestion:

Step 2: Generate service-specific credentials (API key)

Generate the CloudWatch Logs API key using the CreateServiceSpecificCredential API. You can also use the create-service-specific-credential CLI command. For the credential age, you can specify a value between 1–36600 days. If you don't specify a credential age, the API key will not expire.

To generate an API key with an expiration of 30 days:

aws iam create-service-specific-credential \
    --user-name cloudwatch-logs-hlc-user \
    --service-name logs.amazonaws.com \
    --credential-age-days 30

The response is a ServiceSpecificCredential object. The ServiceCredentialSecret value is your CloudWatch Logs API key (bearer token).

Step 3: Create log group and log stream

Create the log group and log stream where your logs will be stored:

# Create the log group
aws logs create-log-group \
    --log-group-name /aws/hlc-logs/my-application \
    --region us-east-1

# Create the log stream
aws logs create-log-stream \
    --log-group-name /aws/hlc-logs/my-application \
    --log-stream-name application-stream-001 \
    --region us-east-1

Step 4: Enable bearer token authentication

Enable bearer token authentication on the log group:

aws logs put-bearer-token-authentication \
    --log-group-identifier /aws/hlc-logs/my-application \
    --bearer-token-authentication-enabled \
    --region us-east-1

Verify the configuration:

aws logs describe-log-groups \
    --log-group-name-prefix /aws/hlc-logs/my-application \
    --region us-east-1

Sending logs to the HLC endpoint

Endpoint format

The HLC endpoint URL follows this format:

https://logs.<region>.amazonaws.com/services/collector/event?logGroup=<name>&logStream=<name>[&entityName=<name>&entityEnvironment=<environment>]

Required parameters:

Optional parameters:

You can optionally associate your log events with a Service entity by including the following query parameters. Because logs sent through the HLC endpoint are custom telemetry, they are not automatically associated with an entity. By providing these parameters, CloudWatch Logs creates an entity with KeyAttributes.Type set to Service and associates it with your log events. This enables the Explore related feature in CloudWatch to correlate these logs with other telemetry (metrics, traces, and logs) from the same service, making it easier to troubleshoot and monitor your applications across different signal types. For more information about entities and related telemetry, see Adding related information to custom telemetry.

Request format

Send logs using HTTP POST with the following headers and body:

Headers:

Body format:

The request body should be in JSON format with an array of events:

{
    "event": [
        {
            "time": 1730141374.001,
            "event": "Application started successfully",
            "host": "web-server-1",
            "source": "application.log",
            "severity": "info"
        },
        {
            "time": 1730141374.457,
            "event": "User login successful",
            "host": "web-server-1",
            "source": "auth.log",
            "user": "john.doe"
        }
    ]
}

Field descriptions:

Additional custom fields can be included as needed.

Example request

curl -X POST \
    'https://logs.<region>.amazonaws.com/services/collector/event?logGroup=/aws/hlc-logs/my-application&logStream=application-stream-001' \
    -H "Authorization: Bearer <your-bearer-token>" \
    -H "Content-Type: application/json" \
    -d '{
        "event": [
            {
                "time": 1730141374.001,
                "event": "Application started",
                "host": "web-server-1",
                "severity": "info"
            }
        ]
    }'

Control permissions for generating and using CloudWatch Logs API keys

The generation and usage of CloudWatch Logs API keys is controlled by actions and condition keys in both the CloudWatch Logs and IAM services.

Controlling the generation of CloudWatch Logs API keys

The iam:CreateServiceSpecificCredential action controls the generation of a service-specific key (such as a CloudWatch Logs API key). You can scope this action to IAM users as a resource to limit the users for which a key can be generated.

You can use the following condition keys to impose conditions on the permission for the iam:CreateServiceSpecificCredential action:

Controlling the usage of CloudWatch Logs API keys

The logs:CallWithBearerToken action controls the use of a CloudWatch Logs API key. To prevent an identity from using CloudWatch Logs API keys, attach a policy that denies the logs:CallWithBearerToken action to the IAM user associated with the key.

Example policies

Prevent an identity from generating and using CloudWatch Logs API keys

{
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
            "Sid": "DenyCWLAPIKeys",
            "Effect": "Deny",
            "Action": [
                "iam:CreateServiceSpecificCredential",
                "logs:CallWithBearerToken"
            ],
            "Resource": "*"
        }
    ]
}

Prevent an identity from using CloudWatch Logs API keys

{
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Deny",
            "Action": "logs:CallWithBearerToken",
            "Resource": "*"
        }
    ]
}

Allow the creation of CloudWatch Logs keys only if they expire within 90 days

{
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:CreateServiceSpecificCredential",
            "Resource": "arn:aws:iam::123456789012:user/username",
            "Condition": {
                "StringEquals": {
                    "iam:ServiceSpecificCredentialServiceName": "logs.amazonaws.com"
                },
                "NumericLessThanEquals": {
                    "iam:ServiceSpecificCredentialAgeDays": "90"
                }
            }
        }
    ]
}

Handle compromised CloudWatch Logs API keys

If your API key becomes compromised, you should revoke permissions to use it. You can use the following IAM API operations to manage compromised keys:

Change the status of a CloudWatch Logs API key

To deactivate a key, use the update-service-specific-credential command:

aws iam update-service-specific-credential \
    --user-name cloudwatch-logs-hlc-user \
    --service-specific-credential-id ACCA1234EXAMPLE1234 \
    --status Inactive

To reactivate the key, change the status to Active.

Reset a CloudWatch Logs API key

If the value of your key has been compromised or you no longer have it, reset it using the reset-service-specific-credential command. The key must not have expired yet. If it's already expired, delete the key and create a new one.

aws iam reset-service-specific-credential \
    --service-specific-credential-id ACCA1234EXAMPLE1234

Delete a CloudWatch Logs API key

If you no longer need a key or it has expired, delete it using the delete-service-specific-credential command:

aws iam delete-service-specific-credential \
    --service-specific-credential-id ACCA1234EXAMPLE1234

Attach IAM policies to remove permissions for using a CloudWatch Logs API key

To prevent an identity from making calls with a CloudWatch Logs API key, attach the following policy to the IAM user associated with the key:

{
    "Version": "2012-10-17",		 	 	 
    "Statement": {
        "Effect": "Deny",
        "Action": "logs:CallWithBearerToken",
        "Resource": "*"
    }
}

Logging API key usage with CloudTrail

You can use AWS CloudTrail to log data events for CloudWatch Logs API key usage. CloudWatch Logs emits AWS::Logs::LogGroupAuthorization data events for CallWithBearerToken calls, enabling you to audit when and how API keys are used to send logs.

To enable CloudTrail logging for CloudWatch Logs API key usage:

Best practices

Batching events

For better performance and efficiency:

Error handling

Implement proper error handling in your application. Common HTTP status codes:

Limitations