Creating Amazon OpenSearch Ingestion pipelines

A pipeline is the mechanism that Amazon OpenSearch Ingestion uses to move data from its source (where the data comes from) to its sink (where the data goes). In OpenSearch Ingestion, the sink will always be a single Amazon OpenSearch Service domain, while the source of your data could be clients like Amazon S3, Fluent Bit, or the OpenTelemetry Collector.

For more information, see Pipelines in the OpenSearch documentation.

Prerequisites and required IAM role

To create an OpenSearch Ingestion pipeline, you must have the following resources:

For instructions to create these resources, see the following topics:

Required IAM permissions

OpenSearch Ingestion uses the following IAM permissions for creating pipelines:

For example, the following policy grants permission to create a pipeline:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Resource":"*",
         "Action":[
            "osis:CreatePipeline",
            "osis:ListPipelineBlueprints",
            "osis:ValidatePipeline"
         ]
      },
      {
         "Resource":[
            "arn:aws:iam::your-account-id:role/pipeline-role"
         ],
         "Effect":"Allow",
         "Action":[
            "iam:CreateRole",
            "iam:AttachPolicy",
            "iam:PassRole"
         ]
      }
   ]
}

OpenSearch Ingestion also includes a permission called osis:Ingest, which is required in order to send signed requests to the pipeline using Signature Version 4. For more information, see Creating an ingestion role.

For more information about each permission, see Actions, resources, and condition keys for OpenSearch Ingestion in the Service Authorization Reference.

Specifying the pipeline version

When you create a pipeline using the configuration editor, you must specify the major version of Data Prepper that the pipeline will run. To specify the version, include the version option in your pipeline configuration:

version: "2"
log-pipeline:
  source:
    ...   

When you choose Create, OpenSearch Ingestion determines the latest available minor version of the major version that you specify, and provisions the pipeline with that version. For example, if you specify version: "2", and the latest supported version of Data Prepper is 2.1.1, OpenSearch Ingestion provisions your pipeline with version 2.1.1. We don't publicly display the minor version that your pipeline is running.

In order to upgrade your pipeline when a new major version of Data Prepper is available, edit the pipeline configuration and specify the new version. You can't downgrade a pipeline to an earlier version.

Any time you make a change to your pipeline that initiates a blue/green deployment, OpenSearch Ingestion can upgrade it to the latest minor version of the major version that's currently configured for the pipeline. For more information, see Blue/green deployments for pipeline updates. OpenSearch Ingestion can't change the major version of your pipeline unless you explicitly update the version option within the pipeline configuration.

Specifying the ingestion path

For pull-based sources like OTel trace and OTel metrics, OpenSearch Ingestion requires the additional path option in your source configuration. The path is a string such as /log/ingest, which represents the URI path for ingestion. This path defines the URI that you use to send data to the pipeline.

For example, say you specify the following path for a pipeline with an HTTP source:

When you ingest data into the pipeline, you must specify the following endpoint in your client configuration: https://pipeline-name-abc123.us-west-2.osis.amazonaws.com/my/test_path.

The path must start with a slash (/) and can contain the special characters '-', '_', '.', and '/', as well as the ${pipelineName} placeholder. If you use ${pipelineName} (such as /${pipelineName}/test_path), OpenSearch Ingestion replaces the variable with the name of the associated sub-pipeline.

Creating pipelines

This section describes how to create OpenSearch Ingestion pipelines using the OpenSearch Service console and the AWS CLI.

aws osis create-pipeline \
  --pipeline-name my-pipeline \
  --min-units 4 \
  --max-units 10 \
  --log-publishing-options  IsLoggingEnabled=true,CloudWatchLogDestination={LogGroup="MyLogGroup"} \
  --vpc-options SecurityGroupIds={sg-12345678,sg-9012345},SubnetIds=subnet-1212234567834asdf \
  --pipeline-configuration-body "file://pipeline-config.yaml" \
  --pipeline-role-arn  arn:aws:iam::1234456789012:role/pipeline-role

After your pipeline is successfully created, you can configure your client and start ingesting data into your OpenSearch Service domain. For more information, see Integrating Amazon OpenSearch Ingestion pipelines with other services and applications.

Tracking the status of pipeline creation

You can track the status of a pipeline as OpenSearch Ingestion provisions it and prepares it to ingest data.

aws osis get-pipeline-change-progress \
    --pipeline-name my-pipeline

{
   "ChangeProgressStatuses": {
      "ChangeProgressStages": [ 
         { 
            "Description": "Validating pipeline configuration",
            "LastUpdated": 1.671055851E9,
            "Name": "VALIDATION",
            "Status": "PENDING"
         }
      ],
      "StartTime": 1.671055851E9,
      "Status": "PROCESSING",
      "TotalNumberOfStages": 5
   }
}