Deploy containerized applications on AWS IoT Greengrass V2 running as a Docker container - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsEpicsTroubleshootingRelated resourcesAdditional information

Deploy containerized applications on AWS IoT Greengrass V2 running as a Docker container

Salih Bakir, Giuseppe Di Bella, and Gustav Svalander, Amazon Web Services

Summary

AWS IoT Greengrass Version 2, when deployed as a Docker container, doesn't natively support running Docker application containers. This pattern shows you how to create a custom container image based on the latest version of AWS IoT Greengrass V2 that enables Docker-in-Docker (DinD) functionality. With DinD, you can run containerized applications within the AWS IoT Greengrass V2 environment.

You can deploy this pattern as a stand-alone solution or integrate it with container orchestration platforms like Amazon ECS Anywhere. In either deployment model, you maintain full AWS IoT Greengrass V2 functionality including AWS IoT SiteWise Edge processing capabilities, while enabling scalable container-based deployments.

Prerequisites and limitations

Prerequisites 

  • An active AWS account.

  • For general AWS IoT Greengrass Version 2 prerequisites, see Prerequisites in the AWS IoT Greengrass Version 2 documentation.

  • Docker Engine, installed and configured on Linux, macOS, or Windows.

  • Docker Compose (if you use the Docker Compose command line interface (CLI) to run Docker images).

  • A Linux operating system.

  • A hypervisor with a host server that supports virtualization.

  • System requirements:

    • 2 GB of RAM (minimum)

    • 5 GB of available disk space (minimum)

    • For AWS IoT SiteWise Edge, an x86_64 quad-core CPU with 16 GB of RAM and 50 GB of available disk space. For more information about AWS IoT SiteWise data processing, see Data processing pack requirements in the AWS IoT SiteWise documentation.

Product versions

  • AWS IoT Greengrass Version 2 version 2.5.3 or later

  • Docker-in-Docker version 1.0.0 or later

  • Docker Compose version 1.22 or later

  • Docker Engine version 20.10.12 or later

Limitations

Architecture

Target technology stack

  • Data sources – IoT devices, sensors, or industrial equipment that generates data for processing

  • AWS IoT Greengrass V2 – Running as a Docker container with D-in-D capabilities, deployed on edge infrastructures

  • Containerized applications – Custom applications running within the AWS IoT Greengrass V2 environment as nested Docker containers

  • (Optional) Amazon ECS Anywhere – Container orchestration that manages the AWS IoT Greengrass V2 container deployment

  • Other AWS services – AWS IoT Core, AWS IoT SiteWise, and other AWS services for data processing and management

Target architecture 

The following diagram shows an example target deployment architecture that uses Amazon ECS Anywhere, which is a container management tool.

Deployment architecture using Amazon ECS Anywhere.

The diagram shows the following workflow:

1: Container image storage – Amazon ECR stores the AWS IoT Greengrass container images and any custom application containers needed for edge processing.

2 and 3: Container deployment – Amazon ECS Anywhere deploys the AWS IoT Greengrass container image from Amazon ECR to the edge location, managing the container lifecycle and deployment process.

4: Component deployment – The deployed AWS IoT Greengrass core automatically deploys its relevant components based on its configuration. Components include AWS IoT SiteWise Edge and other necessary edge processing components within the containerized environment.

5: Data ingestion – After it’s fully configured, AWS IoT Greengrass begins ingesting telemetry and sensor data from various IoT data sources at the edge location.

6: Data processing and cloud integration – The containerized AWS IoT Greengrass core processes data locally using its deployed components (including AWS IoT SiteWise Edge for industrial data). Then, it sends processed data to AWS Cloud services for further analysis and storage.

Tools

AWS services

Other tools

  • Docker is a set of platform as a service (PaaS) products that use virtualization at the operating-system level to deliver software in containers.

  • Docker Compose is a tool for defining and running multi-container applications.

  • Docker Engine is an open source containerization technology for building and containerizing applications.

Code repository

The code for this pattern is available in the GitHub AWS IoT Greengrass v2 Docker-in-Docker repository.

Epics

TaskDescriptionSkills required

Clone and navigate to the repository.

To clone the repository, use the following command:

git clone https://github.com/aws-samples/aws-iot-greengrass-v2-docker-in-docker.git

To navigate to the docker directory, use the following command:

cd aws-iot-greengrass-v2-docker-in-docker/docker

DevOps engineer, AWS DevOps

Build the Docker image.

To build the Docker image with the default (latest) version, run the following command:

docker build -t x86_64/aws-iot-greengrass:latest .

Or, to build the Docker image with a specific version, run the following command:

docker build --build-arg GREENGRASS_RELEASE_VERSION=2.12.0 -t x86_64/aws-iot-greengrass:2.12.0 .

To verify the build, run the following command:

docker images | grep aws-iot-greengrass

 

AWS DevOps, DevOps engineer, App developer

(Optional) Push to Amazon ECR.

  1. To create the repository, run the following command:

    aws ecr create-repository --repository-name aws-iot-greengrass-dind --region us-east-1

  2. To authenticate, run the following command:

    aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <ACCOUNT-ID>.dkr.ecr.us-east-1.amazonaws.com

  3. To tag and push, run the following commands:

    docker tag x86_64/aws-iot-greengrass:latest <ACCOUNT-ID>.dkr.ecr.us-east-1.amazonaws.com/aws-iot-greengrass-dind:latest

    docker push <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/aws-iot-greengrass-dind:latest

App developer, AWS DevOps, DevOps engineer
TaskDescriptionSkills required

Select authentication method.

Choose one of the following options:

  • Option 1 (Recommended): IAM roles – Use if running on Amazon EC2, Amazon ECS, or Amazon EKS with proper IAM roles. No additional configuration needed.

  • Option 2: Environment variables – For testing and development.

  • Option 3: Credentials file – Not recommended for production.

  • Option 4: Legacy env.cfg file.

AWS administrator

Configure authentication method.

For the authentication method you selected, use the following configuration guidance:

  • Option 2 (Environment variables) – Prepare AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN.

  • Option 3 (Credentials file) – Create credentials directory and file:

    • (a) Run the following command:

      mkdir -p ~/greengrass-v2-credentials

    • (b) Create the following file at ~/greengrass-v2-credentials/credentials:

      [default]

      aws_access_key_id=YourAccessKey

      aws_secret_access_key=YourSecretKey

    • (c) Secure the file as follows:

      chmod 600 ~/greengrass-v2-credentials/credentials

  • Option 4 (env.cfg) – Create the env.cfg file in the docker directory as follows:

    GGC_ROOT_PATH=/greengrass/v2

    AWS_REGION=us-east-1

    PROVISION=true

    THING_NAME=MyGreengrassCore

    THING_GROUP_NAME=MyGreengrassCoreGroup

    TES_ROLE_NAME=GreengrassV2TokenExchangeRole

    TES_ROLE_ALIAS_NAME=GreengrassCoreTokenExchangeRoleAlias

    COMPONENT_DEFAULT_USER=ggc_user:ggc_group

AWS administrator
TaskDescriptionSkills required

Configure docker-compose.yml.

Update the docker-compose.yml file with environment variables as follows:

  • Key variables: PROVISION, AWS_REGION, THING_NAME, THING_GROUP_NAME, TES_ROLE_NAME, TES_ROLE_ALIAS_NAME

  • For credentials through environment variables, uncomment the following:

    environment:

    - AWS_ACCESS_KEY_ID=YourKey

    - AWS_SECRET_ACCESS_KEY=YourSecret

  • For credentials file, uncomment the following:

    volumes:

    - ~/greengrass-v2-credentials:/root/.aws/:ro

  • For log persistence, add the following:

    volumes:

    - ./logs:/greengrass/v2/logs

DevOps engineer

Start and verify container.

To start in the foreground, run the following command:

docker-compose up --build

Or, to start in the background, run the following command:

docker-compose up --build -d

To verify status, run the following command:

docker-compose ps

To monitor logs, run the following command:

docker-compose logs -f

DevOps engineer
TaskDescriptionSkills required

Run container with Docker CLI.

  • To initiate a basic run without provisioning, run the following command:

    docker run --init --privileged -it --name aws-iot-greengrass x86_64/aws-iot-greengrass:latest

  • To initiate a run with environment variables, run the following command:

    docker run --init --privileged -it --name aws-iot-greengrass -e PROVISION=true -e AWS_REGION=us-east-1 -e AWS_ACCESS_KEY_ID=YourKey -e AWS_SECRET_ACCESS_KEY=YourSecret -e THING_NAME=MyGreengrassCore -e THING_GROUP_NAME=MyGreengrassCoreGroup -p 8883:8883 x86_64/aws-iot-greengrass:latest

  • To initiate a run with a credentials file, run the following command:

    docker run --init --privileged -it --name aws-iot-greengrass -v ~/greengrass-v2-credentials:/root/.aws:ro -e PROVISION=true -e AWS_REGION=us-east-1 -e THING_NAME=MyGreengrassCore -p 8883:8883 x86_64/aws-iot-greengrass:latest

  • To initiate a run with an env.cfg file, run the following command:

    docker run --init --privileged -it --name aws-iot-greengrass -v ~/greengrass-v2-credentials:/root/.aws:ro --env-file env.cfg -p 8883:8883 x86_64/aws-iot-greengrass:latest

  • To initiate a run with persistent logs, run the following command:

    docker run --init --privileged -it --name aws-iot-greengrass -v \$(pwd)/logs:/greengrass/v2/logs x86_64/aws-iot-greengrass:latest

DevOps engineer

Verify container.

  • To check the container’s status, run the following command:

    docker ps | grep aws-iot-greengrass

  • To monitor logs, run the following command:

    docker logs -f aws-iot-greengrass

DevOps engineer
TaskDescriptionSkills required

Deploy applications.

  • To deploy the application container, run the following command:

    docker run -d --name my-app nginx:alpine

  • To verify deployment, run the following command:

    docker ps

  • To access logs, run the following command:

    cd /greengrass/v2/logs && cat greengrass.log

App developer

Access and test Docker-in-Docker.

  1. To open a shell in the container, run the following command:

    docker exec -it aws-iot-greengrass sh

  2. To verify Docker, run the following commands:

    docker --version

    docker ps

  3. To test Docker-in-Docker, run the following command:

    docker run --rm hello-world

DevOps engineer
TaskDescriptionSkills required

Set up Amazon ECS cluster.

  • To create an Amazon ECS cluster, run the following command:

    aws ecs create-cluster --cluster-name greengrass-cluster --region us-east-1

  • To create SSM activation, run the following command:

    aws ssm create-activation --default-instance-name greengrass-edge --iam-role GreengrassECSAnywhereRole --registration-limit 10 --region us-east-1

  • To install SSM Agent on an edge device, run the following command:

    curl -o install.sh https://s3.amazonaws.com/aws-ssm-downloads/latest/linux_amd64/install.sh

    chmod +x install.sh

    ./install.sh activation-code activation-id us-east-1

AWS administrator

Deploy Amazon ECS task.

  • Create greengrass-task-definition.json with task configuration including containerDefinitions with privileged:true, environment variables, and volume mounts for /var/lib/docker and /greengrass/v2.Register task. Run the following command:

    aws ecs register-task-definition --cli-input-json file://greengrass-task-definition.json

  • To create the service, run the following command:

    aws ecs create-service --cluster greengrass-cluster --service-name greengrass-service --task-definition greengrass-dind --desired-count 1 --launch-type EXTERNAL --region us-east-1

  • To verify deployment, run the following command:

    aws ecs describe-services --cluster greengrass-cluster --services greengrass-service --region us-east-1

AWS administrator
TaskDescriptionSkills required

Stop container.

  • To stop the container by using Docker Compose, run the following commands:

    docker-compose stop

    docker-compose down

    docker-compose down -v

  • To stop the container (removes volumes) by using Docker CLI, run the following commands:

    docker stop aws-iot-greengrass

    docker rm aws-iot-greengrass

  • To stop the container by using force remove, run the following command:

    docker rm -f aws-iot-greengrass

DevOps engineer

Troubleshooting

IssueSolution

Container fails to start with permission errors.

  1. To make sure that the container runs with the --privileged flag required for Docker-in-Docker, run the following command:

    docker run --init --privileged -it --name aws-iot-greengrass x86_64/aws-iot-greengrass:latest

  2. In the docker-compose.yml file, set the following:

    privileged: true

Warning

--privileged grants extended privileges to the container.

Provisioning fails with credential errors.

To verify credentials are configured correctly, use the following steps:

  1. Check IAM role:

    aws sts get-caller-identity

  2. Verify environment variables:

    docker exec -it aws-iot-greengrass sh -c 'echo \$AWS_ACCESS_KEY_ID'

  3. Check credentials file:

    docker exec -it aws-iot-greengrass sh -c 'cat /root/.aws/credentials'

Make sure that IAM permissions include iot:CreateThing, iot:CreatePolicy, iot:AttachPolicy, iam:CreateRole, and iam:AttachRolePolicy.

Cannot connect to Docker daemon inside container.

  1. Check the Docker daemon status:

    docker exec -it aws-iot-greengrass sh -c 'ps aux | grep dockerd'

  2. Verify that the Docker socket exists:

    docker exec -it aws-iot-greengrass sh -c 'ls -la /var/run/docker.sock'

  3. Manually start if needed:

    docker exec -it aws-iot-greengrass sh -c 'dockerd > /var/log/docker.log 2>&1 &'

Container runs out of disk space.

  1. Check disk space:

    df -h

  2. Clean Docker resources:

    docker system prune -a --volumes

  3. Remove old Greengrass artifacts:

    docker exec -it aws-iot-greengrass sh -c 'rm -rf /greengrass/v2/work/*'

Ensure minimum disk space: 5 GB for basic operations and 50 GB for AWS IoT SiteWise Edge

Build issues.

  1. Make sure that you’re in the docker directory:

    cd docker && pwd

  2. Verify that Dockerfile exists:

    ls -la Dockerfile

  3. Check that Docker is installed and running:

    docker --version && docker ps

Network connectivity issues.

  1. Test DNS resolution:

    docker exec -it aws-iot-greengrass sh -c 'nslookup google.com'

  2. Test AWS connectivity:

    docker exec -it aws-iot-greengrass sh -c 'curl https://iot.us-east-1.amazonaws.com'

Verify that the firewall allows outbound HTTPS (443) and MQTT (8883) traffic.

Greengrass components fail to deploy.

  1. Check Greengrass logs:

    docker exec -it aws-iot-greengrass sh -c 'cat /greengrass/v2/logs/greengrass.log'

  2. Verify TES role configuration:

    aws iot describe-role-alias --role-alias TES_ROLE_ALIAS_NAME

Check component-specific logs in the /greengrass/v2/logs/ directory.

Container exits immediately after starting.

  1. Check container logs for errors:

    docker logs aws-iot-greengrass

  2. Run with interactive shell for debugging:

    docker run --init --privileged -it --entrypoint sh x86_64/aws-iot-greengrass:latest

Verify all required environment variables are set correctly if PROVISION=true. Make sure that the --init flag is used when starting the container.

Related resources

AWS resources

Other resources

Additional information

  • For AWS IoT SiteWise Edge data processing, Docker must be available within the AWS IoT Greengrass environment.

  • To run a nested container, you must run the AWS IoT Greengrass container with administrator-level credentials.