Apache Airflow CLI command reference
This topic describes the supported and unsupported Apache Airflow CLI commands on Amazon Managed Workflows for Apache Airflow.
Tip
REST API is more modern than the CLI and is designed for programmatic integration with external systems. REST is the preferred way of interacting with Apache Airflow.
Contents
Prerequisites
The following section describes the preliminary steps required to use the commands and scripts on this page.
Access
-
AWS account access in AWS Identity and Access Management (IAM) to the Amazon MWAA permissions policy in Apache Airflow UI access policy: AmazonMWAAWebServerAccess.
-
AWS account access in AWS Identity and Access Management (IAM) to the Amazon MWAA permissions policy Full API and console access policy: AmazonMWAAFullApiAccess.
AWS CLI
The AWS Command Line Interface (AWS CLI) is an open source tool that you can use to interact with AWS services using commands in your command-line shell. To complete the steps on this page, you need the following:
What changed?
-
v3: Airflow architecture. Apache Airflow v3 introduces breaking architectural changes to provide improved security and scalability, and to make maintainance easier. To learn more, refer to Upgrading To Airflow 3
. -
v2: Airflow CLI command structure. The Apache Airflow v2 CLI is organized so that related commands are grouped together as subcommands, which means you need to update Apache Airflow v1 scripts if you want to upgrade to Apache Airflow v2. For example,
unpausein Apache Airflow v1 isdags unpausein Apache Airflow v2. To learn more, refer to Airflow CLI changes in 2.0.
Supported CLI commands
The following section lists the Apache Airflow CLI commands available on Amazon MWAA.
Supported commands
Using commands that parse DAGs
If your environment is running Apache Airflow v2.0.2, CLI commands that parse DAGs will fail if the DAG uses plugins that depend on
packages installed through a requirements.txt:
Apache Airflow v2.0.2
-
dags backfill -
dags list -
dags list-runs -
dags next-execution
You can use these CLI commands if your DAGs do not use plugins that depend on packages installed through a requirements.txt.
Sample code
The following section contains examples of different ways to use the Apache Airflow CLI.
Set, get or delete an Apache Airflow v2 variable
You can use the following sample code to set, get or delete a variable in the format of <script> <mwaa env name> get | set | delete <variable> <variable value> </variable> </variable>.
[ $# -eq 0 ] && echo "Usage: $0 MWAA environment name " && exit if [[ $2 == "" ]]; then dag="variables list" elif [ $2 == "get" ] || [ $2 == "delete" ] || [ $2 == "set" ]; then dag="variables $2 $3 $4 $5" else echo "Not a valid command" exit 1 fi CLI_JSON=$(aws mwaa --region $AWS_REGION create-cli-token --name $1) \ && CLI_TOKEN=$(echo $CLI_JSON | jq -r '.CliToken') \ && WEB_SERVER_HOSTNAME=$(echo $CLI_JSON | jq -r '.WebServerHostname') \ && CLI_RESULTS=$(curl --request POST "https://$WEB_SERVER_HOSTNAME/aws_mwaa/cli" \ --header "Authorization: Bearer $CLI_TOKEN" \ --header "Content-Type: text/plain" \ --data-raw "$dag" ) \ && echo "Output:" \ && echo $CLI_RESULTS | jq -r '.stdout' | base64 --decode \ && echo "Errors:" \ && echo $CLI_RESULTS | jq -r '.stderr' | base64 --decode
Add a configuration when triggering a DAG
You can use the following sample code with Apache Airflow v2 to add a configuration when triggering a DAG, such as airflow trigger_dag 'dag_name' —conf '{"key":"value"}'.
import boto3 import json import requests import base64 mwaa_env_name = 'YOUR_ENVIRONMENT_NAME' dag_name = 'YOUR_DAG_NAME' key = "YOUR_KEY" value = "YOUR_VALUE" conf = "{\"" + key + "\":\"" + value + "\"}" client = boto3.client('mwaa') mwaa_cli_token = client.create_cli_token( Name=mwaa_env_name ) mwaa_auth_token = 'Bearer ' + mwaa_cli_token['CliToken'] mwaa_webserver_hostname = 'https://{0}/aws_mwaa/cli'.format(mwaa_cli_token['WebServerHostname']) raw_data = "trigger_dag {0} -c '{1}'".format(dag_name, conf) mwaa_response = requests.post( mwaa_webserver_hostname, headers={ 'Authorization': mwaa_auth_token, 'Content-Type': 'text/plain' }, data=raw_data ) mwaa_std_err_message = base64.b64decode(mwaa_response.json()['stderr']).decode('utf8') mwaa_std_out_message = base64.b64decode(mwaa_response.json()['stdout']).decode('utf8') print(mwaa_response.status_code) print(mwaa_std_err_message) print(mwaa_std_out_message)
Run CLI commands on an SSH tunnel to a bastion host
Use the following example to run Airflow CLI commands using an SSH tunnel proxy to a Linux Bastion Host.
Using curl
-
ssh -D 8080 -f -C -q -NYOUR_USER@YOUR_BASTION_HOST -
curl -x socks5h://0:8080 --request POST https://YOUR_HOST_NAME/aws_mwaa/cli --headerYOUR_HEADERS--data-rawYOUR_CLI_COMMAND
