# Working with DAGs on Amazon MWAA To run Directed Acyclic Graphs (DAGs) on an Amazon Managed Workflows for Apache Airflow environment, you copy your files to the Amazon S3 storage bucket attached to your environment, then let Amazon MWAA know where your DAGs and supporting files are located on the Amazon MWAA console. Amazon MWAA takes care of synchronizing the DAGs among workers, schedulers, and the webserver. This guide describes how to add or update your DAGs and install custom plugins and Python dependencies on an Amazon MWAA environment. **Topics** + [Amazon S3 bucket overview](#working-dags-s3-about) + [Adding or updating DAGs](configuring-dag-folder.md) + [Installing custom plugins](configuring-dag-import-plugins.md) + [Installing Python dependencies](working-dags-dependencies.md) + [Deleting files on Amazon S3](working-dags-delete.md) ## Amazon S3 bucket overview An Amazon S3 bucket for an Amazon MWAA environment must have *Public Access Blocked*. By default, all Amazon S3 resources—buckets, objects, and related sub-resources (for example, lifecycle configuration)—are private. + Only the resource owner, the AWS account that created the bucket, can access the resource. The resource owner (for example, your administrator) can grant access permissions to others by writing an access control policy. + The access policy you set up must have permission to add DAGs, custom plugins in `plugins.zip`, and Python dependencies in `requirements.txt` to your Amazon S3 bucket. For an example policy that contains the required permissions, refer to [AmazonMWAAFullConsoleAccess](access-policies.md#console-full-access). An Amazon S3 bucket for an Amazon MWAA environment must have *Versioning Enabled*. When Amazon S3 bucket versioning is enabled, anytime a new version is created, a new copy is created. + Versioning is enabled for the custom plugins in a `plugins.zip`, and Python dependencies in a `requirements.txt` on your Amazon S3 bucket. + You must specify the version of a `plugins.zip`, and `requirements.txt` on the Amazon MWAA console each time these files are updated on your Amazon S3 bucket. # Adding or updating DAGs Directed Acyclic Graphs (DAGs) are defined within a Python file that defines the DAG's structure as code. You can use the AWS CLI or the Amazon S3 console to upload DAGs to your environment. This topic describes the steps to add or update Apache Airflow DAGs on your Amazon Managed Workflows for Apache Airflow environment using the `dags` folder in your Amazon S3 bucket. **Topics** + [Prerequisites](#configuring-dag-folder-prereqs) + [How it works](#configuring-dag-folder-how) + [What's changed?](#configuring-dag-folder-changed) + [Testing DAGs using the Amazon MWAA CLI utility](#working-dag-folder-cli-utility) + [Uploading DAG code to Amazon S3](#configuring-dag-folder-uploading) + [Specifying the path to your DAGs folder on the Amazon MWAA console (the first time)](#configuring-dag-folder-mwaaconsole) + [Accessing changes on your Apache Airflow UI](#configuring-dag-folder-mwaaconsole-view) + [What's next?](#configuring-dag-folder-next-up) ## Prerequisites You'll need the following before you can complete the steps on this page. + **Permissions** — Your AWS account must have been granted access by your administrator to the [AmazonMWAAFullConsoleAccess](access-policies.md#console-full-access) access control policy for your environment. In addition, your Amazon MWAA environment must be permitted by your [execution role](mwaa-create-role.md) to access the AWS resources used by your environment. + **Access** — If you require access to public repositories to install dependencies directly on the webserver, your environment must be configured with **public network** webserver access. For more information, refer to [Apache Airflow access modes](configuring-networking.md). + **Amazon S3 configuration** — The [Amazon S3 bucket](mwaa-s3-bucket.md) used to store your DAGs, custom plugins in `plugins.zip`, and Python dependencies in `requirements.txt` must be configured with *Public Access Blocked* and *Versioning Enabled*. ## How it works A Directed Acyclic Graph (DAG) is defined within a single Python file that defines the DAG's structure as code. It consists of the following: + A [DAG](https://airflow.apache.org/docs/stable/concepts.html#dags) definition. + [Operators](https://airflow.apache.org/concepts.html#operators) that describe how to run the DAG and the [tasks](https://airflow.apache.org/docs/stable/concepts.html#tasks) to run. + [Operator relationships](https://airflow.apache.org/concepts.html#bitshift-composition) that describe the order in which to run the tasks. To run an Apache Airflow platform on an Amazon MWAA environment, you need to copy your DAG definition to the `dags` folder in your storage bucket. For example, the DAG folder in your storage bucket should resemble: **Example DAG folder** ``` dags/ └ dag_def.py ``` Amazon MWAA automatically syncs new and changed objects from your Amazon S3 bucket to Amazon MWAA scheduler and worker containers’ `/usr/local/airflow/dags` folder every 30 seconds, preserving the Amazon S3 source’s file hierarchy, regardless of file type. The time that new DAGs take to be listed in your Apache Airflow UI is controlled by `scheduler.dag\$1dir\$1list\$1interval`. Changes to existing DAGs will be picked up on the next [DAG processing loop](best-practices-tuning.md#best-practices-tuning-scheduler). **Note** You do not need to include the `airflow.cfg` configuration file in your DAG folder. You can override the default Apache Airflow configurations from the Amazon MWAA console. For more information, refer to [Using Apache Airflow configuration options on Amazon MWAA](configuring-env-variables.md). ## What's changed? To review changes for a specific Apache Airflow release, refer to the [Release Notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#release-notes) page. + Apache Airflow v3 configurations: [Configuration Reference](https://airflow.apache.org/docs/apache-airflow/3.0.6/configurations-ref.html) + Apache Airflow v2 public interface information: [Public Interface of Airflow](https://airflow.apache.org/docs/apache-airflow/2.10.3/public-airflow-interface.html) ## Testing DAGs using the Amazon MWAA CLI utility + The command line interface (CLI) utility replicates an Amazon Managed Workflows for Apache Airflow environment locally. + The CLI builds a Docker container image locally that’s similar to an Amazon MWAA production image. You can use this to run a local Apache Airflow environment to develop and test DAGs, custom plugins, and dependencies before deploying to Amazon MWAA. + To run the CLI, refer to [aws-mwaa-docker-images](https://github.com/aws/amazon-mwaa-docker-images) on GitHub. ## Uploading DAG code to Amazon S3 You can use the Amazon S3 console or the AWS Command Line Interface (AWS CLI) to upload DAG code to your Amazon S3 bucket. The following steps assume you are uploading code (`.py`) to a folder named `dags` in your Amazon S3 bucket. ### Using the AWS CLI The AWS Command Line Interface (AWS CLI) is an open source tool that you can use to interact with AWS services using commands in your command-line shell. To complete the steps on this page, you need the following: + [AWS CLI – Install version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html). + [AWS CLI – Quick configuration with `aws configure`](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). **To upload using the AWS CLI** 1. Use the following command to list all of your Amazon S3 buckets. ``` aws s3 ls ``` 1. Use the following command to list the files and folders in the Amazon S3 bucket for your environment. ``` aws s3 ls s3://YOUR_S3_BUCKET_NAME ``` 1. The following command uploads a `dag_def.py` file to a `dags` folder. ``` aws s3 cp dag_def.py s3://amzn-s3-demo-bucket/dags/ ``` If a folder named `dags` does not already exist on your Amazon S3 bucket, this command creates the `dags` folder and uploads the file named `dag_def.py` to the new folder. ### Using the Amazon S3 console The Amazon S3 console is a web-based user interface that you can use to create and manage the resources in your Amazon S3 bucket. The following steps assume you have a DAGs folder named `dags`. **To upload using the Amazon S3 console** 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Select the **S3 bucket** link in the **DAG code in S3** pane to open your storage bucket in the console. 1. Choose the `dags` folder. 1. Choose **Upload**. 1. Choose **Add file**. 1. Select the local copy of your `dag_def.py`, choose **Upload**. ## Specifying the path to your DAGs folder on the Amazon MWAA console (the first time) The following steps assume you are specifying the path to a folder on your Amazon S3 bucket named `dags`. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose the environment where you want to run DAGs. 1. Choose **Edit**. 1. On the **DAG code in Amazon S3** pane, choose **Browse S3** adjacent to the **DAG folder** field. 1. Select your `dags` folder. 1. Choose **Choose**. 1. Choose **Next**, **Update environment**. ## Accessing changes on your Apache Airflow UI You need [Apache Airflow UI access policy: AmazonMWAAWebServerAccess](access-policies.md#web-ui-access) permissions for your AWS account in AWS Identity and Access Management (IAM) to access your Apache Airflow UI. **To access your Apache Airflow UI** 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Choose **Open Airflow UI**. ## What's next? Test your DAGs, custom plugins, and Python dependencies locally using [aws-mwaa-docker-images](https://github.com/aws/amazon-mwaa-docker-images) on GitHub. # Installing custom plugins Amazon Managed Workflows for Apache Airflow supports Apache Airflow's built-in plugin manager, allowing you to use custom Apache Airflow operators, hooks, sensors, or interfaces. This page describes the steps to install [Apache Airflow custom plugins](https://airflow.incubator.apache.org/plugins.html) on your Amazon MWAA environment using a `plugins.zip` file. **Contents** + [Prerequisites](#configuring-dag-plugins-prereqs) + [How it works](#configuring-dag-plugins-how) + [When to use the plugins](#configuring-dag-plugins-changed) + [Custom plugins overview](#configuring-dag-plugins-overview) + [Custom plugins directory and size limits](#configuring-dag-plugins-quota) + [Examples of custom plugins](#configuring-dag-plugins-airflow-ex) + [Example using a flat directory structure in plugins.zip](#configuring-dag-plugins-overview-simple) + [Example using a nested directory structure in plugins.zip](#configuring-dag-plugins-overview-complex) + [Creating a plugins.zip file](#configuring-dag-plugins-test-create) + [Step one: Test custom plugins using the Amazon MWAA CLI utility](#configuring-dag-plugins-cli-utility) + [Step two: Create the plugins.zip file](#configuring-dag-plugins-zip) + [Uploading `plugins.zip` to Amazon S3](#configuring-dag-plugins-upload) + [Using the AWS CLI](#configuring-dag-plugins-upload-cli) + [Using the Amazon S3 console](#configuring-dag-plugins-upload-console) + [Installing custom plugins on your environment](#configuring-dag-plugins-mwaa-installing) + [Specifying the path to `plugins.zip` on the Amazon MWAA console (the first time)](#configuring-dag-plugins-mwaa-first) + [Specifying the `plugins.zip` version on the Amazon MWAA console](#configuring-dag-plugins-s3-mwaaconsole) + [Example use cases for plugins.zip](#configuring-dag-plugins-examples) + [What's next?](#configuring-dag-plugins-next-up) ## Prerequisites You'll need the following before you can complete the steps on this page. + **Permissions** — Your AWS account must have been granted access by your administrator to the [AmazonMWAAFullConsoleAccess](access-policies.md#console-full-access) access control policy for your environment. In addition, your Amazon MWAA environment must be permitted by your [execution role](mwaa-create-role.md) to access the AWS resources used by your environment. + **Access** — If you require access to public repositories to install dependencies directly on the webserver, your environment must be configured with **public network** webserver access. For more information, refer to [Apache Airflow access modes](configuring-networking.md). + **Amazon S3 configuration** — The [Amazon S3 bucket](mwaa-s3-bucket.md) used to store your DAGs, custom plugins in `plugins.zip`, and Python dependencies in `requirements.txt` must be configured with *Public Access Blocked* and *Versioning Enabled*. ## How it works To run custom plugins on your environment, you must do three things: 1. Create a `plugins.zip` file locally. 1. Upload the local `plugins.zip` file to your Amazon S3 bucket. 1. Specify the version of this file in the **Plugins file** field on the Amazon MWAA console. **Note** If this is the first time you're uploading a `plugins.zip` to your Amazon S3 bucket, you also need to specify the path to the file on the Amazon MWAA console. You only need to complete this step once. ## When to use the plugins Plugins are required only for extending the Apache Airflow user interface, as outlined in the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html#plugins). Custom operators can be placed directly in the `/dags` folder alongside your `DAG` code. If you need to create your own integrations with external systems, place them in the /`dags` folder or a subfolder within it, but not in the `plugins.zip` folder. In Apache Airflow 2.x, plugins are primarily used for extending the UI. Similarly, other dependencies can not be placed in `plugins.zip`. Instead, they can be stored in a location in the Amazon S3 `/dags` folder, where they will be synchronized to each Amazon MWAA container before Apache Airflow starts. **Note** Any file in the `/dags` folder or in `plugins.zip` that does not explicitly define an Apache Airflow DAG object must be listed in an `.airflowignore` file. ## Custom plugins overview Apache Airflow's built-in plugin manager can integrate external features to its core by simply dropping files in an `$AIRFLOW_HOME/plugins` folder. It you can use to use custom Apache Airflow operators, hooks, sensors, or interfaces. The following section provides an example of flat and nested directory structures in a local development environment and the resulting import statements, which determines the directory structure within a plugins.zip. ### Custom plugins directory and size limits The Apache Airflow scheduler and the workers search for custom plugins during startup on the AWS-managed Fargate container for your environment at `/usr/local/airflow/plugins/*`. + **Directory structure**. The directory structure (at `/*`) is based on the contents of your `plugins.zip` file. For example, if your `plugins.zip` contains the `operators` directory as a main-level directory, then the directory will be extracted to `/usr/local/airflow/plugins/operators` on your environment. + **Size limit**. We recommend a `plugins.zip` file less than than 1 GB. The larger the size of a `plugins.zip` file, the longer the startup time on an environment. Although Amazon MWAA doesn't limit the size of a `plugins.zip` file explicitly, if dependencies can't be installed within ten minutes, the Fargate service will time-out and attempt to rollback the environment to a stable state. **Note** For environments using Apache Airflow v2.0.2, Amazon MWAA limits outbound traffic on the Apache Airflow webserver, and does not permit installing plugins nor Python dependencies directly on the webserver. Starting with Apache Airflow v2.2.2, Amazon MWAA can install plugins and dependencies directly on the webserver. ## Examples of custom plugins The following section uses sample code in the *Apache Airflow reference guide* to explain how to structure your local development environment. ### Example using a flat directory structure in plugins.zip ------ #### [ Apache Airflow v3 ] The following example presents a `plugins.zip` file with a flat directory structure for Apache Airflow v3. **Example flat directory with `PythonVirtualenvOperator` plugins.zip** The following example displays the main-level tree of a plugins.zip file for the `PythonVirtualenvOperator` custom plugin in [Creating a custom plugin for Apache Airflow PythonVirtualenvOperator](samples-virtualenv.md). ``` ├── virtual_python_plugin.py ``` **Example plugins/virtual\$1python\$1plugin.py** The following example displays the `PythonVirtualenvOperator` custom plugin. ``` """ Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """ from airflow.plugins_manager import AirflowPlugin import airflow.utils.python_virtualenv from typing import List def _generate_virtualenv_cmd(tmp_dir: str, python_bin: str, system_site_packages: bool) -> List[str]: cmd = ['python3','/usr/local/airflow/.local/lib/python3.7/site-packages/virtualenv', tmp_dir] if system_site_packages: cmd.append('--system-site-packages') if python_bin is not None: cmd.append(f'--python={python_bin}') return cmd airflow.utils.python_virtualenv._generate_virtualenv_cmd=_generate_virtualenv_cmd class VirtualPythonPlugin(AirflowPlugin): name = 'virtual_python_plugin' ``` ------ #### [ Apache Airflow v2 ] The following example presents a `plugins.zip` file with a flat directory structure for Apache Airflow v2. **Example flat directory with `PythonVirtualenvOperator` plugins.zip** The following example displays the main-level tree of a plugins.zip file for the `PythonVirtualenvOperator` custom plugin in [Creating a custom plugin for Apache Airflow PythonVirtualenvOperator](samples-virtualenv.md). ``` ├── virtual_python_plugin.py ``` **Example plugins/virtual\$1python\$1plugin.py** The following example displays the `PythonVirtualenvOperator` custom plugin. ``` """ Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """ from airflow.plugins_manager import AirflowPlugin import airflow.utils.python_virtualenv from typing import List def _generate_virtualenv_cmd(tmp_dir: str, python_bin: str, system_site_packages: bool) -> List[str]: cmd = ['python3','/usr/local/airflow/.local/lib/python3.7/site-packages/virtualenv', tmp_dir] if system_site_packages: cmd.append('--system-site-packages') if python_bin is not None: cmd.append(f'--python={python_bin}') return cmd airflow.utils.python_virtualenv._generate_virtualenv_cmd=_generate_virtualenv_cmd class VirtualPythonPlugin(AirflowPlugin): name = 'virtual_python_plugin' ``` ------ ### Example using a nested directory structure in plugins.zip ------ #### [ Apache Airflow v3 ] The following example presents a `plugins.zip` file with separate directories for `hooks`, `operators`, and a `sensors` directory. **Example plugins.zip** ``` __init__.py my_airflow_plugin.py hooks/ |-- __init__.py |-- my_airflow_hook.py operators/ |-- __init__.py |-- my_airflow_operator.py |-- hello_operator.py sensors/ |-- __init__.py |-- my_airflow_sensor.py ``` The following example displays the import statements in the DAG ([DAGs folder](https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-folder.html#configuring-dag-folder-how)) that uses the custom plugins. **Example dags/your\$1dag.py** ``` from airflow import DAG from datetime import datetime, timedelta from operators.my_airflow_operator import MyOperator from sensors.my_airflow_sensor import MySensor from operators.hello_operator import HelloOperator default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime(2018, 1, 1), 'email_on_failure': False, 'email_on_retry': False, 'retries': 1, 'retry_delay': timedelta(minutes=5), } with DAG('customdag', max_active_runs=3, schedule_interval='@once', default_args=default_args) as dag: sens = MySensor( task_id='taskA' ) op = MyOperator( task_id='taskB', my_field='some text' ) hello_task = HelloOperator(task_id='sample-task', name='foo_bar') sens >> op >> hello_task ``` **Example plugins/my\$1airflow\$1plugin.py** ``` from airflow.plugins_manager import AirflowPlugin from hooks.my_airflow_hook import * from operators.my_airflow_operator import * class PluginName(AirflowPlugin): name = 'my_airflow_plugin' hooks = [MyHook] operators = [MyOperator] sensors = [MySensor] ``` The following examples present each of the import statements needed in the custom plugin files. **Example hooks/my\$1airflow\$1hook.py** ``` from airflow.hooks.base import BaseHook class MyHook(BaseHook): def my_method(self): print("Hello World") ``` **Example sensors/my\$1airflow\$1sensor.py** ``` from airflow.sensors.base import BaseSensorOperator from airflow.utils.decorators import apply_defaults class MySensor(BaseSensorOperator): @apply_defaults def __init__(self, *args, **kwargs): super(MySensor, self).__init__(*args, **kwargs) def poke(self, context): return True ``` **Example operators/my\$1airflow\$1operator.py** ``` from airflow.operators.bash import BaseOperator from airflow.utils.decorators import apply_defaults from hooks.my_airflow_hook import MyHook class MyOperator(BaseOperator): @apply_defaults def __init__(self, my_field, *args, **kwargs): super(MyOperator, self).__init__(*args, **kwargs) self.my_field = my_field def execute(self, context): hook = MyHook('my_conn') hook.my_method() ``` **Example operators/hello\$1operator.py** ``` from airflow.models.baseoperator import BaseOperator from airflow.utils.decorators import apply_defaults class HelloOperator(BaseOperator): @apply_defaults def __init__( self, name: str, **kwargs) -> None: super().__init__(**kwargs) self.name = name def execute(self, context): message = "Hello {}".format(self.name) print(message) return message ``` Follow the steps in [Testing custom plugins using the Amazon MWAA CLI utility](#configuring-dag-plugins-cli-utility), and then [Creating a plugins.zip file](#configuring-dag-plugins-zip) to zip the contents **within** your `plugins` directory. For example, `cd plugins`. ------ #### [ Apache Airflow v2 ] The following example presents a `plugins.zip` file with separate directories for `hooks`, `operators`, and a `sensors` directory. **Example plugins.zip** ``` __init__.py my_airflow_plugin.py hooks/ |-- __init__.py |-- my_airflow_hook.py operators/ |-- __init__.py |-- my_airflow_operator.py |-- hello_operator.py sensors/ |-- __init__.py |-- my_airflow_sensor.py ``` The following example displays the import statements in the DAG ([DAGs folder](https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-folder.html#configuring-dag-folder-how)) that uses the custom plugins. **Example dags/your\$1dag.py** ``` from airflow import DAG from datetime import datetime, timedelta from operators.my_airflow_operator import MyOperator from sensors.my_airflow_sensor import MySensor from operators.hello_operator import HelloOperator default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime(2018, 1, 1), 'email_on_failure': False, 'email_on_retry': False, 'retries': 1, 'retry_delay': timedelta(minutes=5), } with DAG('customdag', max_active_runs=3, schedule_interval='@once', default_args=default_args) as dag: sens = MySensor( task_id='taskA' ) op = MyOperator( task_id='taskB', my_field='some text' ) hello_task = HelloOperator(task_id='sample-task', name='foo_bar') sens >> op >> hello_task ``` **Example plugins/my\$1airflow\$1plugin.py** ``` from airflow.plugins_manager import AirflowPlugin from hooks.my_airflow_hook import * from operators.my_airflow_operator import * class PluginName(AirflowPlugin): name = 'my_airflow_plugin' hooks = [MyHook] operators = [MyOperator] sensors = [MySensor] ``` The following examples present each of the import statements needed in the custom plugin files. **Example hooks/my\$1airflow\$1hook.py** ``` from airflow.hooks.base import BaseHook class MyHook(BaseHook): def my_method(self): print("Hello World") ``` **Example sensors/my\$1airflow\$1sensor.py** ``` from airflow.sensors.base import BaseSensorOperator from airflow.utils.decorators import apply_defaults class MySensor(BaseSensorOperator): @apply_defaults def __init__(self, *args, **kwargs): super(MySensor, self).__init__(*args, **kwargs) def poke(self, context): return True ``` **Example operators/my\$1airflow\$1operator.py** ``` from airflow.operators.bash import BaseOperator from airflow.utils.decorators import apply_defaults from hooks.my_airflow_hook import MyHook class MyOperator(BaseOperator): @apply_defaults def __init__(self, my_field, *args, **kwargs): super(MyOperator, self).__init__(*args, **kwargs) self.my_field = my_field def execute(self, context): hook = MyHook('my_conn') hook.my_method() ``` **Example operators/hello\$1operator.py** ``` from airflow.models.baseoperator import BaseOperator from airflow.utils.decorators import apply_defaults class HelloOperator(BaseOperator): @apply_defaults def __init__( self, name: str, **kwargs) -> None: super().__init__(**kwargs) self.name = name def execute(self, context): message = "Hello {}".format(self.name) print(message) return message ``` Follow the steps in [Testing custom plugins using the Amazon MWAA CLI utility](#configuring-dag-plugins-cli-utility), and then [Creating a plugins.zip file](#configuring-dag-plugins-zip) to zip the contents **within** your `plugins` directory. For example, `cd plugins`. ------ ## Creating a plugins.zip file The following steps describe the steps we recommend to create a plugins.zip file locally. ### Step one: Test custom plugins using the Amazon MWAA CLI utility + The command line interface (CLI) utility replicates an Amazon Managed Workflows for Apache Airflow environment locally. + The CLI builds a Docker container image locally that’s similar to an Amazon MWAA production image. You can use this to run a local Apache Airflow environment to develop and test DAGs, custom plugins, and dependencies before deploying to Amazon MWAA. + To run the CLI, refer to [aws-mwaa-docker-images](https://github.com/aws/amazon-mwaa-docker-images) on GitHub. ### Step two: Create the plugins.zip file You can use a built-in ZIP archive utility, or any other ZIP utility (such as [7zip](https://www.7-zip.org/download.html)) to create a .zip file. **Note** The built-in zip utility for Windows OS might add subfolders when you create a .zip file. We recommend verifying the contents of the plugins.zip file before uploading to your Amazon S3 bucket to ensure no additional directories were added. 1. Change directories to your local Airflow plugins directory. For example: ``` myproject$ cd plugins ``` 1. Run the following command to ensure that the contents have executable permissions (macOS and Linux only). ``` plugins$ chmod -R 755 . ``` 1. Zip the contents **within** your `plugins` folder. ``` plugins$ zip -r plugins.zip . ``` ## Uploading `plugins.zip` to Amazon S3 You can use the Amazon S3 console or the AWS Command Line Interface (AWS CLI) to upload a `plugins.zip` file to your Amazon S3 bucket. ### Using the AWS CLI The AWS Command Line Interface (AWS CLI) is an open source tool that you can use to interact with AWS services using commands in your command-line shell. To complete the steps on this page, you need the following: + [AWS CLI – Install version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html). + [AWS CLI – Quick configuration with `aws configure`](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). **To upload using the AWS CLI** 1. In your command prompt, navigate to the directory where your `plugins.zip` file is stored. For example: ``` cd plugins ``` 1. Use the following command to list all of your Amazon S3 buckets. ``` aws s3 ls ``` 1. Use the following command to list the files and folders in the Amazon S3 bucket for your environment. ``` aws s3 ls s3://YOUR_S3_BUCKET_NAME ``` 1. Use the following command to upload the `plugins.zip` file to the Amazon S3 bucket for your environment. ``` aws s3 cp plugins.zip s3://amzn-s3-demo-bucket/plugins.zip ``` ### Using the Amazon S3 console The Amazon S3 console is a web-based user interface that you can use to create and manage the resources in your Amazon S3 bucket. **To upload using the Amazon S3 console** 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Select the **S3 bucket** link in the **DAG code in S3** pane to open your storage bucket in the console. 1. Choose **Upload**. 1. Choose **Add file**. 1. Select the local copy of your `plugins.zip`, choose **Upload**. ## Installing custom plugins on your environment This section describes how to install the custom plugins you uploaded to your Amazon S3 bucket by specifying the path to the plugins.zip file, and specifying the version of the plugins.zip file each time the zip file is updated. ### Specifying the path to `plugins.zip` on the Amazon MWAA console (the first time) If this is the first time you're uploading a `plugins.zip` to your Amazon S3 bucket, you also need to specify the path to the file on the Amazon MWAA console. You only need to complete this step once. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Choose **Edit**. 1. On the **DAG code in Amazon S3** pane, choose **Browse S3** adjacent to the **Plugins file - optional** field. 1. Select the `plugins.zip` file on your Amazon S3 bucket. 1. Choose **Choose**. 1. Choose **Next**, **Update environment**. ### Specifying the `plugins.zip` version on the Amazon MWAA console You need to specify the version of your `plugins.zip` file on the Amazon MWAA console each time you upload a new version of your `plugins.zip` in your Amazon S3 bucket. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Choose **Edit**. 1. On the **DAG code in Amazon S3** pane, choose a `plugins.zip` version in the dropdown list. 1. Choose **Next**. ## Example use cases for plugins.zip + Learn how to create a custom plugin in [Custom plugin with Apache Hive and Hadoop](samples-hive.md). + Learn how to create a custom plugin in [Custom plugin to patch PythonVirtualenvOperator ](samples-virtualenv.md). + Learn how to create a custom plugin in [Custom plugin with Oracle](samples-oracle.md). + Learn how to create a custom plugin in [Changing a DAG's timezone on Amazon MWAA](samples-plugins-timezone.md). ## What's next? Test your DAGs, custom plugins, and Python dependencies locally using [aws-mwaa-docker-images](https://github.com/aws/amazon-mwaa-docker-images) on GitHub. # Installing Python dependencies A Python dependency is any package or distribution not included in the Apache Airflow base install for your Apache Airflow version on your Amazon Managed Workflows for Apache Airflow environment. This topic describes the steps to install Apache Airflow Python dependencies on your Amazon MWAA environment using a `requirements.txt` file in your Amazon S3 bucket. **Contents** + [Prerequisites](#working-dags-dependencies-prereqs) + [How it works](#working-dags-dependencies-how) + [Python dependencies overview](#working-dags-dependencies-overview) + [Python dependencies location and size limits](#working-dags-dependencies-quota) + [Creating a requirements.txt file](#working-dags-dependencies-test-create) + [Step one: Test Python dependencies using the Amazon MWAA CLI utility](#working-dags-dependencies-cli-utility) + [Step two: Create the `requirements.txt`](#working-dags-dependencies-syntax-create) + [Uploading `requirements.txt` to Amazon S3](#configuring-dag-dependencies-upload) + [Using the AWS CLI](#configuring-dag-dependencies-upload-cli) + [Using the Amazon S3 console](#configuring-dag-dependencies-upload-console) + [Installing Python dependencies on your environment](#configuring-dag-dependencies-installing) + [Specifying the path to `requirements.txt` on the Amazon MWAA console (the first time)](#configuring-dag-dependencies-first) + [Specifying the `requirements.txt` version on the Amazon MWAA console](#working-dags-dependencies-mwaaconsole-version) + [Accessing logs for your `requirements.txt`](#working-dags-dependencies-logs) + [What's next?](#working-dags-dependencies-next-up) ## Prerequisites You'll need the following before you can complete the steps on this page. + **Permissions** — Your AWS account must have been granted access by your administrator to the [AmazonMWAAFullConsoleAccess](access-policies.md#console-full-access) access control policy for your environment. In addition, your Amazon MWAA environment must be permitted by your [execution role](mwaa-create-role.md) to access the AWS resources used by your environment. + **Access** — If you require access to public repositories to install dependencies directly on the webserver, your environment must be configured with **public network** webserver access. For more information, refer to [Apache Airflow access modes](configuring-networking.md). + **Amazon S3 configuration** — The [Amazon S3 bucket](mwaa-s3-bucket.md) used to store your DAGs, custom plugins in `plugins.zip`, and Python dependencies in `requirements.txt` must be configured with *Public Access Blocked* and *Versioning Enabled*. ## How it works On Amazon MWAA, you install all Python dependencies by uploading a `requirements.txt` file to your Amazon S3 bucket, then specifying the version of the file on the Amazon MWAA console each time you update the file. Amazon MWAA runs `pip3 install -r requirements.txt` to install the Python dependencies on the Apache Airflow scheduler and each of the workers. To run Python dependencies on your environment, you must do three things: 1. Create a `requirements.txt` file locally. 1. Upload the local `requirements.txt` to your Amazon S3 bucket. 1. Specify the version of this file in the **Requirements file** field on the Amazon MWAA console. **Note** If this is the first time you're creating and uploading a `requirements.txt` to your Amazon S3 bucket, you also need to specify the path to the file on the Amazon MWAA console. You only need to complete this step once. ## Python dependencies overview You can install Apache Airflow extras and other Python dependencies from the Python Package Index (PyPi.org), Python wheels (`.whl`), or Python dependencies hosted on a private PyPi/PEP-503 Compliant Repo on your environment. ### Python dependencies location and size limits The Apache Airflow scheduler and the workers search for the packages in the `requirements.txt` file and the packages are installed on the environment at `/usr/local/airflow/.local/bin`. + **Size limit**. We recommend a `requirements.txt` file that references libraries whose combined size is less than than 1 GB. The more libraries Amazon MWAA needs to install, the longer the *startup* time on an environment. Although Amazon MWAA doesn't limit the size of installed libraries explicitly, if dependencies can't be installed within ten minutes, the Fargate service will time-out and attempt to rollback the environment to a stable state. ## Creating a requirements.txt file The following steps describe the steps we recommend to create a requirements.txt file locally. ### Step one: Test Python dependencies using the Amazon MWAA CLI utility + The command line interface (CLI) utility replicates an Amazon Managed Workflows for Apache Airflow environment locally. + The CLI builds a Docker container image locally that’s similar to an Amazon MWAA production image. You can use this to run a local Apache Airflow environment to develop and test DAGs, custom plugins, and dependencies before deploying to Amazon MWAA. + To run the CLI, refer to [aws-mwaa-docker-images](https://github.com/aws/amazon-mwaa-docker-images) on GitHub. ### Step two: Create the `requirements.txt` The following section describes how to specify Python dependencies from the [Python Package Index](https://pypi.org/) in a `requirements.txt` file. ------ #### [ Apache Airflow v3 ] 1. **Test locally**. Add additional libraries iteratively to find the right combination of packages and their versions, before creating a `requirements.txt` file. To run the Amazon MWAA CLI utility, refer to [aws-mwaa-docker-images](https://github.com/aws/amazon-mwaa-docker-images) on GitHub. 1. **Review the Apache Airflow package extras**. To access a list of the packages installed for Apache Airflow v3 on Amazon MWAA, refer to [aws-mwaa-docker-images `requirements.txt`](https://github.com/aws/amazon-mwaa-docker-images/blob/main/requirements.txt) on the GitHub website. 1. **Add a constraints statement**. Add the constraints file for your Apache Airflow v3 environment at the top of your `requirements.txt` file. Apache Airflow constraints files specify the provider versions available at the time of a Apache Airflow release. In the following example, replace *\$1environment-version\$1* with your environment's version number, and *\$1Python-version\$1* with the version of Python that's compatible with your environment. For information about the version of Python compatible with your Apache Airflow environment, refer to [Apache Airflow Versions](airflow-versions.md#airflow-versions-official). ``` --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-{Airflow-version}/constraints-{Python-version}.txt" ``` If the constraints file determines that `xyz==1.0` package is not compatible with other packages in your environment, `pip3 install` will fail to prevent incompatible libraries from being installed to your environment. If installation fails for any packages, you can access error logs for each Apache Airflow component (the scheduler, worker, and webserver) in the corresponding log stream on CloudWatch Logs. For more information about log types, refer to [Accessing Airflow logs in Amazon CloudWatch](monitoring-airflow.md). 1. **Apache Airflow packages**. Add the [package extras](http://airflow.apache.org/docs/apache-airflow/2.5.1/extra-packages-ref.html) and the version (`==`). This helps to prevent packages of the same name, but different version, from being installed on your environment. ``` apache-airflow[package-extra]==2.5.1 ``` 1. **Python libraries**. Add the package name and the version (`==`) in your `requirements.txt` file. This helps to prevent a future breaking update from [PyPi.org](https://pypi.org) from being automatically applied. ``` library == version ``` **Example Boto3 and psycopg2-binary** This example is provided for demonstration purposes. The boto and psycopg2-binary libraries are included with the base install for Apache Airflow v3 and don't need to be specified in a `requirements.txt` file. ``` boto3==1.17.54 boto==2.49.0 botocore==1.20.54 psycopg2-binary==2.8.6 ``` If a package is specified without a version, Amazon MWAA installs the latest version of the package from [PyPi.org](https://pypi.org). This version might conflict with other packages in your `requirements.txt`. ------ #### [ Apache Airflow v2 ] 1. **Test locally**. Add additional libraries iteratively to find the right combination of packages and their versions, before creating a `requirements.txt` file. To run the Amazon MWAA CLI utility, refer to [aws-mwaa-docker-images](https://github.com/aws/amazon-mwaa-docker-images) on GitHub. 1. **Review the Apache Airflow package extras**. To access a list of the packages installed for Apache Airflow v2 on Amazon MWAA, access [aws-mwaa-docker-images `requirements.txt`](https://github.com/aws/amazon-mwaa-docker-images/blob/main/requirements.txt) on the GitHub website. 1. **Add a constraints statement**. Add the constraints file for your Apache Airflow v2 environment at the top of your `requirements.txt` file. Apache Airflow constraints files specify the provider versions available at the time of a Apache Airflow release. Beginning with Apache Airflow v2.7.2, your requirements file must include a `--constraint` statement. If you do not provide a constraint, Amazon MWAA will specify one for you to ensure the packages listed in your requirements are compatible with the version of Apache Airflow you are using. In the following example, replace *\$1environment-version\$1* with your environment's version number, and *\$1Python-version\$1* with the version of Python that's compatible with your environment. For information about the version of Python compatible with your Apache Airflow environment, refer to [Apache Airflow Versions](airflow-versions.md#airflow-versions-official). ``` --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-{Airflow-version}/constraints-{Python-version}.txt" ``` If the constraints file determines that `xyz==1.0` package is not compatible with other packages in your environment, `pip3 install` will fail to prevent incompatible libraries from being installed to your environment. If installation fails for any packages, you can access error logs for each Apache Airflow component (the scheduler, worker, and webserver) in the corresponding log stream on CloudWatch Logs. For more information about log types, refer to [Accessing Airflow logs in Amazon CloudWatch](monitoring-airflow.md). 1. **Apache Airflow packages**. Add the [package extras](http://airflow.apache.org/docs/apache-airflow/2.5.1/extra-packages-ref.html) and the version (`==`). This helps to prevent packages of the same name, but different version, from being installed on your environment. ``` apache-airflow[package-extra]==2.5.1 ``` 1. **Python libraries**. Add the package name and the version (`==`) in your `requirements.txt` file. This helps to prevent a future breaking update from [PyPi.org](https://pypi.org) from being automatically applied. ``` library == version ``` **Example Boto3 and psycopg2-binary** This example is provided for demonstration purposes. The boto and psycopg2-binary libraries are included with the Apache Airflow v2 base install and don't need to be specified in a `requirements.txt` file. ``` boto3==1.17.54 boto==2.49.0 botocore==1.20.54 psycopg2-binary==2.8.6 ``` If a package is specified without a version, Amazon MWAA installs the latest version of the package from [PyPi.org](https://pypi.org). This version might conflict with other packages in your `requirements.txt`. ------ ## Uploading `requirements.txt` to Amazon S3 You can use the Amazon S3 console or the AWS Command Line Interface (AWS CLI) to upload a `requirements.txt` file to your Amazon S3 bucket. ### Using the AWS CLI The AWS Command Line Interface (AWS CLI) is an open source tool that you can use to interact with AWS services using commands in your command-line shell. To complete the steps on this page, you need the following: + [AWS CLI – Install version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html). + [AWS CLI – Quick configuration with `aws configure`](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). **To upload using the AWS CLI** 1. Use the following command to list all of your Amazon S3 buckets. ``` aws s3 ls ``` 1. Use the following command to list the files and folders in the Amazon S3 bucket for your environment. ``` aws s3 ls s3://YOUR_S3_BUCKET_NAME ``` 1. The following command uploads a `requirements.txt` file to an Amazon S3 bucket. ``` aws s3 cp requirements.txt s3://amzn-s3-demo-bucket/requirements.txt ``` ### Using the Amazon S3 console The Amazon S3 console is a web-based user interface that you can use to create and manage the resources in your Amazon S3 bucket. **To upload using the Amazon S3 console** 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Select the **S3 bucket** link in the **DAG code in S3** pane to open your storage bucket in the console. 1. Choose **Upload**. 1. Choose **Add file**. 1. Select the local copy of your `requirements.txt`, choose **Upload**. ## Installing Python dependencies on your environment This section describes how to install the dependencies you uploaded to your Amazon S3 bucket by specifying the path to the requirements.txt file, and specifying the version of the requirements.txt file each time it's updated. ### Specifying the path to `requirements.txt` on the Amazon MWAA console (the first time) If this is the first time you're creating and uploading a `requirements.txt` to your Amazon S3 bucket, you also need to specify the path to the file on the Amazon MWAA console. You only need to complete this step once. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Choose **Edit**. 1. On the **DAG code in Amazon S3** pane, choose **Browse S3** adjacent to the **Requirements file - optional** field. 1. Select the `requirements.txt` file on your Amazon S3 bucket. 1. Choose **Choose**. 1. Choose **Next**, **Update environment**. You can begin using the new packages immediately after your environment finishes updating. ### Specifying the `requirements.txt` version on the Amazon MWAA console You need to specify the version of your `requirements.txt` file on the Amazon MWAA console each time you upload a new version of your `requirements.txt` in your Amazon S3 bucket. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Choose **Edit**. 1. On the **DAG code in Amazon S3 ** pane, choose a `requirements.txt` version in the dropdown list. 1. Choose **Next**, **Update environment**. You can begin using the new packages immediately after your environment finishes updating. ## Accessing logs for your `requirements.txt` You can view Apache Airflow logs for the scheduler scheduling your workflows and parsing your `dags` folder. The following steps describe how to open the log group for the scheduler on the Amazon MWAA console, and access Apache Airflow logs on the CloudWatch Logs console. **To access logs for a `requirements.txt`** 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Choose the **Airflow scheduler log group** on the **Monitoring** pane. 1. Choose the `requirements_install_ip` log in **Log streams**. 1. Refer to the list of packages that were installed on the environment at `/usr/local/airflow/.local/bin`. For example: ``` Collecting appdirs==1.4.4 (from -r /usr/local/airflow/.local/bin (line 1)) Downloading https://files.pythonhosted.org/packages/3b/00/2344469e2084fb28kjdsfiuyweb47389789vxbmnbjhsdgf5463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl Collecting astroid==2.4.2 (from -r /usr/local/airflow/.local/bin (line 2)) ``` 1. Review the list of packages and whether any of these encountered an error during installation. If something went wrong, you get an error similar to the following: ``` 2021-03-05T14:34:42.731-07:00 No matching distribution found for LibraryName==1.0.0 (from -r /usr/local/airflow/.local/bin (line 4)) No matching distribution found for LibraryName==1.0.0 (from -r /usr/local/airflow/.local/bin (line 4)) ``` ## What's next? Test your DAGs, custom plugins, and Python dependencies locally using [aws-mwaa-docker-images](https://github.com/aws/amazon-mwaa-docker-images) on GitHub. # Deleting files on Amazon S3 This page describes how versioning works in an Amazon S3 bucket for an Amazon Managed Workflows for Apache Airflow environment, and the steps to delete a DAG, `plugins.zip`, or `requirements.txt` file. **Contents** + [Prerequisites](#working-dags-delete-prereqs) + [Versioning overview](#working-dags-delete-overview) + [How it works](#working-dags-delete-how) + [Deleting a DAG on Amazon S3](#working-dags-s3-dag-delete) + [Removing a "current" requirements.txt or plugins.zip from an environment](#working-dags-s3-delete-version-c) + [Deleting a "non-current" (previous) requirements.txt or plugins.zip version](#working-dags-s3-delete-version-p) + [Using lifecycles to delete "non-current" (previous) versions and delete markers automatically](#working-dags-s3-delete-lifecycle) + [Example lifecycle policy to delete requirements.txt "non-current" versions and delete markers automatically](#working-dags-s3-delete-lifecycle-ex) + [What's next?](#working-dags-s3-delete-next-up) ## Prerequisites You'll need the following before you can complete the steps on this page. + **Permissions** — Your AWS account must have been granted access by your administrator to the [AmazonMWAAFullConsoleAccess](access-policies.md#console-full-access) access control policy for your environment. In addition, your Amazon MWAA environment must be permitted by your [execution role](mwaa-create-role.md) to access the AWS resources used by your environment. + **Access** — If you require access to public repositories to install dependencies directly on the webserver, your environment must be configured with **public network** webserver access. For more information, refer to [Apache Airflow access modes](configuring-networking.md). + **Amazon S3 configuration** — The [Amazon S3 bucket](mwaa-s3-bucket.md) used to store your DAGs, custom plugins in `plugins.zip`, and Python dependencies in `requirements.txt` must be configured with *Public Access Blocked* and *Versioning Enabled*. ## Versioning overview The `requirements.txt` and `plugins.zip` in your Amazon S3 bucket are versioned. When Amazon S3 bucket versioning is enabled for an object, and an artifact (for example, plugins.zip) is deleted from an Amazon S3 bucket, the file doesn't get deleted entirely. Anytime an artifact is deleted on Amazon S3, a new copy of the file is created that is a 404 (Object not found) error/0k file that says `I'm not here`. Amazon S3 calls this a *delete marker*. A delete marker is a "null" version of the file with a key name (or key) and version ID like any other object. We recommend deleting file versions and delete markers periodically to reduce storage costs for your Amazon S3 bucket. To delete "non-current" (previous) file versions entirely, you must delete the versions of the files, and then the *delete marker* for the version. ## How it works Amazon MWAA runs a sync operation on your Amazon S3 bucket every thirty seconds. This causes any DAG deletions in an Amazon S3 bucket to be synced to the Airflow image of your Fargate container. For `plugins.zip` and `requirements.txt` files, changes occur only after an environment update when Amazon MWAA builds a new Airflow image of your Fargate container with the custom plugins and Python dependencies. If you delete the *current* version of any of a `requirements.txt` or `plugins.zip` file, and then update your environment without providing a new version for the deleted file, then the update will fail with an error message, such as, `Unable to read version {version number} of file {file name}`. ## Deleting a DAG on Amazon S3 A DAG file (`.py`) is not versioned and can be deleted directly on the Amazon S3 console. The following steps describe how to delete a DAG on your Amazon S3 bucket. **To delete a DAG** 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Select the **S3 bucket** link in the **DAG code in S3** pane to open your storage bucket in the console. 1. Choose the `dags` folder. 1. Select the DAG, **Delete**. 1. Under **Delete objects?**, type `delete`. 1. Choose **Delete objects**. **Note** Apache Airflow preserves historical DAG runs. After a DAG has been run in Apache Airflow, it remains in the Airflow DAGs list regardless of the file status, until you delete it in Apache Airflow. To delete a DAG in Apache Airflow, choose the red "delete" button in the **Links** column. ## Removing a "current" requirements.txt or plugins.zip from an environment Currently, there isn't a way to remove a plugins.zip or requirements.txt from an environment after they’ve been added, but we're working on the issue. In the interim, a workaround is to point to an empty text or zip file, respectively. ## Deleting a "non-current" (previous) requirements.txt or plugins.zip version The `requirements.txt` and `plugins.zip` files in your Amazon S3 bucket are versioned on Amazon MWAA. If you want to delete these files on your Amazon S3 bucket entirely, you must retrieve the current version (121212) of the object (for example, plugins.zip), delete the version, and then remove the *delete marker* for the file versions. You can also delete "non-current" (previous) file versions on the Amazon S3 console; however, you'll still need to delete the *delete marker* using one of the following options. + To retrieve the object version, refer to [Retrieving object versions from a versioning-enabled bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RetrievingObjectVersions.html) *in the Amazon S3 guide*. + To delete the object version, refer to [Deleting object versions from a versioning-enabled bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html) *in the Amazon S3 guide*. + To remove a delete marker, refer to [Managing delete markers](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManagingDelMarkers.html) *in the Amazon S3 guide*. ## Using lifecycles to delete "non-current" (previous) versions and delete markers automatically You can configure a lifecycle policy for your Amazon S3 bucket to delete "non-current" (previous) versions of the plugins.zip and requirements.txt files in your Amazon S3 bucket after a certain number of days, or to remove an expired object's delete marker. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Under **DAG code in Amazon S3**, choose your Amazon S3 bucket. 1. Choose **Create lifecycle rule**. ## Example lifecycle policy to delete requirements.txt "non-current" versions and delete markers automatically Use the following example to create a lifecycle rule that permanently deletes "non-current" versions of a requirements.txt file and their delete markers after thirty days. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Under **DAG code in Amazon S3**, choose your Amazon S3 bucket. 1. Choose **Create lifecycle rule**. 1. In **Lifecycle rule name**, type `Delete previous requirements.txt versions and delete markers after thirty days`. 1. In **Prefix**, **requirements**. 1. In **Lifecycle rule actions**, choose **Permanently delete previous versions of objects** and **Delete expired delete markers or incomplete multipart uploads**. 1. In **Number of days after objects become previous versions**, type `30`. 1. In **Expired object delete markers**, choose **Delete expired object delete markers, objects are permanently deleted after 30 days**. ## What's next? + Learn more about Amazon S3 delete markers in [Managing delete markers](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html). + Learn more about Amazon S3 lifecycles in [Expiring objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html).