# Workflows To use Amazon MWAA Serverless for orchestration, first create a workflow as a YAML definition file. If you already have a Python DAG file, convert it to YAML with the [DAG converter](https://pypi.org/project/python-to-yaml-dag-converter-mwaa-serverless/) tool. For instructions, refer to [Convert Python DAG to YAML definition](workflows-migrate.md). **Important** Each workflow needs an IAM role. This role defines what AWS resources the workflow can access during execution. Amazon MWAA Serverless workflows can be run on-demand or scheduled. Amazon MWAA Serverless workflows can be one of the following three types: + **Scheduled**: Your workflow is run on the schedule you define in the workflow definition. You can still run your workflow on-demand outside of this schedule. This is similar to an `unpaused` DAG in Apache Airflow. + **Disabled**: A disabled workflow can not be run on schedule or on-demand. This is like a `paused` DAG in Apache Airflow. + **Manual only**: Your workflow ignores any defined schedules and can be only run on-demand. This option is best for testing or temporarily disabling automatic runs. **Note** Unlike Apache Airflow, Amazon MWAA Serverless doesn't automatically stop executing workflow runs when a workflow is set to **Manual only** or **Disabled**. Amazon MWAA Serverless uses a separate API [https://docs.aws.amazon.com/MWAA-serverless/APIReference/API_StopWorkflowRun.html](https://docs.aws.amazon.com/MWAA-serverless/APIReference/API_StopWorkflowRun.html) call to stop all workflow runs. **Topics** + [Manage your Amazon MWAA Serverless workflows](#workflows-manage) + [Workflow states](#workflows-states) + [Convert Python DAG to YAML definition](workflows-migrate.md) + [Tag a workflow](workflows-tags.md) ## Manage your Amazon MWAA Serverless workflows You can use [Amazon SageMaker Unified Studio](https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/iam-visual-workflows.html) to create and manage Amazon MWAA Serverless workflows. Amazon SageMaker Unified Studio provides visual tools that simplify workflow creation and management. Workflows are synchronized between Apache Airflow SageMaker Unified Studio and Amazon MWAA Serverless, allowing you to create workflows in either platform and access them from both. An Amazon MWAA Serverless workflow can also be managed using CLI or APIs. After uploading the workflow definition YAML in an Amazon S3 bucket, create the workflow. If you already have a Python workflow (DAG file), you can migrate it to Amazon MWAA Serverless by converting it into a YAML file. For more information, refer to [Convert Python DAG to YAML definition](workflows-migrate.md). To create your workflow using CLI, use `create-workflow` ------ #### [ CLI ] ``` aws mwaa-serverless create-workflow --name my-workflow-name --definition-s3-location'{"Bucket": "my-bucket-name", "ObjectKey": "my-yaml-file"}' ``` ------ To run your workflow, use [https://docs.aws.amazon.com/MWAA-serverless/APIReference/API_StartWorkflowRun.html](https://docs.aws.amazon.com/MWAA-serverless/APIReference/API_StartWorkflowRun.html) and provide the `workflow-arn`. This command starts a new workflow execution. ------ #### [ CLI ] ``` aws mwaa-serverless get-workflow --workflow-arn arn:aws:airflow-serverless:us-east-1:111122223333:workflow/workflow-name ``` ------ The following table contains all the actions you can perform on an Amazon MWAA Serverless workflow: | Action | API | | --- | --- | | Create workflow (the YAML file must already be in your Amazon S3 bucket). | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_CreateWorkflow.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_CreateWorkflow.html) | | Delete a workflow and all its versions. You must stop a workflow before you can delete it. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_DeleteWorkflow.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_DeleteWorkflow.html). | | Retrieve detailed information about a workflow. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_GetWorkflow.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_GetWorkflow.html) | | List all workflows. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_ListWorkflows.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_ListWorkflows.html) | | List all the verisions of a specified workflow. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_ListWorkflowVersions.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_ListWorkflowVersions.html) | | Start a workflow run. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_StartWorkflowRun.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_StartWorkflowRun.html) | | Stop a workflow run. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_StopWorkflowRun.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_StopWorkflowRun.html) | | Edit the configuration settings of a workflow. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_UpdateWorkflow.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_UpdateWorkflow.html) | | List tags for a workflow. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_ListTagsForResource.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_ListTagsForResource.html) | | Tag a workflow. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_TagResource.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_TagResource.html) | | Untag a workflow. | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_UntagResource.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_UntagResource.html) | ## Workflow states When you create a workflow with Amazon MWAA Serverless, it can have a `READY` or a `DELETING` state. When you run a workflow, it can adopt the following states. | State | Description | | --- | --- | | STARTING | The workflow is ready to run. | | QUEUED | The workflow is waiting in the queue to run. | | RUNNING | The workflow is in progress. | | SUCCESS | The workflow completed successfully. | | FAILED | The workflow failed. | | TIMEOUT | The workflow timed out before it completed. | | STOPPING | The workflow run is being stopped. | | STOPPED | The workflow is stopped and no resources are running. | # Convert Python DAG to YAML definition Amazon MWAA Serverless provides a Python to YAML DAG converter tool that helps convert you Python based DAG to YAML based definition that is ready to create a Amazon MWAA Serverless workflow. Refer to the following steps to convert your Python based DAGs to YAML based definition using AWS CLI: ------ #### [ CLI ] 1. Install the library ([dag-converter](https://pypi.org/project/python-to-yaml-dag-converter-mwaa-serverless/)): ``` pip install python-to-yaml-dag-converter-mwaa-serverless ``` 1. (Optional) If your Python DAG files are in S3, copy them locally: ``` aws s3 cp s3://source-bucket/dags/source.py/ my-local-folder/source.py ``` 1. Run following command. This tool only converts Python based DAG files and provides an output file `dag_id.yml` ``` dag-converter convert my-local-folder/source.py --output /output_yaml/destination ``` **Important** A single Python DAG file can potentially create multiple YAML definitions workflows. 1. Move the YAML workflow to an Amazon S3 location. ``` aws s3 cp /output_yaml/dag_id.yml s3://destination-bucket/dags/ddag_id.yml ``` 1. Refer to [Workflows](workflows.md) for steps to create Amazon MWAA Serverless workflows ------ # Tag a workflow You can assign tags to each Amazon MWAA Serverless workflow to help you manage resources. This section provides an overview of the tag functions and shows you how to create tags. ## What is a tag? A tag is a label that you assign to an AWS resource. Each tag consists of a key and a value, both of which you define. Tags enable you to categorize your AWS resources by attributes such as purpose, owner, and workflow. When you have many resources of the same type, you can quickly identify a specific resource based on the tags you've assigned to it. For example, you can define a set of tags for your Amazon MWAA Serverless applications to help you track each application's owner and stack level. We recommend that you use consistent set of tag keys for each resource type. Tags are not automatically assigned to your resources. After you add a tag to a resource, you can modify a tag’s value or remove the tag from the resource at any time. Tags do not have any semantic meaning to Amazon MWAA Serverless and are interpreted strictly as strings of characters. If you add a tag that has the same key as an existing tag on that resource, the new value overwrites the earlier value. You can control which user in your AWS account has permissions to manage tags through IAM For tag-based access control policy examples, see [ABAC with Amazon MWAA Serverless](security_iam_service-with-iam.md#security_iam_service-with-iam-tags). ## Tagging your resources Workflows are the only Amazon MWAA Serverless resources that you can tag. If you're using the Amazon MWAA Serverless API, the AWS CLI, or an AWS SDK, you can apply tags to new workflows using the `tags` parameter on the relevant API action. You can apply tags to existing workflows using the `TagResource` API action. You can use some resource-creating actions to specify tags for a resource when the resource is created. In this case, if tags cannot be applied while the resource is being created, the resource fails to be created. This mechanism ensures that resources you intended to tag on creation are either created with specified tags or not created at all. The following table describes the Amazon MWAA Serverless resources that can be tagged. **** | Resource | Supports tags | Supports tag propagation | Supports tagging on creation (Amazon MWAA Serverless API, AWS CLI, and AWS SDK) | API for creation (tags can be added during creation) | | --- | --- | --- | --- | --- | | Workflow | Yes | No. Tags associated with a workflow do not propagate to tasks within the workflow. | Yes | CreateWorkflow | ## Tagging limitations The following basic limitations apply to tags: + Each workflow can have a maximum of 50 user-created tags. + For each workflow, each tag key must be unique, and each tag key can have only one value. + The maximum key length is 128 Unicode characters in UTF-8. + The maximum value length is 256 Unicode characters in UTF-8. + Allowed characters are letters, numbers, spaces representable in UTF-8, and the following characters: \$1 . : / = \$1 - @. + A tag key cannot be an empty string. A tag value can be an empty string, but not null. + Tag keys and values are case sensitive. + Do not use `AWS:` or any upper or lowercase combination of such as a prefix for either keys or values. These are reserved only for AWS use. ## Working with tags using the AWS CLI and the Amazon MWAA Serverless API Use the following AWS CLI commands or Amazon MWAA Serverless API operations to add, update, list, and delete the tags for your workflows. **** | Resource | Supports tags | Supports tag propagation | | --- | --- | --- | | Add or overwrite one or more tags | tag-resource | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_TagResource.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_TagResource.html) | | List tags for a resource | list-tags-for-resource | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_ListTagsForResource.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_ListTagsForResource.html) | | Delete one or more tags | untag-resource | [https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_UntagResource.html](https://docs.aws.amazon.com/mwaa-serverless/latest/APIReference/API_UntagResource.html) | The following examples show how to tag or untag resources using the AWS CLI. **Tag an existing workflow** The following command tags an existing workflow. ------ #### [ CLI ] ``` aws mwaa-serverless tag-resource --resource-arn workflow-arn --tags your-tag-key=your-tag-value ``` ------ **Untag an existing workflow** The following command deletes a tag from an existing workflow. ------ #### [ CLI ] ``` aws mwaa-serverless untag-resource --resource-arn workflow-arn --tag-keys your-tag-key ``` ------ **List tags for a workflow** The following command lists the tags associated with an existing workflow. ------ #### [ CLI ] ``` aws mwaa-serverless list-tags-for-resource --resource-arn workflow-arn ``` ------