# MLflow Tracking Servers Tracking servers An [MLflow Tracking Server](https://mlflow.org/docs/latest/tracking.html#mlflow-tracking-server-optional) is a stand-alone HTTP server that serves multiple REST API endpoints for tracking runs and experiments. A tracking server is required to begin tracking your machine learning (ML) experiments with SageMaker AI and MLflow. You can create a tracking server through the Studio UI, or through the AWS CLI for more granular security customization. You must have the correct IAM permissions configured to create an MLflow Tracking Server. **Topics** + [ # Set up IAM permissions for MLflow ](mlflow-create-tracking-server-iam.md) + [ # Create a tracking server using Studio ](mlflow-create-tracking-server-studio.md) + [ # Create a tracking server using the AWS CLI ](mlflow-create-tracking-server-cli.md) # Set up IAM permissions for MLflow IAM permissions You must configure the necessary IAM service roles to get started with MLflow in Amazon SageMaker AI. If you create a new Amazon SageMaker AI domain to access your experiments in Studio, you can configure the necessary IAM permissions during domain setup. For more information, see [Set up MLflow IAM permissions when creating a new domain](#mlflow-create-tracking-server-iam-role-manager). To set up permissions using the IAM console, see [Create necessary IAM service roles in the IAM console](#mlflow-create-tracking-server-iam-service-roles). You must configure authorization controls for `sagemaker-mlflow` actions. You can optionally define more granular authorization controls to govern action-specific MLflow permissions. For more information, see [Create action-specific authorization controls](#mlflow-create-tracking-server-update-iam-actions). ## Set up MLflow IAM permissions when creating a new domain When setting up a new Amazon SageMaker AI domain for your organization, you can configure IAM permissions for your domain service role through the **Users and ML Activities** settings. **To configure IAM permissions for using MLflow with SageMaker AI when setting up a new domain** 1. Set up a new domain using the SageMaker AI console. On the **Set up SageMaker AI domain** page, choose **Set up for organizations**. For more information, see [Custom setup using the console](onboard-custom.md#onboard-custom-instructions-console). 1. When setting up **Users and ML Activities**, choose from the following ML activities for MLflow: **Use MLflow**, **Manage MLflow Tracking Servers**, and **Access required to AWS Services for MLflow**. For more information about these activities, see the explanations that follow this procedure. 1. Complete the setup and creation of your new domain. The following MLflow ML activities are available in Amazon SageMaker Role Manager: + **Use MLflow**: This ML activity grants the domain service role permission to call MLflow REST APIs in order to manage experiments, runs, and models in MLflow. + **Manage MLflow Tracking Servers**: This ML activity grants the domain service role permission to create, update, start, stop, and delete tracking servers. + **Access required to AWS Services for MLflow**: This ML activity provides the domain service role permissions needed to access Amazon S3 and the SageMaker AI Model Registry. This allows you to use the domain service role as the tracking server service role. For more information about ML activities in Role Manager, see [ML activity reference](role-manager-ml-activities.md). ## Create necessary IAM service roles in the IAM console If you did not create or update your domain service role, you must instead create the following service roles in the IAM console in order to create and use an MLflow Tracking Server: + A tracking server IAM service role that the tracking server can use to access SageMaker AI resources + A SageMaker AI IAM service role that SageMaker AI can use to create and manage MLflow resources ### IAM policies for the tracking server IAM service role The tracking server IAM service role is used by the tracking server to access the resources it needs such as Amazon S3 and the SageMaker Model Registry. When creating the tracking server IAM service role, use the following IAM trust policy: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "sagemaker.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] } ``` ------ In the IAM console, add the following permissions policy to your tracking server service role: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:Get*", "s3:Put*", "s3:List*", "sagemaker:AddTags", "sagemaker:CreateModelPackageGroup", "sagemaker:CreateModelPackage", "sagemaker:UpdateModelPackage", "sagemaker:DescribeModelPackageGroup" ], "Resource": "*" } ] } ``` ------ ### IAM policy for the SageMaker AI IAM service role The SageMaker AI service role is used by the client accessing the MLflow Tracking Server and needs permissions to call MLflow REST APIs. The SageMaker AI service role also needs SageMaker API permissions to create, view update, start, stop, and delete tracking servers. You can create a new role or update an existing role. The SageMaker AI service role needs the following policy: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker-mlflow:*", "sagemaker:CreateMlflowTrackingServer", "sagemaker:ListMlflowTrackingServers", "sagemaker:UpdateMlflowTrackingServer", "sagemaker:DeleteMlflowTrackingServer", "sagemaker:StartMlflowTrackingServer", "sagemaker:StopMlflowTrackingServer", "sagemaker:CreatePresignedMlflowTrackingServerUrl" ], "Resource": "*" } ] } ``` ------ ## Create action-specific authorization controls You must set up authorization controls for `sagemaker-mlflow`, and can optionally configure action-specific authorization controls to govern more granular MLflow permissions that your users have on an MLflow Tracking Server. **Note** The following steps assume that you have an ARN for an MLflow Tracking Server already available. To learn how to create a tracking server, see [Create a tracking server using Studio](mlflow-create-tracking-server-studio.md) or [Create a tracking server using the AWS CLI](mlflow-create-tracking-server-cli.md). The following command creates a file called `mlflow-policy.json` that provides your tracking server with IAM permissions for all available SageMaker AI MLflow actions. You can optionally limit the permissions a user has by choosing the specific actions you want that user to perform. For a list of available actions, see [IAM actions supported for MLflow](#mlflow-create-tracking-server-iam-actions). ``` # Replace "Resource":"*" with "Resource":"TrackingServerArn" # Replace "sagemaker-mlflow:*" with specific actions printf '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sagemaker-mlflow:*", "Resource": "*" } ] }' > mlflow-policy.json ``` Use the `mlflow-policy.json` file to create an IAM policy using the AWS CLI. ``` aws iam create-policy \ --policy-name MLflowPolicy \ --policy-document file://mlflow-policy.json ``` Retrieve your account ID and attach the policy to your IAM role. ``` # Get your account ID aws sts get-caller-identity # Attach the IAM policy using your exported role and account ID aws iam attach-role-policy \ --role-name $role_name \ --policy-arn arn:aws:iam::123456789012:policy/MLflowPolicy ``` ### IAM actions supported for MLflow The following SageMaker AI MLflow actions are supported for authorization access control: + sagemaker-mlflow:AccessUI + sagemaker-mlflow:CreateExperiment + sagemaker-mlflow:SearchExperiments + sagemaker-mlflow:GetExperiment + sagemaker-mlflow:GetExperimentByName + sagemaker-mlflow:DeleteExperiment + sagemaker-mlflow:RestoreExperiment + sagemaker-mlflow:UpdateExperiment + sagemaker-mlflow:CreateRun + sagemaker-mlflow:DeleteRun + sagemaker-mlflow:RestoreRun + sagemaker-mlflow:GetRun + sagemaker-mlflow:LogMetric + sagemaker-mlflow:LogBatch + sagemaker-mlflow:LogModel + sagemaker-mlflow:LogInputs + sagemaker-mlflow:SetExperimentTag + sagemaker-mlflow:SetTag + sagemaker-mlflow:DeleteTag + sagemaker-mlflow:LogParam + sagemaker-mlflow:GetMetricHistory + sagemaker-mlflow:SearchRuns + sagemaker-mlflow:ListArtifacts + sagemaker-mlflow:UpdateRun + sagemaker-mlflow:CreateRegisteredModel + sagemaker-mlflow:GetRegisteredModel + sagemaker-mlflow:RenameRegisteredModel + sagemaker-mlflow:UpdateRegisteredModel + sagemaker-mlflow:DeleteRegisteredModel + sagemaker-mlflow:GetLatestModelVersions + sagemaker-mlflow:CreateModelVersion + sagemaker-mlflow:GetModelVersion + sagemaker-mlflow:UpdateModelVersion + sagemaker-mlflow:DeleteModelVersion + sagemaker-mlflow:SearchModelVersions + sagemaker-mlflow:GetDownloadURIForModelVersionArtifacts + sagemaker-mlflow:TransitionModelVersionStage + sagemaker-mlflow:SearchRegisteredModels + sagemaker-mlflow:SetRegisteredModelTag + sagemaker-mlflow:DeleteRegisteredModelTag + sagemaker-mlflow:DeleteModelVersionTag + sagemaker-mlflow:DeleteRegisteredModelAlias + sagemaker-mlflow:SetRegisteredModelAlias + sagemaker-mlflow:GetModelVersionByAlias + sagemaker-mlflow:FinalizeLoggedModel + sagemaker-mlflow:GetLoggedModel + sagemaker-mlflow:DeleteLoggedModel + sagemaker-mlflow:SearchLoggedModels + sagemaker-mlflow:SetLoggedModelTags + sagemaker-mlflow:DeleteLoggedModelTag + sagemaker-mlflow:ListLoggedModelArtifacts + sagemaker-mlflow:LogLoggedModelParams + sagemaker-mlflow:LogOutputs # Create a tracking server using Studio Studio You can create a tracking server from the SageMaker Studio MLflow UI. If you created your SageMaker Studio domain following the **Set up for organizations** workflow, the service role for your SageMaker Studio domain has sufficient permissions to serve as the SageMaker AI IAM service roles and the tracking server IAM service role. Create a tracking server from the SageMaker Studio MLflow UI with the following steps: 1. Navigate to Studio from the SageMaker AI console. Be sure that you are using the new Studio experience and have updated from Studio Classic. For more information, see [Migration from Amazon SageMaker Studio Classic](studio-updated-migrate.md). 1. Choose **MLflow** in the **Applications** pane of the Studio UI. 1. **(Optional)** If have not already created a Tracking Server or if you need to create a new one, you can choose **Create**. Then provide a unique tracking server name and S3 URI for artifact storage and create a tracking server. You can optionally choose **Configure** for more granular tracking server customization. 1. Choose **Create** in the **MLflow Tracking Servers** pane. The Studio domain IAM service role is used for the tracking server IAM service role. 1. Provide a unique name for your tracking server and an Amazon S3 URI for your tracking server artifact store. Your tracking server and the Amazon S3 bucket must be in the **same AWS Region**. **Important** When you provide the Amazon S3 URI for your artifact store, ensure the Amazon S3 bucket is in the same AWS Region as your tracking server. **Cross-region artifact storage is not supported**. 1. **(Optional)** Choose **Configure** to change default settings such as tracking server size, tags, and the IAM service role. 1. Choose **Create**. **Note** It may take up to 25 minutes to complete tracking server creation. If the tracking server takes over 25 minutes to create, check that you have the necessary IAM permissions. For more information on IAM permissions, see [Set up IAM permissions for MLflow](mlflow-create-tracking-server-iam.md). When you successfully create a tracking server, it automatically starts. 1. After creating your tracking server, you can launch the MLflow UI. For more information, see [Launch the MLflow UI using a presigned URL](mlflow-launch-ui.md). ![\[The Create MLflow Tracking Server prompt in the Studio UI.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/mlflow/mlflow-studio-create.png) # Create a tracking server using the AWS CLI AWS CLI You can create a tracking server using the AWS CLI for more granular security customization. ## Prerequisites To create a tracking server using the AWS CLI, you must have the following: + **Access to a terminal. **This can include local IDEs, an Amazon EC2 instance, or AWS CloudShell. + **Access to a development environment.** This can include local IDEs or a Jupyter notebook environment within Studio or Studio Classic. + **A configured AWS CLI installation**. For more information, see [Configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). + **An IAM role with appropriate permissions**. The following steps require your environment to have `iam:CreateRole`, `iam:CreatePolicy`, `iam:AttachRolePolicy`, and `iam:ListPolicies` permissions. These permissions are needed on the role that is being used to run the steps in this user guide. The instructions in this guide create an IAM role that is used as the execution role of the MLflow Tracking Server so that it can access data in your Amazon S3 buckets. Additionally, a policy is created to give the IAM role of the user that is interacting with the Tracking Server via the MLflow SDK permission to call MLflow APIs. For more information, see [Modifying a role permissions policy (console) ](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-console.html#roles-modify_permissions-policy). If using a SageMaker Studio Notebook, update the service role for your Studio user profile with these IAM permissions. To update the service role, navigate to the SageMaker AI console and select the domain you are using. Then, under the domain, select the user profile you are using. You will see the service role listed there. Navigate to the IAM console, search for the service role under **Roles**, and update your role with a policy that allows the `iam:CreateRole`, `iam:CreatePolicy`, `iam:AttachRolePolicy`, and `iam:ListPolicies` actions. ## Set up AWS CLI model Follow these command line steps within a terminal to set up the AWS CLI for Amazon SageMaker AI with MLflow. 1. Install an updated version of the AWS CLI. For more information, see [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) in the *AWS CLI User Guide*. 1. Verify that the AWS CLI is installed using the following command: ``` aws sagemaker help ``` Press `q` to exit the prompt. For troubleshooting help, see [Troubleshoot common setup issues](mlflow-troubleshooting.md). ## Set up MLflow infrastructure The following section shows you how to set up an MLflow Tracking Server along with the Amazon S3 bucket and IAM role needed for the tracking server. ### Create an S3 bucket Within your terminal, use the following commands to create a general purpose Amazon S3 bucket: **Important** When you provide the Amazon S3 URI for your artifact store, ensure the Amazon S3 bucket is in the same AWS Region as your tracking server. **Cross-region artifact storage is not supported**. ``` bucket_name=bucket-name region=valid-region aws s3api create-bucket \ --bucket $bucket_name \ --region $region \ --create-bucket-configuration LocationConstraint=$region ``` The output should look similar to the following: ``` { "Location": "/bucket-name" } ``` ### Set up IAM trust policies Use the following steps to create an IAM trust policy. For more information about roles and trust policies, see [Roles terms and concepts](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html) in the *AWS Identity and Access Management User Guide*. 1. Within your terminal, use the following command to create a file called `mlflow-trust-policy.json`. ``` cat < /tmp/mlflow-trust-policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "sagemaker.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] } EOF ``` 1. Within your terminal, use the following command to create a file called `custom-policy.json`. ``` cat < /tmp/custom-policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:Get*", "s3:Put*", "sagemaker:AddTags", "sagemaker:CreateModelPackageGroup", "sagemaker:CreateModelPackage", "sagemaker:DescribeModelPackageGroup", "sagemaker:UpdateModelPackage", "s3:List*" ], "Resource": "*" } ] } EOF ``` 1. Use the trust policy file to create a role. Then, attach IAM role policies that allow MLflow to access Amazon S3 and SageMaker Model Registry within your account. MLflow must have access to Amazon S3 for your tracking server's artifact store and SageMaker Model Registry for automatic model registration. **Note** If you are updating an existing role, use the following command instead: `aws iam update-assume-role-policy --role-name $role_name --policy-document file:///tmp/mlflow-trust-policy.json`. ``` role_name=role-name aws iam create-role \ --role-name $role_name \ --assume-role-policy-document file:///tmp/mlflow-trust-policy.json aws iam put-role-policy \ --role-name $role_name \ --policy-name custom-policy \ --policy-document file:///tmp/custom-policy.json role_arn=$(aws iam get-role --role-name $role_name --query 'Role.Arn' --output text) ``` ## Create MLflow tracking server Within your terminal, use the `create-mlflow-tracking-server` API to create a tracking server in the AWS Region of your choice. This step can take up to 25 minutes. You can optionally specify the size of your tracking server with the parameter `--tracking-server-config`. Choose between `"Small"`, `"Medium"`, and `"Large"`. The default MLflow Tracking Server configuration size is `"Small"`. You can choose a size depending on the projected use of the tracking server such as the volume of data logged, number of users, and frequency of use. For more information, see [MLflow Tracking Server sizes](mlflow.md#mlflow-create-tracking-server-sizes). The following command creates a new tracking server with automatic model registration enabled. To deactivate automatic model registration, specify `--no-automatic-model-registration`. After creating your tracking server, you can launch the MLflow UI. For more information, see [Launch the MLflow UI using a presigned URL](mlflow-launch-ui.md). **Note** It may take up to 25 minutes to complete tracking server creation. If the tracking server takes over 25 minutes to create, check that you have the necessary IAM permissions. For more information on IAM permissions, see [Set up IAM permissions for MLflow](mlflow-create-tracking-server-iam.md). When you successfully create a tracking server, it automatically starts. When you create a tracking server, we recommend specifying the latest version. For information about the available versions, see [Tracking server versions](mlflow.md#mlflow-create-tracking-server-versions). By default, the tracking server that's created is the latest version. However, we recommend always specifying the latest version explicitly because the underlying MLflow APIs can change. ``` ts_name=tracking-server-name region=valid-region version=valid-version aws sagemaker create-mlflow-tracking-server \ --tracking-server-name $ts_name \ --artifact-store-uri s3://$bucket_name \ --role-arn $role_arn \ --automatic-model-registration \ --region $region \ --mlflow-version $version ``` The output should be similar to the following: ``` { "TrackingServerArn": "arn:aws:sagemaker:region:123456789012:mlflow-tracking-server/tracking-server-name" } ``` **Important** **Take note of the tracking server ARN for later use.** You will also need the `$bucket_name` for clean up steps.