# SageMaker Autopilot
**Important**
As of November 30, 2023, Autopilot's UI is migrating to [Amazon SageMaker Canvas](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas.html) as part of the updated [Amazon SageMaker Studio](studio-updated.md) experience. SageMaker Canvas provides analysts and citizen data scientists no-code capabilities for tasks such as data preparation, feature engineering, algorithm selection, training and tuning, inference, and more. Users can leverage built-in visualizations and what-if analysis to explore their data and different scenarios, with automated predictions enabling them to easily productionize their models. Canvas supports a variety of use cases, including computer vision, demand forecasting, intelligent search, and generative AI.
Users of [Amazon SageMaker Studio Classic](studio.md), the previous experience of [Studio](studio-updated.md), can continue using the Autopilot UI in Studio Classic. Users with coding experience can continue using all [API references](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-reference.html) in any supported SDK for technical implementation.
If you have been using Autopilot in Studio Classic until now and want to migrate to SageMaker Canvas, you might have to grant additional permissions to your user profile or IAM role so that you can create and use the SageMaker Canvas application. For more information, see [(Optional) Migrate from Autopilot in Studio Classic to SageMaker Canvas](studio-updated-migrate-ui.md#studio-updated-migrate-autopilot).
All UI-related instructions in this guide pertain to Autopilot's standalone features before migrating to [Amazon SageMaker Canvas](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas.html). Users following these instructions should use [Studio Classic](studio.md).
Amazon SageMaker Autopilot is a feature set that simplifies and accelerates various stages of the machine learning workflow by automating the process of building and deploying machine learning models (AutoML). The following page explains key information about Amazon SageMaker Autopilot.
Autopilot performs the following key tasks that you can use on autopilot or with various degrees of human guidance:
+ **Data analysis and preprocessing:** Autopilot identifies your specific problem type, handles missing values, normalizes your data, selects features, and overall prepares the data for model training.
+ **Model selection:** Autopilot explores a variety of algorithms and uses a cross-validation resampling technique to generate metrics that evaluate the predictive quality of the algorithms based on predefined objective metrics.
+ **Hyperparameter optimization:** Autopilot automates the search for optimal hyperparameter configurations.
+ **Model training and evaluation:** Autopilot automates the process of training and evaluating various model candidates. It splits the data into training and validation sets, trains the selected model candidates using the training data, and evaluates their performance on the unseen data of the validation set. Lastly, it ranks the optimized model candidates based on their performance and identifies the best performing model.
+ **Model deployment:** Once Autopilot has identified the best performing model, it provides the option to deploy the model automatically by generating the model artifacts and the endpoint exposing an API. External applications can send data to the endpoint and receive the corresponding predictions or inferences.
Autopilot supports building machine learning models on large datasets up to hundreds of GBs.
The following diagram outlines the tasks of this AutoML process managed by Autopilot.
![\[Overview of Amazon SageMaker Autopilot AutoML process.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/Autopilot-process-graphic-1.png)
Depending on your comfort level with the machine learning process and coding experience, you can use Autopilot in different ways:
+ **Using the Studio Classic UI**, users can choose between a no-code experience or have some level of human input.
**Note**
Only experiments created from tabular data for problem types such as regression or classification are available via the Studio Classic UI.
+ **Using the AutoML API**, users with coding experience can use available SDKs to create AutoML jobs. This approach provides greater flexibility and customization options and is available for all problem types.
Autopilot currently supports the following problem types:
**Note**
For regression or classification problems involving tabular data, users can choose between two options: using the Studio Classic user interface or the [API Reference](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-reference.html).
Tasks such as text and image classification, time-series forecasting, and fine-tuning of large language models are exclusively available through the version 2 of the [AutoML REST API](autopilot-reference.md). If your language of choice is Python, you can refer to [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_auto_ml_job_v2.html) or the [AutoMLV2 object](https://sagemaker.readthedocs.io/en/stable/api/training/automlv2.html#sagemaker.automl.automlv2.AutoMLV2) of the Amazon SageMaker Python SDK directly.
Users who prefer the convenience of a user interface can use [Amazon SageMaker Canvas](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-getting-started.html) to access pre-trained models and generative AI foundation models, or create custom models tailored for specific text, image classification, forecasting needs, or generative AI.
+ **Regression, binary, and multiclass classification** with tabular data formatted as CSV or Parquet files in which each column contains a feature with a specific data type and each row contains an observation. The column data types accepted include numerical, categorical, text, and time series that consists of strings of comma-separated numbers.
+ To create an Autopilot job as a pilot experiment using the SageMaker API reference, see [Create Regression or Classification Jobs for Tabular Data Using the AutoML API](autopilot-automate-model-development-create-experiment.md).
+ To create an Autopilot job as a pilot experiment using the Studio Classic UI, see [Create a Regression or Classification Autopilot experiment for tabular data using the Studio Classic UI](autopilot-automate-model-development-create-experiment-ui.md).
+ If you are an administrator looking to pre-configure default infrastructure, networking, or security parameters of Autopilot experiments in Studio Classic UI, see [Configure the default parameters of an Autopilot experiment (for administrators)](autopilot-set-default-parameters-create-experiment.md).
+ **Text classification** with data formatted as CSV or Parquet files in which a column provides the sentences to be classified, while another column should provide the corresponding class label. See [Create an AutoML job for text classification using the API](autopilot-create-experiment-text-classification.md).
+ **Image classification** with image formats such as PNG, JPEG, or a combination of both.See [Create an Image Classification Job using the AutoML API](autopilot-create-experiment-image-classification.md).
+ **Time-series forecasting** with time-series data formatted as CSV or Parquet files.See [Create an AutoML job for time-series forecasting using the API](autopilot-create-experiment-timeseries-forecasting.md).
+ Fine-tuning of large language models (LLMs) for **text generation** with data formatted as CSV or Parquet files.See [Create an AutoML job to fine-tune text generation models using the API](autopilot-create-experiment-finetune-llms.md).
Additionally, Autopilot helps users understand how models make predictions by automatically generating reports that show the importance of each individual feature. This provides transparency and insights into the factors influencing the predictions, which can be used by risk and compliance teams and external regulators. Autopilot also provides a model performance report, which encompasses a summary of evaluation metrics, a confusion matrix, various visualizations such as receiver operating characteristic curves and precision-recall curves, and more. The specific content of each report vary depending on the problem type of the Autopilot experiment.
The explainability and performance reports for the best model candidate in an Autopilot experiment are available for text, image, and tabular data classification problem types.
For tabular data use cases such as regression or classification, Autopilot offers additional visibility into how the data was wrangled and how the model candidates were selected, trained, and tuned by generating notebooks that contain the code used to explore the data and find the best performing model. These notebooks provide an interactive and exploratory environment to help you learn about the impact of various inputs or the trade-offs made in the experiments. You can experiment further with the higher performing model candidate by making your own modifications to the data exploration and candidate definition notebooks provided by Autopilot.
With Amazon SageMaker AI, you pay only for what you use. You pay for the underlying compute and storage resources within SageMaker AI or other AWS services, based on your usage. For more information about the cost of using SageMaker AI, see [Amazon SageMaker Pricing](https://aws.amazon.com/sagemaker/pricing).
**Topics**
+ [Create Regression or Classification Jobs for Tabular Data Using the AutoML API](autopilot-automate-model-development-create-experiment.md)
+ [Create an Image Classification Job using the AutoML API](autopilot-create-experiment-image-classification.md)
+ [Create an AutoML job for text classification using the API](autopilot-create-experiment-text-classification.md)
+ [Create an AutoML job for time-series forecasting using the API](autopilot-create-experiment-timeseries-forecasting.md)
+ [Create an AutoML job to fine-tune text generation models using the API](autopilot-create-experiment-finetune-llms.md)
+ [Create a Regression or Classification Autopilot experiment for tabular data using the Studio Classic UI](autopilot-automate-model-development-create-experiment-ui.md)
+ [Amazon SageMaker Autopilot example notebooks](autopilot-example-notebooks.md)
+ [Videos: Use Autopilot to automate and explore the machine learning process](autopilot-videos.md)
+ [Autopilot quotas](autopilot-quotas.md)
+ [API Reference guide for Autopilot](autopilot-reference.md)
# Create Regression or Classification Jobs for Tabular Data Using the AutoML API
You can create an Autopilot regression or classification job for tabular data programmatically by calling the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html) API action in any language supported by Autopilot or the AWS CLI. The following is a collection of mandatory and optional input request parameters for the `CreateAutoMLJobV2` API action. You can find the alternative information for the previous version of this action, `CreateAutoMLJob`. However, we recommend using `CreateAutoMLJobV2`.
For information on how this API action translates into a function in the language of your choice, see the [ See Also](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_SeeAlso) section of `CreateAutoMLJobV2` and choose an SDK. As an example, for Python users, see the full request syntax of `[create\$1auto\$1ml\$1job\$1v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_auto_ml_job_v2)` in AWS SDK for Python (Boto3).
**Note**
[CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html) and [DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html) are new versions of [CreateAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html) and [DescribeAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html) which offer backward compatibility.
We recommend using the `CreateAutoMLJobV2`. `CreateAutoMLJobV2` can manage tabular problem types identical to those of its previous version `CreateAutoMLJob`, as well as non-tabular problem types such as image or text classification, or time-series forecasting.
At a minimum, all experiments on tabular data require the specification of the experiment name, providing locations for the input and output data, and specifying which target data to predict. Optionally, you can also specify the type of problem that you want to solve (regression, classification, multiclass classification), choose your modeling strategy (*stacked ensembles* or *hyperparameters optimization*), select the list of algorithms used by the Autopilot job to train the data, and more.
After the experiment runs, you can compare trials and delve into the details of the pre-processing steps, algorithms, and hyperparameter ranges of each model. You also have the option to download their [explainability](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-explainability.html) and [performance](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-model-insights.html) reports. Use the provided [ notebooks](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-automate-model-development-notebook-output.html ) to see the results of the automated data exploration or the candidate model definitions.
Find guidelines on how to migrate a `CreateAutoMLJob` to `CreateAutoMLJobV2` in [Migrate a CreateAutoMLJob to CreateAutoMLJobV2](#autopilot-create-experiment-api-migrate-v1-v2).
## Required parameters
------
#### [ CreateAutoMLJobV2 ]
When calling `[CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html)` to create an Autopilot experiment for tabular data, you must provide the following values:
+ An `[AutoMLJobName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_RequestSyntax)` to specify the name of your job.
+ At least one `[AutoMLJobChannel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html)` in `[AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig)` to specify your data source.
+ Both an `[AutoMLJobObjective](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobObjective)` metric and your chosen type of supervised learning problem (binary classification, multiclass classification, regression) in `AutoMLProblemTypeConfig`, or none at all. For tabular data, you must choose `[TabularJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)` as the type of `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)`. You set the supervised learning problem in the `ProblemType` attribute of `TabularJobConfig`.
+ An `[OutputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html)` to specify the Amazon S3 output path to store the artifacts of your AutoML job.
+ A `[RoleArn](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-RoleArn)` to specify the ARN of the role used to access your data.
------
#### [ CreateAutoMLJob ]
When calling `[CreateAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html)` to create an AutoML experiment, you must provide the following four values:
+ An `[AutoMLJobName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-AutoMLJobName)` to specify the name of your job.
+ At least one `[AutoMLChannel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLChannel.html)` in `[InputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-InputDataConfig)` to specify your data source.
+ An `[OutputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html)` to specify the Amazon S3 output path to store the artifacts of your AutoML job.
+ A `[RoleArn](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-RoleArn)` to specify the ARN of the role used to access your data.
------
All other parameters are optional.
## Optional parameters
The following sections provide details of some optional parameters that you can pass to your `CreateAutoMLJobV2` API action when using tabular data. You can find the alternative information for the previous version of this action, `CreateAutoMLJob`. However, we recommend using `CreateAutoMLJobV2`.
### How to set the training mode of an AutoML job
For tabular data, the set of algorithms run on your data to train your model candidates is dependent on your modeling strategy (`ENSEMBLING` or `HYPERPARAMETER_TUNING`). The following details how to set this training mode.
If you keep blank (or `null`), the `Mode` is inferred based on the size of your dataset.
For information on Autopilot's *stacked ensembles* and *hyperparameters optimization* training methods, see [Training modes and algorithm support](autopilot-model-support-validation.md)
------
#### [ CreateAutoMLJobV2 ]
For tabular data, you must choose `[TabularJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)` as the type of `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)`.
You can set the [training method](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-model-support-validation.html) of an AutoML job V2 with the `[TabularJobConfig.Mode](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)` parameter.
------
#### [ CreateAutoMLJob ]
You can set the [training method](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-model-support-validation.html) of an AutoML job with the `[AutoMLJobConfig.Mode](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobConfig.html#sagemaker-Type-AutoMLJobConfig-Mode)` parameter.
------
### How to select features and algorithms for training an AutoML job
#### Features selection
Autopilot provides automatic data-preprocessing steps including feature selection and feature extraction. However, you can manually provide the features to be used in training with the `FeatureSpecificatioS3Uri` attribute.
Selected features should be contained within a JSON file in the following format:
```
{ "FeatureAttributeNames":["col1", "col2", ...] }
```
The values listed in `["col1", "col2", ...]` are case sensitive. They should be a list of strings containing unique values that are subsets of the column names in the input data.
**Note**
The list of columns provided as features cannot include the target column.
------
#### [ CreateAutoMLJobV2 ]
For tabular data, you must choose `[TabularJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)` as the type of `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)`.
You can set the URL to your selected features with the `[TabularJobConfig.FeatureSpecificationS3Uri](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)` parameter.
------
#### [ CreateAutoMLJob ]
You can set the `FeatureSpecificatioS3Uri` attribute of [AutoMLCandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidateGenerationConfig.html) within the [CreateAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html) API with the following format:
```
{
"[AutoMLJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-AutoMLJobConfig)": {
"[CandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobConfig.html#sagemaker-Type-AutoMLJobConfig-CandidateGenerationConfig)": {
"[FeatureSpecificationS3Uri](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidateGenerationConfig.html#sagemaker-Type-AutoMLCandidateGenerationConfig-FeatureSpecificationS3Uri)":"string"
},
}
}
```
------
#### Algorithms selection
By default, your Autopilot job runs a pre-defined list of algorithms on your dataset to train model candidates. The list of algorithms depends on the training mode (`ENSEMBLING` or `HYPERPARAMETER_TUNING`) used by the job.
You can provide a subset of the default selection of algorithms.
------
#### [ CreateAutoMLJobV2 ]
For tabular data, you must choose `[TabularJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)` as the type of `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)`.
You can specify an array of selected `AutoMLAlgorithms` in the `AlgorithmsConfig` attribute of [CandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateGenerationConfig.html).
The following is an example of an `AlgorithmsConfig` attribute listing exactly three algorithms ("xgboost", "fastai", "catboost") in its `AutoMLAlgorithms` field for the ensembling training mode.
```
{
"[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)": {
"[TabularJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)": {
"[Mode](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)": "ENSEMBLING",
"[CandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateGenerationConfig.html)": {
"[AlgorithmsConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateGenerationConfig.html#sagemaker-Type-CandidateGenerationConfig-AlgorithmsConfig)":[
{"[AutoMLAlgorithms](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLAlgorithmConfig.html)":["xgboost", "fastai", "catboost"]}
]
},
},
},
}
```
------
#### [ CreateAutoMLJob ]
You can specify an array of selected `AutoMLAlgorithms` in the `AlgorithmsConfig` attribute of [AutoMLCandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidateGenerationConfig.html).
The following is an example of an `AlgorithmsConfig` attribute listing exactly three algorithms ("xgboost", "fastai", "catboost") in its `AutoMLAlgorithms` field for the ensembling training mode.
```
{
"[AutoMLJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-AutoMLJobConfig)": {
"[CandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobConfig.html#sagemaker-Type-AutoMLJobConfig-CandidateGenerationConfig)": {
"[AlgorithmsConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidateGenerationConfig.html#sagemaker-Type-AutoMLCandidateGenerationConfig-AlgorithmsConfig)":[
{"[AutoMLAlgorithms](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLAlgorithmConfig.html#sagemaker-Type-AutoMLAlgorithmConfig-AutoMLAlgorithms)":["xgboost", "fastai", "catboost"]}
]
},
"Mode": "ENSEMBLING"
}
```
------
For the list of available algorithms per training `Mode`, see [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLAlgorithmConfig.html#sagemaker-Type-AutoMLAlgorithmConfig-AutoMLAlgorithms](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLAlgorithmConfig.html#sagemaker-Type-AutoMLAlgorithmConfig-AutoMLAlgorithms). For details on each algorithm, see [Training modes and algorithm support](autopilot-model-support-validation.md).
### How to specify the training and validation datasets of an AutoML job
You can provide your own validation dataset and custom data split ratio, or let Autopilot split the dataset automatically.
------
#### [ CreateAutoMLJobV2 ]
Each [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html) object (see the required parameter [AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker-api/src/AWSSageMakerAPIDoc/build/server-root/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig)) has a `ChannelType`, which can be set to either `training` or `validation` values that specify how the data is to be used when building a machine learning model. At least one data source must be provided and a maximum of two data sources is allowed: one for training data and one for validation data.
How you split the data into training and validation datasets depends on whether you have one or two data sources.
+ If you only have **one data source**, the `ChannelType` is set to `training` by default and must have this value.
+ If the `ValidationFraction` value in [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html) is not set, 0.2 (20%) of the data from this source is used for validation by default.
+ If the `ValidationFraction` is set to a value between 0 and 1, the dataset is split based on the value specified, where the value specifies the fraction of the dataset used for validation.
+ If you have **two data sources**, the `ChannelType` of one of the `AutoMLJobChannel` objects must be set to `training`, the default value. The `ChannelType` of the other data source must be set to `validation`. The two data sources must have the same format, either CSV or Parquet, and the same schema. You must not set the value for the `ValidationFraction` in this case because all of the data from each source is used for either training or validation. Setting this value causes an error.
------
#### [ CreateAutoMLJob ]
Each [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLChannel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLChannel.html) object (see the required parameter [InputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-InputDataConfig)) has a `ChannelType`, which can be set to either `training` or `validation` values that specify how the data is to be used when building a machine learning model. At least one data source must be provided and a maximum of two data sources is allowed: one for training data and one for validation data.
How you split the data into training and validation datasets depends on whether you have one or two data sources.
+ If you only have **one data source**, the `ChannelType` is set to `training` by default and must have this value.
+ If the `ValidationFraction` value in [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html) is not set, 0.2 (20%) of the data from this source is used for validation by default.
+ If the `ValidationFraction` is set to a value between 0 and 1, the dataset is split based on the value specified, where the value specifies the fraction of the dataset used for validation.
+ If you have **two data sources**, the `ChannelType` of one of the `AutoMLChannel` objects must be set to `training`, the default value. The `ChannelType` of the other data source must be set to `validation`. The two data sources must have the same format, either CSV or Parquet, and the same schema. You must not set the value for the `ValidationFraction` in this case because all of the data from each source is used for either training or validation. Setting this value causes an error.
------
For information on split and cross-validation in Autopilot see [Cross-validation in Autopilot](autopilot-metrics-validation.md#autopilot-cross-validation).
### How to set the problem type of an AutoML job
------
#### [ CreateAutoMLJobV2 ]
For tabular data, you must choose `[TabularJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)` as the type of `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)`.
You can further specify the type of supervised learning problem (binary classification, multiclass classification, regression) available for the model candidates of your AutoML job V2 with the `[TabularJobConfig.ProblemType](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)` parameter.
------
#### [ CreateAutoMLJob ]
You can set the [type of problem](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-datasets-problem-types.html#autopilot-problem-types) on an AutoML job with the `[CreateAutoPilot.ProblemType](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-ProblemType)` parameter. This limits the kind of preprocessing and algorithms that Autopilot tries. After the job is finished, if you had set the `[CreateAutoPilot.ProblemType](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-ProblemType)`, then the `[ResolvedAttribute.ProblemType](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ResolvedAttributes.html)` matches the `ProblemType` you set. If you keep it blank (or `null`), the `ProblemType` is inferred on your behalf.
------
**Note**
In some cases, Autopilot is unable to infer the `ProblemType` with high enough confidence, in which case you must provide the value for the job to succeed.
### How to add sample weights to an AutoML job
You can add a sample weights column to your tabular dataset and then pass it to your AutoML job to request dataset rows to be weighted during training and evaluation.
Support for sample weights is available in [ensembling mode](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-model-support-validation.html#autopilot-training-mode) only. Your weights should be numeric and non-negative. Data points with invalid or no weight value are excluded. For more information on the available objective metrics, see [Autopilot weighted metrics](autopilot-metrics-validation.md#autopilot-weighted-metrics).
------
#### [ CreateAutoMLJobV2 ]
For tabular data, you must choose `[TabularJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)` as the type of `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)`.
To set sample weights when creating an experiment (see [CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html)), you can pass the name of your sample weights column in the `SampleWeightAttributeName` attribute of the `TabularJobConfig` object. This ensures that your objective metric uses the weights for the training, evaluation, and selection of model candidates.
------
#### [ CreateAutoMLJob ]
To set sample weights when creating an experiment (see [CreateAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html)), you can pass the name of your sample weights column in the `SampleWeightAttributeName` attribute of the [AutoMLChannel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLChannel.html) object. This ensures that your objective metric uses the weights for the training, evaluation, and selection of model candidates.
------
### How to configure AutoML to initiate a remote job on EMR Serverless for large datasets
You can configure your AutoML job V2 to automatically initiate a remote job on Amazon EMR Serverless when additional compute resources are needed to process large datasets. By seamlessly transitioning to EMR Serverless when required, the AutoML job can handle datasets that would otherwise exceed the initially provisioned resources, without any manual intervention from you. EMR Serverless is available for the tabular and time series problem types. We recommend setting up this option for tabular datasets larger than 5 GB.
To allow your AutoML job V2 to automatically transition to EMR Serverless for large dataset, you need to provide an `EmrServerlessComputeConfig` object, which includes an `ExecutionRoleARN` field, to the `AutoMLComputeConfig` of the AutoML job V2 input request.
The `ExecutionRoleARN` is the ARN of the IAM role granting the AutoML job V2 the necessary permissions to run EMR Serverless jobs.
This role should have the following trust relationship:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "emr-serverless.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
------
And grant the permissions to:
+ Create, list, and update EMR Serverless applications.
+ Start, list, get, or cancel job runs on an EMR Serverless application.
+ Tag EMR Serverless resources.
+ Pass an IAM role to the EMR Serverless service for execution.
By granting the `iam:PassRole` permission, the AutoML job V2 can temporarily assume the `EMRServerlessRuntimeRole-*` role and pass it to the EMR Serverless service. These are the IAM roles used by the EMR Serverless job execution environments to access other AWS services and resources needed during runtime, such as Amazon S3 for data access, CloudWatch for logging, access to the AWS Glue Data Catalog or other services based on your workload requirements.
See [Job runtime roles for Amazon EMR Serverless](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/security-iam-runtime-role.html) for details on this role permissions.
The IAM policy defined in the provided JSON document grants those permissions:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [{
"Sid": "EMRServerlessCreateApplicationOperation",
"Effect": "Allow",
"Action": "emr-serverless:CreateApplication",
"Resource": "arn:aws:emr-serverless:*:*:/*",
"Condition": {
"StringEquals": {
"aws:RequestTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessListApplicationOperation",
"Effect": "Allow",
"Action": "emr-serverless:ListApplications",
"Resource": "arn:aws:emr-serverless:*:*:/*",
"Condition": {
"StringEquals": {
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessApplicationOperations",
"Effect": "Allow",
"Action": [
"emr-serverless:UpdateApplication",
"emr-serverless:GetApplication"
],
"Resource": "arn:aws:emr-serverless:*:*:/applications/*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessStartJobRunOperation",
"Effect": "Allow",
"Action": "emr-serverless:StartJobRun",
"Resource": "arn:aws:emr-serverless:*:*:/applications/*",
"Condition": {
"StringEquals": {
"aws:RequestTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessListJobRunOperation",
"Effect": "Allow",
"Action": "emr-serverless:ListJobRuns",
"Resource": "arn:aws:emr-serverless:*:*:/applications/*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessJobRunOperations",
"Effect": "Allow",
"Action": [
"emr-serverless:GetJobRun",
"emr-serverless:CancelJobRun"
],
"Resource": "arn:aws:emr-serverless:*:*:/applications/*/jobruns/*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessTagResourceOperation",
"Effect": "Allow",
"Action": "emr-serverless:TagResource",
"Resource": "arn:aws:emr-serverless:*:*:/*",
"Condition": {
"StringEquals": {
"aws:RequestTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "IAMPassOperationForEMRServerless",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/EMRServerlessRuntimeRole-*",
"Condition": {
"StringEquals": {
"iam:PassedToService": "emr-serverless.amazonaws.com",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
}
]
}
```
------
## Migrate a CreateAutoMLJob to CreateAutoMLJobV2
We recommend users of `CreateAutoMLJob` to migrate to `CreateAutoMLJobV2`.
This section explains the differences in the input parameters between [CreateAutoMLJob](https://docs.aws.amazon.com/sagemaker-api/src/AWSSageMakerAPIDoc/build/server-root/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#API_CreateAutoMLJob_RequestSyntax) and [CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_RequestSyntax) by highlighting the changes in the position, name, or structure of the objects and attributes of the input request between the two versions.
+ **Request attributes that did not change between versions.**
```
{
"AutoMLJobName": "string",
"AutoMLJobObjective": {
"MetricName": "string"
},
"ModelDeployConfig": {
"AutoGenerateEndpointName": boolean,
"EndpointName": "string"
},
"OutputDataConfig": {
"KmsKeyId": "string",
"S3OutputPath": "string"
},
"RoleArn": "string",
"Tags": [
{
"Key": "string",
"Value": "string"
}
]
}
```
+ **Request attributes that changed position and structure between versions.**
The following attributes changed position: `DataSplitConfig`, `Security Config`, `CompletionCriteria`, `Mode`, `FeatureSpecificationS3Uri`, `SampleWeightAttributeName`, `TargetAttributeName`.
------
#### [ CreateAutoMLJob ]
```
{
"AutoMLJobConfig": {
"Mode": "string",
"CompletionCriteria": {
"MaxAutoMLJobRuntimeInSeconds": number,
"MaxCandidates": number,
"MaxRuntimePerTrainingJobInSeconds": number
},
"DataSplitConfig": {
"ValidationFraction": number
},
"SecurityConfig": {
"EnableInterContainerTrafficEncryption": boolean,
"VolumeKmsKeyId": "string",
"VpcConfig": {
"SecurityGroupIds": [ "string" ],
"Subnets": [ "string" ]
}
},
"CandidateGenerationConfig": {
"FeatureSpecificationS3Uri": "string"
}
},
"GenerateCandidateDefinitionsOnly": boolean,
"ProblemType": "string"
}
```
------
#### [ CreateAutoMLJobV2 ]
```
{
"AutoMLProblemTypeConfig": {
"TabularJobConfig": {
"Mode": "string",
"ProblemType": "string",
"GenerateCandidateDefinitionsOnly": boolean,
"CompletionCriteria": {
"MaxAutoMLJobRuntimeInSeconds": number,
"MaxCandidates": number,
"MaxRuntimePerTrainingJobInSeconds": number
},
"FeatureSpecificationS3Uri": "string",
"SampleWeightAttributeName": "string",
"TargetAttributeName": "string"
}
},
"DataSplitConfig": {
"ValidationFraction": number
},
"SecurityConfig": {
"EnableInterContainerTrafficEncryption": boolean,
"VolumeKmsKeyId": "string",
"VpcConfig": {
"SecurityGroupIds": [ "string" ],
"Subnets": [ "string" ]
}
}
}
```
------
+ **The following attributes changed position and structure between versions.**
The following JSON illustrates how [AutoMLJobConfig.CandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobConfig.html#sagemaker-Type-AutoMLJobConfig-CandidateGenerationConfig) of type [AutoMLCandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidateGenerationConfig.html) moved to [AutoMLProblemTypeConfig.TabularJobConfig.CandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_RequestSyntax) of type [CandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateGenerationConfig.html) in V2.
------
#### [ CreateAutoMLJob ]
```
{
"AutoMLJobConfig": {
"CandidateGenerationConfig": {
"AlgorithmsConfig": [
{
"AutoMLAlgorithms": [ "string" ]
}
],
"FeatureSpecificationS3Uri": "string"
}
}
```
------
#### [ CreateAutoMLJobV2 ]
```
{
"AutoMLProblemTypeConfig": {
"TabularJobConfig": {
"CandidateGenerationConfig": {
"AlgorithmsConfig": [
{
"AutoMLAlgorithms": [ "string" ]
}
],
},
}
},
}
```
------
+ **Request attributes that changed name and structure.**
The following JSON illustrates how [InputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-InputDataConfig) (An array of [AutoMLChannel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLChannel.html)) changed to [AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig) (An array of [AutoMLJobChannel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html)) in V2. Note that the attributes `SampleWeightAttributeName` and `TargetAttributeName` move out of `InputDataConfig` and into `AutoMLProblemTypeConfig`.
------
#### [ CreateAutoMLJob ]
```
{
"InputDataConfig": [
{
"ChannelType": "string",
"CompressionType": "string",
"ContentType": "string",
"DataSource": {
"S3DataSource": {
"S3DataType": "string",
"S3Uri": "string"
}
},
"SampleWeightAttributeName": "string",
"TargetAttributeName": "string"
}
]
}
```
------
#### [ CreateAutoMLJobV2 ]
```
{
"AutoMLJobInputDataConfig": [
{
"ChannelType": "string",
"CompressionType": "string",
"ContentType": "string",
"DataSource": {
"S3DataSource": {
"S3DataType": "string",
"S3Uri": "string"
}
}
}
]
}
```
------
# Autopilot datasets and problem types
For tabular data (that is data in which each column contains a feature with a specific data type and each row contains an observation), Autopilot gives you the option of specifying the type of supervised learning problem available for the model candidates of the AutoML job, such as binary classification or regression, or of detecting it on your behalf based on the data you provide. Autopilot also supports multiple data formats and data types.
**Topics**
+ [Autopilot datasets, data types, and formats](#autopilot-datasets)
+ [Autopilot problem types](#autopilot-problem-types)
## Autopilot datasets, data types, and formats
Autopilot supports tabular data formatted as CSV files or as Parquet files: each column contains a feature with a specific data type and each row contains an observation. The properties of these two file formats differ considerably.
+ **CSV** (comma-separated-values) is a row-based file format that stores data in human readable plaintext which a popular choice for data exchange as they are supported by a wide range of applications.
+ **Parquet** is a column-based file format where the data is stored and processed more efficiently than row-based file formats. This makes them a better option for big data problems.
The **data types** accepted for columns include numerical, categorical, text, and time series that consists of strings of comma-separated numbers. If Autopilot detects it is dealing with** time series** sequences, it processes them through specialized feature transformers provided by the [tsfresh](https://tsfresh.readthedocs.io/en/latest/text/list_of_features.html) library. This library takes the time series as an input and outputs a feature such as the highest absolute value of the time series or descriptive statistics on autocorrelation. These outputted features are then used as inputs to one of the three problem types.
Autopilot supports building machine learning models on large datasets up to hundreds of GBs. For details on the default resource limits for input datasets and how to increase them, see [Autopilot quotas](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-quotas.html).
## Autopilot problem types
For the tabular data, you further specify the type of supervised learning problems available for the model candidates as follows:
### Regression
Regression estimates the values of a dependent target variable based on one or more other variables or attributes that are correlated with it. An example is the prediction of house prices using features like the number of bathrooms and bedrooms, square footage of the house and garden. Regression analysis can create a model that takes one or more of these features as an input and predicts the price of a house.
### Binary classification
Binary classification is a type of supervised learning that assigns an individual to one of two predefined and mutually exclusive classes based on their attributes. It is supervised because the models are trained using examples where the attributes are provided with correctly labeled objects. A medical diagnosis for whether an individual has a disease or not based on the results of diagnostic tests is an example of binary classification.
### Multiclass classification
Multiclass classification is a type of supervised learning that assigns an individual to one of several classes based on their attributes. It is supervised because the models are trained using examples where the attributes are provided with correctly labelled objects. An example is the prediction of the topic most relevant to a text document. A document may be classified as being about, say, religion or politics or finance, or about one of several other predefined topic classes.
# Training modes and algorithm support
Autopilot supports different training modes and algorithms to address machine learning problems, report on quality and objective metrics, and to use cross-validation automatically, when needed.
## Training modes
SageMaker Autopilot can automatically select the training method based on the dataset size, or you can select it manually. The choices are as follows:
+ **Ensembling** – Autopilot uses the [AutoGluon](https://auto.gluon.ai/scoredebugweight/tutorials/tabular_prediction/index.html) library to train several base models. To find the best combination for your dataset, ensemble mode runs 10 trials with different model and meta parameter settings. Then Autopilot combines these models using a stacking ensemble method to create an optimal predictive model. For a list of algorithms that Autopilot supports in ensembling mode for tabular data, see the following **Algorithms support** section.
+ **Hyperparameter optimization (HPO)** – Autopilot finds the best version of a model by tuning hyperparameters using Bayesian optimization or multi-fidelity optimization while running training jobs on your dataset. HPO mode selects the algorithms that are most relevant to your dataset and selects the best range of hyperparameters to tune your models. To tune your models, HPO mode runs up to 100 trials (default) to find the optimal hyperparameters settings within the selected range. If your dataset size is less than 100 MB, Autopilot uses Bayesian optimization. Autopilot chooses multi-fidelity optimization if your dataset is larger than 100 MB.
In multi-fidelity optimization, metrics are continuously emitted from the training containers. A trial that is performing poorly against a selected objective metric is stopped early. A trial that is performing well is allocated more resources.
For a list of algorithms that Autopilot supports in HPO mode, see the following **Algorithm support** section.
+ **Auto** – Autopilot automatically chooses either ensembling mode or HPO mode based on your dataset size. If your dataset is larger than 100 MB, Autopilot chooses HPO. Otherwise, it chooses ensembling mode. Autopilot can fail to read the size of your dataset in the following cases.
+ If you enable Virtual Private Cloud (VPC) mode, for an AutoML job but the S3 bucket containing the dataset only allows access from the VPC.
+ The input [S3DataType](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLS3DataSource.html#sagemaker-Type-AutoMLS3DataSource-S3DataType) of your dataset is a `ManifestFile`.
+ The input [S3Uri](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLS3DataSource.html#sagemaker-Type-AutoMLS3DataSource-S3Uri) contains more than 1000 items.
If Autopilot is unable to read your dataset size, it defaults to choosing HPO mode.
**Note**
For optimal runtime and performance, use ensemble training mode for datasets that are smaller than 100 MB.
## Algorithms support
In **HPO mode**, Autopilot supports the following types of machine learning algorithms:
+ [Linear learner](https://docs.aws.amazon.com/sagemaker/latest/dg/linear-learner.html) – A supervised learning algorithm that can solve either classification or regression problems.
+ [XGBoost](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) – A supervised learning algorithm that attempts to accurately predict a target variable by combining an ensemble of estimates from a set of simpler and weaker models.
+ Deep learning algorithm – A multilayer perceptron (MLP) and feedforward artificial neural network. This algorithm can handle data that is not linearly separable.
**Note**
You don't need to specify an algorithm to use for your machine learning problem. Autopilot automatically selects the appropriate algorithm to train.
In **ensembling mode**, Autopilot supports the following types of machine learning algorithms:
+ [LightGBM](https://docs.aws.amazon.com/sagemaker/latest/dg/lightgbm.html) – An optimized framework that uses tree-based algorithms with gradient boosting. This algorithm uses trees that grow in breadth, rather than depth, and is highly optimized for speed.
+ [CatBoost](https://docs.aws.amazon.com/sagemaker/latest/dg/catboost.html) – A framework that uses tree-based algorithms with gradient boosting. Optimized for handling categorical variables.
+ [XGBoost](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) – A framework that uses tree-based algorithms with gradient boosting that grows in depth, rather than breadth.
+ [Random Forest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) – A tree-based algorithm that uses several decision trees on random sub-samples of the data with replacement. The trees are split into optimal nodes at each level. The decisions of each tree are averaged together to prevent overfitting and improve predictions.
+ [Extra Trees](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier) – A tree-based algorithm that uses several decision trees on the entire dataset. The trees are split randomly at each level. The decisions of each tree are averaged to prevent overfitting and to improve predictions. Extra trees add a degree of randomization in comparison to the random forest algorithm.
+ [Linear Models](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model) – A framework that uses a linear equation to model the relationship between two variables in observed data.
+ Neural network torch – A neural network model that's implemented using [Pytorch](https://pytorch.org/).
+ Neural network fast.ai – A neural network model that's implemented using [fast.ai](https://www.fast.ai/).
# Metrics and validation
This guide shows metrics and validation techniques that you can use to measure machine learning model performance. Amazon SageMaker Autopilot produces metrics that measure the predictive quality of machine learning model candidates. The metrics calculated for candidates are specified using an array of [MetricDatum](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_MetricDatum.html) types.
## Autopilot metrics
The following list contains the names of the metrics that are currently available to measure model performance within Autopilot.
**Note**
Autopilot supports sample weights. To learn more about sample weights and the available objective metrics, see [Autopilot weighted metrics](#autopilot-weighted-metrics).
The following are the available metrics.
**`Accuracy`**
The ratio of the number of correctly classified items to the total number of (correctly and incorrectly) classified items. It is used for both binary and multiclass classification. Accuracy measures how close the predicted class values are to the actual values. Values for accuracy metrics vary between zero (0) and one (1). A value of 1 indicates perfect accuracy, and 0 indicates perfect inaccuracy.
**`AUC`**
The area under the curve (AUC) metric is used to compare and evaluate binary classification by algorithms that return probabilities, such as logistic regression. To map the probabilities into classifications, these are compared against a threshold value.
The relevant curve is the receiver operating characteristic curve. The curve plots the true positive rate (TPR) of predictions (or recall) against the false positive rate (FPR) as a function of the threshold value, above which a prediction is considered positive. Increasing the threshold results in fewer false positives, but more false negatives.
AUC is the area under this receiver operating characteristic curve. Therefore, AUC provides an aggregated measure of the model performance across all possible classification thresholds. AUC scores vary between 0 and 1. A score of 1 indicates perfect accuracy, and a score of one half (0.5) indicates that the prediction is not better than a random classifier.
**`BalancedAccuracy`**
`BalancedAccuracy` is a metric that measures the ratio of accurate predictions to all predictions. This ratio is calculated after normalizing true positives (TP) and true negatives (TN) by the total number of positive (P) and negative (N) values. It is used in both binary and multiclass classification and is defined as follows: 0.5\$1((TP/P)\$1(TN/N)), with values ranging from 0 to 1. `BalancedAccuracy` gives a better measure of accuracy when the number of positives or negatives differ greatly from each other in an imbalanced dataset, such as when only 1% of email is spam.
**`F1`**
The `F1` score is the harmonic mean of the precision and recall, defined as follows: F1 = 2 \$1 (precision \$1 recall) / (precision \$1 recall). It is used for binary classification into classes traditionally referred to as positive and negative. Predictions are said to be true when they match their actual (correct) class, and false when they do not.
Precision is the ratio of the true positive predictions to all positive predictions, and it includes the false positives in a dataset. Precision measures the quality of the prediction when it predicts the positive class.
Recall (or sensitivity) is the ratio of the true positive predictions to all actual positive instances. Recall measures how completely a model predicts the actual class members in a dataset.
F1 scores vary between 0 and 1. A score of 1 indicates the best possible performance, and 0 indicates the worst.
**`F1macro`**
The `F1macro` score applies F1 scoring to multiclass classification problems. It does this by calculating the precision and recall, and then taking their harmonic mean to calculate the F1 score for each class. Lastly, the `F1macro` averages the individual scores to obtain the `F1macro` score. `F1macro` scores vary between 0 and 1. A score of 1 indicates the best possible performance, and 0 indicates the worst.
**`InferenceLatency`**
Inference latency is the approximate amount of time between making a request for a model prediction to receiving it from a real time endpoint to which the model is deployed. This metric is measured in seconds and only available in ensembling mode.
**`LogLoss`**
Log loss, also known as cross-entropy loss, is a metric used to evaluate the quality of the probability outputs, rather than the outputs themselves. It is used in both binary and multiclass classification and in neural nets. It is also the cost function for logistic regression. Log loss is an important metric to indicate when a model makes incorrect predictions with high probabilities. Values range from 0 to infinity. A value of 0 represents a model that perfectly predicts the data.
**`MAE`**
The mean absolute error (MAE) is a measure of how different the predicted and actual values are, when they're averaged over all values. MAE is commonly used in regression analysis to understand model prediction error. If there is linear regression, MAE represents the average distance from a predicted line to the actual value. MAE is defined as the sum of absolute errors divided by the number of observations. Values range from 0 to infinity, with smaller numbers indicating a better model fit to the data.
**`MSE`**
The mean squared error (MSE) is the average of the squared differences between the predicted and actual values. It is used for regression. MSE values are always positive. The better a model is at predicting the actual values, the smaller the MSE value is.
**`Precision`**
Precision measures how well an algorithm predicts the true positives (TP) out of all of the positives that it identifies. It is defined as follows: Precision = TP/(TP\$1FP), with values ranging from zero (0) to one (1), and is used in binary classification. Precision is an important metric when the cost of a false positive is high. For example, the cost of a false positive is very high if an airplane safety system is falsely deemed safe to fly. A false positive (FP) reflects a positive prediction that is actually negative in the data.
**`PrecisionMacro`**
The precision macro computes precision for multiclass classification problems. It does this by calculating precision for each class and averaging scores to obtain precision for several classes. `PrecisionMacro` scores range from zero (0) to one (1). Higher scores reflect the model's ability to predict true positives (TP) out of all of the positives that it identifies, averaged across multiple classes.
**`R2`**
R2, also known as the coefficient of determination, is used in regression to quantify how much a model can explain the variance of a dependent variable. Values range from one (1) to negative one (-1). Higher numbers indicate a higher fraction of explained variability. `R2` values close to zero (0) indicate that very little of the dependent variable can be explained by the model. Negative values indicate a poor fit and that the model is outperformed by a constant function. For linear regression, this is a horizontal line.
**`Recall`**
Recall measures how well an algorithm correctly predicts all of the true positives (TP) in a dataset. A true positive is a positive prediction that is also an actual positive value in the data. Recall is defined as follows: Recall = TP/(TP\$1FN), with values ranging from 0 to 1. Higher scores reflect a better ability of the model to predict true positives (TP) in the data. It is used in binary classification.
Recall is important when testing for cancer because it's used to find all of the true positives. A false negative (FN) reflects a negative prediction that is actually positive in the data. It is often insufficient to measure only recall, because predicting every output as a true positive yields a perfect recall score.
**`RecallMacro`**
The `RecallMacro` computes recall for multiclass classification problems by calculating recall for each class and averaging scores to obtain recall for several classes. `RecallMacro` scores range from 0 to 1. Higher scores reflect the model's ability to predict true positives (TP) in a dataset, whereas a true positive reflects a positive prediction that is also an actual positive value in the data. It is often insufficient to measure only recall, because predicting every output as a true positive will yield a perfect recall score.
**`RMSE`**
Root mean squared error (RMSE) measures the square root of the squared difference between predicted and actual values, and is averaged over all values. It is used in regression analysis to understand model prediction error. It's an important metric to indicate the presence of large model errors and outliers. Values range from zero (0) to infinity, with smaller numbers indicating a better model fit to the data. RMSE is dependent on scale, and should not be used to compare datasets of different sizes.
Metrics that are automatically calculated for a model candidate are determined by the type of problem being addressed.
Refer to the [ Amazon SageMaker API reference documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobObjective.html) for the list of available metrics supported by Autopilot.
## Autopilot weighted metrics
**Note**
Autopilot supports sample weights in ensembling mode only for all [available metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-metrics-validation.html#autopilot-metrics) with the exception of `Balanced Accuracy` and `InferenceLatency`. `BalanceAccuracy` comes with its own weighting scheme for imbalanced datasets that does not require sample weights. `InferenceLatency` does not support sample weights. Both objective `Balanced Accuracy` and `InferenceLatency` metrics ignore any existing sample weights when training and evaluating a model.
Users can add a sample weights column to their data to ensure that each observation used to train a machine learning model is given a weight corresponding to its perceived importance to the model. This is especially useful in scenarios in which the observations in the dataset have varying degrees of importance, or in which a dataset contains a disproportionate number of samples from one class compared to others. Assigning a weight to each observation based on its importance or greater importance to a minority class can help a model’s overall performance, or ensure that a model is not biased toward the majority class.
For information about how to pass sample weights when creating an experiment in the Studio Classic UI, see *Step 7* in [Create an Autopilot experiment using Studio Classic](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-automate-model-development-create-experiment.html).
For information about how to pass sample weights programmatically when creating an Autopilot experiment using the API, see *How to add sample weights to an AutoML job* in [Create an Autopilot experiment programmatically](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-automate-model-development-create-experiment.html).
## Cross-validation in Autopilot
Cross-validation is used in to reduce overfitting and bias in model selection. It is also used to assess how well a model can predict the values of an unseen validation dataset, if the validation dataset is drawn from the same population. This method is especially important when training on datasets that have a limited number of training instances.
Autopilot uses cross-validation to build models in hyperparameter optimization (HPO) and ensemble training mode. The first step in the Autopilot cross-validation process is to split the data into k-folds.
### K-fold splitting
K-fold splitting is a method that separates an input training dataset into multiple training and validation datasets. The dataset is split into `k` equally-sized sub-samples called folds. Models are then trained on `k-1` folds and tested against the remaining kth fold, which is the validation dataset. The process is repeated `k` times using a different data set for validation.
The following image depicts k-fold splitting with k = 4 folds. Each fold is represented as a row. The dark-toned boxes represent the parts of the data used in training. The remaining light-toned boxes indicate the validation datasets.
![\[K-fold splitting with 4-folds depicted as boxes: dark for data used; light for validation datasets.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-metrics-kfold-splits.png)
Autopilot uses k-fold cross-validation for both hyperparameter optimization (HPO) mode and ensembling mode.
You can deploy Autopilot models that are built using cross-validation like you would with any other Autopilot or SageMaker AI model.
### HPO mode
K-fold cross-validation uses the k-fold splitting method for cross-validation. In HPO mode, Autopilot automatically implements k-fold cross-validation for small datasets with 50,000 or fewer training instances. Performing cross-validation is especially important when training on small datasets because it protects against overfitting and selection bias.
HPO mode uses a *k* value of 5 on each of the candidate algorithms that are used to model the dataset. Multiple models are trained on different splits, and the models are stored separately. When training is complete, validation metrics for each of the models are averaged to produce a single estimation metric. Lastly, Autopilot combines the models from the trial with the best validation metric into an ensemble model. Autopilot uses this ensemble model to make predictions.
The validation metric for the models trained by Autopilot is presented as the objective metric in the model leaderboard. Autopilot uses the default validation metric for each problem type that it handles, unless you specify otherwise. For the list of all metrics that Autopilot uses, see [Autopilot metrics](#autopilot-metrics).
For example, the [Boston Housing dataset](http://lib.stat.cmu.edu/datasets/boston) contains only 861 samples. If you build a model to predict house sale prices using this dataset without cross-validation, you risk training on a dataset that is not representative of the Boston housing stock. If you split the data only once into training and validation subsets, the training fold may only contain data mainly from the suburbs. As a result, you would train on data that isn't representative of the rest of the city. In this example, your model would likely overfit on this biased selection. K-fold cross-validation can reduce the risk of this kind of error by making full and randomized use of the available data for both training and validation.
Cross-validation can increase training times by an average of 20%. Training times may also increase significantly for complex datasets.
**Note**
In HPO mode, you can see the training and validation metrics from each fold in your `/aws/sagemaker/TrainingJobs` CloudWatch Logs. For more information about CloudWatch Logs, see [CloudWatch Logs for Amazon SageMaker AI](logging-cloudwatch.md).
### Ensembling mode
**Note**
Autopilot supports sample weights in ensembling mode. For the list of available metrics supporting sample weights, see [Autopilot metrics](#autopilot-metrics).
In ensembling mode, cross-validation is performed regardless of dataset size. Customers can either provide their own validation dataset and custom data split ratio, or let Autopilot split the dataset automatically into an 80-20% split ratio. The training data is then split into `k`-folds for cross-validation, where the value of `k` is determined by the AutoGluon engine. An ensemble consists of multiple machine learning models, where each model is known as the base model. A single base model is trained on (`k`-1) folds and makes out-of-fold predictions on the remaining fold. This process is repeated for all `k` folds, and the out-of-fold (OOF) predictions are concatenated to form a single set of predictions. All base models in the ensemble follow this same process of generating OOF predictions.
The following image depicts k-fold validation with `k` = 4 folds. Each fold is represented as a row. The dark-toned boxes represent the parts of the data used in training. The remaining light-toned boxes indicate the validation datasets.
In the upper part of the image, in each fold, the first base model makes predictions on the validation dataset after training on the training datasets. At each subsequent fold, the datasets change roles. A dataset that was previously used for training is now used for validation, and this also applies in reverse. At the end of `k` folds, all of the predictions are concatenated to form a single set of predictions called an out-of-fold (OOF) prediction. This process is repeated for each `n` base models.
![\[k-fold validation: Four rows of boxes depict 4-folds that generate a row of OOF predictions.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-metrics-kfold.PNG)
The OOF predictions for each base model are then used as features to train a stacking model. The stacking model learns the importance weights for each base model. These weights are used to combine the OOF predictions to form the final prediction. Performance on the validation dataset determines which base or stacking model is the best, and this model is returned as the final model.
In ensemble mode, you can either provide your own validation dataset or let Autopilot split the input dataset automatically into 80% train and 20% validation datasets. The training data is then split into `k`-folds for cross-validation and produces an OOF prediction and a base model for each fold.
These OOF predictions are used as features to train a stacking model, which simultaneously learns weights for each base model. These weights are used to combine the OOF predictions to form the final prediction. The validation datasets for each fold are used for hyperparameter tuning of all base models and the stacking model. Performance on the validation datasets determines which base or stacking model is the best model, and this model is returned as the final model.
# Autopilot model deployment and prediction
This Amazon SageMaker Autopilot guide includes steps for model deployment, setting up real-time inference, and running inference with batch jobs.
After you train your Autopilot models, you can deploy them to get predictions in one of two ways:
1. Use [Deploy models for real-time inference](autopilot-deploy-models-realtime.md) to set up an endpoint and obtain predictions interactively. Real-time inference is ideal for inference workloads where you have real-time, interactive, low latency requirements.
1. Use [Run batch inference jobs](autopilot-deploy-models-batch.md) to make predictions in parallel on batches of observations on an entire dataset. Batch inference is a good option for large datasets or if you don't need an immediate response to a model prediction request.
**Note**
To avoid incurring unnecessary charges: After the endpoints and resources that were created from model deployment are no longer needed, you can delete them. For information about pricing of instances by Region, see [Amazon SageMaker Pricing](https://aws.amazon.com/sagemaker/pricing/).
# Deploy models for real-time inference
Real-time inference is ideal for inference workloads where you have real-time, interactive, low latency requirements. This section shows how you can use real-time inferencing to obtain predictions interactively from your model.
To deploy the model that produced the best validation metric in an Autopilot experiment, you have several options. For example, when using Autopilot in SageMaker Studio Classic, you can deploy the model automatically or manually. You can also use SageMaker APIs to manually deploy an Autopilot model.
The following tabs show three options for deploying your model. These instructions assume that you have already created a model in Autopilot. If you don't have a model, see [Create Regression or Classification Jobs for Tabular Data Using the AutoML API](autopilot-automate-model-development-create-experiment.md). To see examples for each option, open each tab.
## Deploy using the Autopilot User Interface (UI)
The Autopilot UI contains helpful dropdown menus, toggles, tooltips, and more to help you navigate through model deployment. You can deploy using either one of the following procedures: Automatic or Manual.
+ **Automatic Deployment**: To automatically deploy the best model from an Autopilot experiment to an endpoint
1. [Create an experiment](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-automate-model-development-create-experiment.html) in SageMaker Studio Classic.
1. Toggle the **Auto deploy** value to **Yes**.
**Note**
**Automatic deployment will fail if either the default resource quota or your customer quota for endpoint instances in a Region is too limited.** In hyperparameter optimization (HPO) mode, you are required to have at least two ml.m5.2xlarge instances. In ensembling mode, you are required to have at least one ml.m5.12xlarge instance. If you encounter a failure related to quotas, you can [request a service limit increase](https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html) for SageMaker AI endpoint instances.
+ **Manual Deployment**: To manually deploy the best model from an Autopilot experiment to an endpoint
1. [Create an experiment](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-automate-model-development-create-experiment.html) in SageMaker Studio Classic.
1. Toggle the **Auto deploy** value to **No**.
1. Select the model that you want to deploy under **Model name**.
1. Select the orange **Deployment and advanced settings** button located on the right of the leaderboard. This opens a new tab.
1. Configure the endpoint name, instance type, and other optional information.
1. Select the orange **Deploy model** to deploy to an endpoint.
1. Check the progress of the endpoint creation process in the [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/) by navigating to the Endpoints section. That section is located in the **Inference** dropdown menu in the navigation panel.
1. After the endpoint status changes from **Creating** to **InService**, as shown below, return to Studio Classic and invoke the endpoint.
![\[SageMaker AI console: Endpoints page to create an endpoint or check endpoint status.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-check-progress.PNG)
## Deploy using SageMaker APIs
You can also obtain real-time inference by deploying your model using **API calls**. This section shows the five steps of this process using AWS Command Line Interface (AWS CLI) code snippets.
For complete code examples for both AWS CLI commands and AWS SDK for Python (boto3), open the tabs directly following these steps.
1. **Obtain candidate definitions**
Obtain the candidate container definitions from [InferenceContainers](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html#sagemaker-Type-AutoMLCandidate-InferenceContainers). These candidate definitions are used to create a SageMaker AI model.
The following example uses the [DescribeAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html) API to obtain candidate definitions for the best model candidate. See the following AWS CLI command as an example.
```
aws sagemaker describe-auto-ml-job --auto-ml-job-name --region
```
1. **List candidates**
The following example uses the [ListCandidatesForAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListCandidatesForAutoMLJob.html) API to list all candidates. See the following AWS CLI command as an example.
```
aws sagemaker list-candidates-for-auto-ml-job --auto-ml-job-name --region
```
1. **Create a SageMaker AI model**
Use the container definitions from the previous steps to create a SageMaker AI model by using the [CreateModel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) API. See the following AWS CLI command as an example.
```
aws sagemaker create-model --model-name '' \
--containers [', , ]' \
--execution-role-arn '' --region '
```
1. **Create an endpoint configuration**
The following example uses the [CreateEndpointConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html) API to create an endpoint configuration. See the following AWS CLI command as an example.
```
aws sagemaker create-endpoint-config --endpoint-config-name '' \
--production-variants '' \
--region ''
```
1. **Create the endpoint**
The following AWS CLI example uses the [CreateEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpoint.html) API to create the endpoint.
```
aws sagemaker create-endpoint --endpoint-name '' \
--endpoint-config-name '' \
--region ''
```
Check the progress of your endpoint deployment by using the [DescribeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpoint.html) API. See the following AWS CLI command as an example.
```
aws sagemaker describe-endpoint —endpoint-name '' —region
```
After the `EndpointStatus` changes to `InService`, the endpoint is ready to use for real-time inference.
1. **Invoke the endpoint**
The following command structure invokes the endpoint for real-time inferencing.
```
aws sagemaker invoke-endpoint --endpoint-name '' \
--region '' --body '' [--content-type] ''
```
The following tabs contain complete code examples for deploying a model with AWS SDK for Python (boto3) or the AWS CLI.
------
#### [ AWS SDK for Python (boto3) ]
1. **Obtain the candidate definitions** by using the following code example.
```
import sagemaker
import boto3
session = sagemaker.session.Session()
sagemaker_client = boto3.client('sagemaker', region_name='us-west-2')
job_name = 'test-auto-ml-job'
describe_response = sm_client.describe_auto_ml_job(AutoMLJobName=job_name)
# extract the best candidate definition from DescribeAutoMLJob response
best_candidate = describe_response['BestCandidate']
# extract the InferenceContainers definition from the caandidate definition
inference_containers = best_candidate['InferenceContainers']
```
1. **Create the model** by using the following the code example.
```
# Create Model
model_name = 'test-model'
sagemaker_role = 'arn:aws:iam:444455556666:role/sagemaker-execution-role'
create_model_response = sagemaker_client.create_model(
ModelName = model_name,
ExecutionRoleArn = sagemaker_role,
Containers = inference_containers
)
```
1. **Create the endpoint configuration** by using the following the code example.
```
endpoint_config_name = 'test-endpoint-config'
instance_type = 'ml.m5.2xlarge'
# for all supported instance types, see
# https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html#sagemaker-Type-ProductionVariant-InstanceType # Create endpoint config
endpoint_config_response = sagemaker_client.create_endpoint_config(
EndpointConfigName=endpoint_config_name,
ProductionVariants=[
{
"VariantName": "variant1",
"ModelName": model_name,
"InstanceType": instance_type,
"InitialInstanceCount": 1
}
]
)
print(f"Created EndpointConfig: {endpoint_config_response['EndpointConfigArn']}")
```
1. **Create the endpoint** and deploy the model with the following code example.
```
# create endpoint and deploy the model
endpoint_name = 'test-endpoint'
create_endpoint_response = sagemaker_client.create_endpoint(
EndpointName=endpoint_name,
EndpointConfigName=endpoint_config_name)
print(create_endpoint_response)
```
**Check the status of creating the endpoint** by using the following the code example.
```
# describe endpoint creation status
status = sagemaker_client.describe_endpoint(EndpointName=endpoint_name)["EndpointStatus"]
```
1. **Invoke the endpoint** for real-time inferencing by using the following command structure.
```
# once endpoint status is InService, you can invoke the endpoint for inferencing
if status == "InService":
sm_runtime = boto3.Session().client('sagemaker-runtime')
inference_result = sm_runtime.invoke_endpoint(EndpointName='test-endpoint', ContentType='text/csv', Body='1,2,3,4,class')
```
------
#### [ AWS Command Line Interface (AWS CLI) ]
1. **Obtain the candidate definitions** by using the following code example.
```
aws sagemaker describe-auto-ml-job --auto-ml-job-name 'test-automl-job' --region us-west-2
```
1. **Create the model** by using the following code example.
```
aws sagemaker create-model --model-name 'test-sagemaker-model'
--containers '[{
"Image": "348316444620.dkr.ecr.us-west-2.amazonaws.com/sagemaker-sklearn-automl:2.5-1-cpu-py3", amzn-s3-demo-bucket1
"ModelDataUrl": "s3://amzn-s3-demo-bucket/output/model.tar.gz",
"Environment": {
"AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF": "1",
"AUTOML_TRANSFORM_MODE": "feature-transform",
"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "application/x-recordio-protobuf",
"SAGEMAKER_PROGRAM": "sagemaker_serve",
"SAGEMAKER_SUBMIT_DIRECTORY": "/opt/ml/model/code"
}
}, {
"Image": "348316444620.dkr.ecr.us-west-2.amazonaws.com/sagemaker-xgboost:1.3-1-cpu-py3",
"ModelDataUrl": "s3://amzn-s3-demo-bucket/output/model.tar.gz",
"Environment": {
"MAX_CONTENT_LENGTH": "20971520",
"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv",
"SAGEMAKER_INFERENCE_OUTPUT": "predicted_label",
"SAGEMAKER_INFERENCE_SUPPORTED": "predicted_label,probability,probabilities"
}
}, {
"Image": "348316444620.dkr.ecr.us-west-2.amazonaws.com/sagemaker-sklearn-automl:2.5-1-cpu-py3", aws-region
"ModelDataUrl": "s3://amzn-s3-demo-bucket/output/model.tar.gz",
"Environment": {
"AUTOML_TRANSFORM_MODE": "inverse-label-transform",
"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv",
"SAGEMAKER_INFERENCE_INPUT": "predicted_label",
"SAGEMAKER_INFERENCE_OUTPUT": "predicted_label",
"SAGEMAKER_INFERENCE_SUPPORTED": "predicted_label,probability,labels,probabilities",
"SAGEMAKER_PROGRAM": "sagemaker_serve",
"SAGEMAKER_SUBMIT_DIRECTORY": "/opt/ml/model/code"
}
}]' \
--execution-role-arn 'arn:aws:iam::1234567890:role/sagemaker-execution-role' \
--region 'us-west-2'
```
For additional details, see [creating a model](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sagemaker/create-model.html).
The `create model` command will return a response in the following format.
```
{
"ModelArn": "arn:aws:sagemaker:us-west-2:1234567890:model/test-sagemaker-model"
}
```
1. **Create an endpoint configuration** by using the following code example.
```
aws sagemaker create-endpoint-config --endpoint-config-name 'test-endpoint-config' \
--production-variants '[{"VariantName": "variant1",
"ModelName": "test-sagemaker-model",
"InitialInstanceCount": 1,
"InstanceType": "ml.m5.2xlarge"
}]' \
--region us-west-2
```
The `create endpoint` configuration command will return a response in the following format.
```
{
"EndpointConfigArn": "arn:aws:sagemaker:us-west-2:1234567890:endpoint-config/test-endpoint-config"
}
```
1. **Create an endpoint** by using the following code example.
```
aws sagemaker create-endpoint --endpoint-name 'test-endpoint' \
--endpoint-config-name 'test-endpoint-config' \
--region us-west-2
```
The `create endpoint` command will return a response in the following format.
```
{
"EndpointArn": "arn:aws:sagemaker:us-west-2:1234567890:endpoint/test-endpoint"
}
```
Check the progress of the endpoint deployment by using the following [describe-endpoint](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/describe-endpoint.html) CLI code example.
```
aws sagemaker describe-endpoint --endpoint-name 'test-endpoint' --region us-west-2
```
The previous progress check will return a response in the following format.
```
{
"EndpointName": "test-endpoint",
"EndpointArn": "arn:aws:sagemaker:us-west-2:1234567890:endpoint/test-endpoint",
"EndpointConfigName": "test-endpoint-config",
"EndpointStatus": "Creating",
"CreationTime": 1660251167.595,
"LastModifiedTime": 1660251167.595
}
```
After the `EndpointStatus` changes to `InService`, the endpoint is ready for use in real-time inference.
1. **Invoke the endpoint** for real-time inferencing by using the following command structure.
```
aws sagemaker-runtime invoke-endpoint --endpoint-name 'test-endpoint' \
--region 'us-west-2' \
--body '1,51,3.5,1.4,0.2' \
--content-type 'text/csv' \
'/tmp/inference_output'
```
For more options, see [invoking an endpoint](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sagemaker-runtime/invoke-endpoint.html).
------
## Deploy models from different accounts
You can deploy an Autopilot model from a different account than the original account that a model was generated in. To implement cross-account model deployment, this section shows how to do the following: Grant permission to assume the role to the account you want to deploy from (the generating account). Make a call to `DescribeAutoMLJob` from the deploying account to obtain model information. Grant access rights to the model artifacts from the generating account.
1. **Grant permission to the deploying account**
To assume the role in the generating account, you must grant permission to the deploying account. This allows the deploying account to describe Autopilot jobs in the generating account.
The following example uses a generating account with a trusted `sagemaker-role` entity. The example shows how to give a deploying account with the ID 111122223333 permission to assume the role of the generating account.
```
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"sagemaker.amazonaws.com"
],
"AWS": [ "111122223333"]
},
"Action": "sts:AssumeRole"
}
```
The new account with the ID 111122223333 can now assume the role for the generating account.
Next, call the `DescribeAutoMLJob` API from the deploying account to obtain a description of the job created by the generating account.
The following code example describes the model from the deploying account.
```
import sagemaker
import boto3
session = sagemaker.session.Session()
sts_client = boto3.client('sts')
sts_client.assume_role
role = 'arn:aws:iam::111122223333:role/sagemaker-role'
role_session_name = "role-session-name"
_assumed_role = sts_client.assume_role(RoleArn=role, RoleSessionName=role_session_name)
credentials = _assumed_role["Credentials"]
access_key = credentials["AccessKeyId"]
secret_key = credentials["SecretAccessKey"]
session_token = credentials["SessionToken"]
session = boto3.session.Session()
sm_client = session.client('sagemaker', region_name='us-west-2',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
aws_session_token=session_token)
# now you can call describe automl job created in account A
job_name = "test-job"
response= sm_client.describe_auto_ml_job(AutoMLJobName=job_name)
```
1. **Grant access to the deploying account** to the model artifacts in the generating account.
The deploying account only needs access to the model artifacts in the generating account to deploy it. These are located in the [S3OutputPath](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html#sagemaker-Type-AutoMLOutputDataConfig-S3OutputPath) that was specified in the original `CreateAutoMLJob` API call during model generation.
To give the deploying account access to the model artifacts, choose one of the following options:
1. [Give access](https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/) to the `ModelDataUrl` from the generating account to the deploying account.
Next, you need to give the deploying account permission to assume the role. follow the [real-time inferencing steps](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-deploy-models.html#autopilot-deploy-models-realtime) to deploy.
1. [Copy model artifacts](https://aws.amazon.com/premiumsupport/knowledge-center/copy-s3-objects-account/) from the generating account's original [S3OutputPath](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html#sagemaker-Type-AutoMLOutputDataConfig-S3OutputPath) to the generating account.
To grant access to the model artifacts, you must define a `best_candidate` model and reassign model containers to the new account.
The following example shows how to define a `best_candidate` model and reassign the `ModelDataUrl`.
```
best_candidate = automl.describe_auto_ml_job()['BestCandidate']
# reassigning ModelDataUrl for best_candidate containers below
new_model_locations = ['new-container-1-ModelDataUrl', 'new-container-2-ModelDataUrl', 'new-container-3-ModelDataUrl']
new_model_locations_index = 0
for container in best_candidate['InferenceContainers']:
container['ModelDataUrl'] = new_model_locations[new_model_locations_index++]
```
After this assignment of containers, follow the steps in [Deploy using SageMaker APIs](#autopilot-deploy-models-api) to deploy.
To build a payload in real-time inferencing, see the notebook example to [ define a test payload](https://aws.amazon.com/getting-started/hands-on/machine-learning-tutorial-automatically-create-models). To create the payload from a CSV file and invoke an endpoint, see the **Predict with your model** section in [Create a machine learning model automatically](https://aws.amazon.com/getting-started/hands-on/create-machine-learning-model-automatically-sagemaker-autopilot/#autopilot-cr-room).
# Run batch inference jobs
Batch inferencing, also known as offline inferencing, generates model predictions on a batch of observations. Batch inference is a good option for large datasets or if you don't need an immediate response to a model prediction request. By contrast, online inference ([real-time inferencing](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-deploy-models.html#autopilot-deploy-models-realtime)) generates predictions in real time. You can make batch inferences from an Autopilot model using the [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/), the Autopilot user interface (UI), the [AWS SDK for Python (boto3)](https://aws.amazon.com/sdk-for-python/), or the AWS Command Line Interface ([AWS CLI](https://docs.aws.amazon.com/cli/)).
The following tabs show three options for deploying your model: Using APIs, Autopilot UI, or using APIs to deploy from different accounts. These instructions assume that you have already created a model in Autopilot. If you don't have a model, see [Create Regression or Classification Jobs for Tabular Data Using the AutoML API](autopilot-automate-model-development-create-experiment.md). To see examples for each option, open each tab.
## Deploy a model using Autopilot UI
The Autopilot UI contains helpful dropdown menus, toggles, tooltips, and more to help you navigate through model deployment.
The following steps show how to deploy a model from an Autopilot experiment for batch predictions.
1. Sign in at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/) and select **Studio** from the navigation pane.
1. On the left navigation pane, choose **Studio**.
1. Under **Get started**, select the Domain that you want to launch the Studio application in. If your user profile only belongs to one Domain, you do not see the option for selecting a Domain.
1. Select the user profile that you want to launch the Studio Classic application for. If there is no user profile in the domain, choose **Create user profile**. For more information, see [Add user profiles](https://docs.aws.amazon.com/sagemaker/latest/dg/domain-user-profile-add.html).
1. Choose **Launch Studio**. If the user profile belongs to a shared space, choose **Open Spaces**.
1. When the SageMaker Studio Classic console opens, choose the **Launch SageMaker Studio** button.
1. Select **AutoML** from the left navigation pane.
1. Under **Name**, select the Autopilot experiment corresponding to the model that you want to deploy. This opens a new **AUTOPILOT JOB** tab.
1. In the **Model name** section, select the model that you want to deploy.
1. Choose **Deploy model**. This opens a new tab.
1. Choose **Make batch predictions** at the top of the page.
1. For **Batch transform job configuration**, input the **Instance type**, **Instance count** and other optional information.
1. In the **Input data configuration** section, open the dropdown menu.
1. For **S3 data type**, choose **ManifestFile** or **S3Prefix**.
1. For **Split type**, choose **Line**, **RecordIO**, **TFRecord** or **None**.
1. For **Compression**, choose **Gzip** or **None**.
1. For **S3 location**, enter the Amazon S3 bucket location of the input data and other optional information.
1. Under **Output data configuration**, enter the S3 bucket for the output data, and choose how to [assemble the output](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TransformOutput.html#sagemaker-Type-TransformOutput-AssembleWith) of your job.
1. For **Additional configuration (optional)**, you can enter a MIME type and an **S3 Encryption key**.
1. For **Input/output filtering and data joins (optional)**, you enter a JSONpath expression to filter your input data, join the input source data with your output data, and enter a JSONpath expression to filter your output data.
1. For examples for each type of filter, see the [DataProcessing API](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DataProcessing.html#sagemaker-Type-DataProcessing-InputFilter).
1. To perform batch predictions on your input dataset, select **Create batch transform job**. A new **Batch Transform Jobs** tab appears.
1. In the **Batch Transform Jobs** tab: Locate the name of your job in **Status** section. Then check the progress of the job.
## Deploy using SageMaker APIs
To use the SageMaker APIs for batch inferencing, there are three steps:
1. **Obtain candidate definitions**
Candidate definitions from [InferenceContainers](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html#sagemaker-Type-AutoMLCandidate-InferenceContainers) are used to create a SageMaker AI model.
The following example shows how to use the [DescribeAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html) API to obtain candidate definitions for the best model candidate. See the following AWS CLI command as an example.
```
aws sagemaker describe-auto-ml-job --auto-ml-job-name --region
```
Use the [ListCandidatesForAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListCandidatesForAutoMLJob.html) API to list all candidates. See the following AWS CLI command as an example.
```
aws sagemaker list-candidates-for-auto-ml-job --auto-ml-job-name --region
```
1. **Create a SageMaker AI model**
To create a SageMaker AI model using the [CreateModel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) API, use the container definitions from the previous steps. See the following AWS CLI command as an example.
```
aws sagemaker create-model --model-name '' \
--containers [', , ]' \
--execution-role-arn '' --region '
```
1. **Create a SageMaker AI transform job**
The following example creates a SageMaker AI transform job with the [CreateTransformJob](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/create-transform-job.html) API. See the following AWS CLI command as an example.
```
aws sagemaker create-transform-job --transform-job-name '' --model-name ''\
--transform-input '{
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": ""
}
},
"ContentType": "text/csv",
"SplitType": "Line"
}'\
--transform-output '{
"S3OutputPath": "",
"AssembleWith": "Line"
}'\
--transform-resources '{
"InstanceType": "",
"InstanceCount": 1
}' --region ''
```
Check the progress of your transform job using the [DescribeTransformJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeTransformJob.html) API. See the following AWS CLI command as an example.
```
aws sagemaker describe-transform-job --transform-job-name '' --region
```
After the job is finished, the predicted result will be available in ``.
The output file name has the following format: `.out`. As an example, if your input file is `text_x.csv`, the output name will be `text_x.csv.out`.
The following tabs show code examples for SageMaker Python SDK, AWS SDK for Python (boto3), and the AWS CLI.
------
#### [ SageMaker Python SDK ]
The following example uses the **[SageMaker Python SDK ](https://sagemaker.readthedocs.io/en/stable/overview.html)** to make predictions in batches.
```
from sagemaker import AutoML
sagemaker_session= sagemaker.session.Session()
job_name = 'test-auto-ml-job' # your autopilot job name
automl = AutoML.attach(auto_ml_job_name=job_name)
output_path = 's3://test-auto-ml-job/output'
input_data = 's3://test-auto-ml-job/test_X.csv'
# call DescribeAutoMLJob API to get the best candidate definition
best_candidate = automl.describe_auto_ml_job()['BestCandidate']
best_candidate_name = best_candidate['CandidateName']
# create model
model = automl.create_model(name=best_candidate_name,
candidate=best_candidate)
# create transformer
transformer = model.transformer(instance_count=1,
instance_type='ml.m5.2xlarge',
assemble_with='Line',
output_path=output_path)
# do batch transform
transformer.transform(data=input_data,
split_type='Line',
content_type='text/csv',
wait=True)
```
------
#### [ AWS SDK for Python (boto3) ]
The following example uses **AWS SDK for Python (boto3)** to make predictions in batches.
```
import sagemaker
import boto3
session = sagemaker.session.Session()
sm_client = boto3.client('sagemaker', region_name='us-west-2')
role = 'arn:aws:iam::1234567890:role/sagemaker-execution-role'
output_path = 's3://test-auto-ml-job/output'
input_data = 's3://test-auto-ml-job/test_X.csv'
best_candidate = sm_client.describe_auto_ml_job(AutoMLJobName=job_name)['BestCandidate']
best_candidate_containers = best_candidate['InferenceContainers']
best_candidate_name = best_candidate['CandidateName']
# create model
reponse = sm_client.create_model(
ModelName = best_candidate_name,
ExecutionRoleArn = role,
Containers = best_candidate_containers
)
# Lauch Transform Job
response = sm_client.create_transform_job(
TransformJobName=f'{best_candidate_name}-transform-job',
ModelName=model_name,
TransformInput={
'DataSource': {
'S3DataSource': {
'S3DataType': 'S3Prefix',
'S3Uri': input_data
}
},
'ContentType': "text/csv",
'SplitType': 'Line'
},
TransformOutput={
'S3OutputPath': output_path,
'AssembleWith': 'Line',
},
TransformResources={
'InstanceType': 'ml.m5.2xlarge',
'InstanceCount': 1,
},
)
```
The batch inference job returns a response in the following format.
```
{'TransformJobArn': 'arn:aws:sagemaker:us-west-2:1234567890:transform-job/test-transform-job',
'ResponseMetadata': {'RequestId': '659f97fc-28c4-440b-b957-a49733f7c2f2',
'HTTPStatusCode': 200,
'HTTPHeaders': {'x-amzn-requestid': '659f97fc-28c4-440b-b957-a49733f7c2f2',
'content-type': 'application/x-amz-json-1.1',
'content-length': '96',
'date': 'Thu, 11 Aug 2022 22:23:49 GMT'},
'RetryAttempts': 0}}
```
------
#### [ AWS Command Line Interface (AWS CLI) ]
1. **Obtain the candidate definitions** by using the following the code example.
```
aws sagemaker describe-auto-ml-job --auto-ml-job-name 'test-automl-job' --region us-west-2
```
1. **Create the model** by using the following the code example.
```
aws sagemaker create-model --model-name 'test-sagemaker-model'
--containers '[{
"Image": "348316444620.dkr.ecr.us-west-2.amazonaws.com/sagemaker-sklearn-automl:2.5-1-cpu-py3",
"ModelDataUrl": "s3://amzn-s3-demo-bucket/out/test-job1/data-processor-models/test-job1-dpp0-1-e569ff7ad77f4e55a7e549a/output/model.tar.gz",
"Environment": {
"AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF": "1",
"AUTOML_TRANSFORM_MODE": "feature-transform",
"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "application/x-recordio-protobuf",
"SAGEMAKER_PROGRAM": "sagemaker_serve",
"SAGEMAKER_SUBMIT_DIRECTORY": "/opt/ml/model/code"
}
}, {
"Image": "348316444620.dkr.ecr.us-west-2.amazonaws.com/sagemaker-xgboost:1.3-1-cpu-py3",
"ModelDataUrl": "s3://amzn-s3-demo-bucket/out/test-job1/tuning/flicdf10v2-dpp0-xgb/test-job1E9-244-7490a1c0/output/model.tar.gz",
"Environment": {
"MAX_CONTENT_LENGTH": "20971520",
"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv",
"SAGEMAKER_INFERENCE_OUTPUT": "predicted_label",
"SAGEMAKER_INFERENCE_SUPPORTED": "predicted_label,probability,probabilities"
}
}, {
"Image": "348316444620.dkr.ecr.us-west-2.amazonaws.com/sagemaker-sklearn-automl:2.5-1-cpu-py3",
"ModelDataUrl": "s3://amzn-s3-demo-bucket/out/test-job1/data-processor-models/test-job1-dpp0-1-e569ff7ad77f4e55a7e549a/output/model.tar.gz",
"Environment": {
"AUTOML_TRANSFORM_MODE": "inverse-label-transform",
"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv",
"SAGEMAKER_INFERENCE_INPUT": "predicted_label",
"SAGEMAKER_INFERENCE_OUTPUT": "predicted_label",
"SAGEMAKER_INFERENCE_SUPPORTED": "predicted_label,probability,labels,probabilities",
"SAGEMAKER_PROGRAM": "sagemaker_serve",
"SAGEMAKER_SUBMIT_DIRECTORY": "/opt/ml/model/code"
}
}]' \
--execution-role-arn 'arn:aws:iam::1234567890:role/sagemaker-execution-role' \
--region 'us-west-2'
```
1. **Create the transform job** by using the following the code example.
```
aws sagemaker create-transform-job --transform-job-name 'test-tranform-job'\
--model-name 'test-sagemaker-model'\
--transform-input '{
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "s3://amzn-s3-demo-bucket/data.csv"
}
},
"ContentType": "text/csv",
"SplitType": "Line"
}'\
--transform-output '{
"S3OutputPath": "s3://amzn-s3-demo-bucket/output/",
"AssembleWith": "Line"
}'\
--transform-resources '{
"InstanceType": "ml.m5.2xlarge",
"InstanceCount": 1
}'\
--region 'us-west-2'
```
1. **Check the progress of the transform job** by using the following the code example.
```
aws sagemaker describe-transform-job --transform-job-name 'test-tranform-job' --region us-west-2
```
The following is the response from the transform job.
```
{
"TransformJobName": "test-tranform-job",
"TransformJobArn": "arn:aws:sagemaker:us-west-2:1234567890:transform-job/test-tranform-job",
"TransformJobStatus": "InProgress",
"ModelName": "test-model",
"TransformInput": {
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "s3://amzn-s3-demo-bucket/data.csv"
}
},
"ContentType": "text/csv",
"CompressionType": "None",
"SplitType": "Line"
},
"TransformOutput": {
"S3OutputPath": "s3://amzn-s3-demo-bucket/output/",
"AssembleWith": "Line",
"KmsKeyId": ""
},
"TransformResources": {
"InstanceType": "ml.m5.2xlarge",
"InstanceCount": 1
},
"CreationTime": 1662495635.679,
"TransformStartTime": 1662495847.496,
"DataProcessing": {
"InputFilter": "$",
"OutputFilter": "$",
"JoinSource": "None"
}
}
```
After the `TransformJobStatus` changes to `Completed`, you can check the inference result in the `S3OutputPath`.
------
## Deploy models from different accounts
To create a batch inferencing job in a different account than the one that the model was generated in, follow the instructions in [Deploy models from different accounts](autopilot-deploy-models-realtime.md#autopilot-deploy-models-realtime-across-accounts). Then you can create models and transform jobs by following the [Deploy using SageMaker APIs](#autopilot-deploy-models-batch-steps).
# View model details
Autopilot generates details about the candidate models that you can obtain. These details include the following:
+ A plot of the aggregated SHAP values that indicate the importance of each feature. This helps explain your models predictions.
+ The summary statistics for various training and validation metrics, including the objective metric.
+ A list of the hyperparameters used to train and tune the model.
To view model details after running an Autopilot job, follow these steps:
1. Choose the **Home** icon (![\[Black square icon representing a placeholder or empty image.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/studio/icons/house.png)) from the left navigation pane to view the top-level **Amazon SageMaker Studio Classic** navigation menu.
1. Select the **AutoML** card from the main working area. This opens a new **Autopilot** tab.
1. In the **Name** section, select the Autopilot job that has the details that you want to examine. This opens a new **Autopilot job** tab.
1. The **Autopilot job** panel lists the metric values including the **Objective** metric for each model under **Model name**. The **Best model** is listed at the top of the list under **Model name** and is also highlighted in the **Models** tab.
1. To review model details, select the model that you are interested in and select **View model details**. This opens a new **Model Details** tab.
1. The **Model Details** tab is divided into four subsections.
1. The top of the **Explainability** tab contains a plot of aggregated SHAP values that indicate the importance of each feature. Following that are the metrics and hyperparameter values for this model.
1. The **Performance** tab contains metrics statistics a confusion matrix.
1. The **Artifacts** tab contains information about model inputs, outputs, and intermediate results.
1. The **Network** tab summarizes your network isolation and encryption choices.
**Note**
Feature importance and information in the **Performance** tab is only generated for the **Best model**.
For more information about how the SHAP values help explain predictions based on feature importance, see the whitepaper [Understanding the model explainability](https://pages.awscloud.com/rs/112-TZM-766/images/Amazon.AI.Fairness.and.Explainability.Whitepaper.pdf). Additional information is also available in the [Model Explainability](clarify-model-explainability.md) topic in the SageMaker AI Developer Guide.
# View an Autopilot model performance report
An Amazon SageMaker AI model quality report (also referred to as performance report) provides insights and quality information for the best model candidate generated by an AutoML job. This includes information about the job details, model problem type, objective function, and other information related to the problem type. This guide shows how to view Amazon SageMaker Autopilot performance metrics graphically, or view metrics as raw data in a JSON file.
For example, in classification problems, the model quality report includes the following:
+ Confusion matrix
+ Area under the receiver operating characteristic curve (AUC)
+ Information to understand false positives and false negatives
+ Tradeoffs between true positives and false positives
+ Tradeoffs between precision and recall
Autopilot also provides performance metrics for all of your candidate models. These metrics are calculated using all of the training data and are used to estimate model performance. The main working area includes these metrics by default. The type of metric is determined by the type of problem being addressed.
Refer to the [ Amazon SageMaker API reference documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobObjective.html) for the list of available metrics supported by Autopilot.
You can sort your model candidates with the relevant metric to help you select and deploy the model that addresses your business needs. For definitions of these metrics, see the [Autopilot candidate metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-metrics-validation.html#autopilot-metrics) topic.
To view a performance report from an Autopilot job, follow these steps:
1. Choose the **Home** icon (![\[Black square icon representing a placeholder or empty image.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/studio/icons/house.png)) from the left navigation pane to view the top-level **Amazon SageMaker Studio Classic** navigation menu.
1. Select the **AutoML** card from the main working area. This opens a new **Autopilot** tab.
1. In the **Name** section, select the Autopilot job that has the details that you want to examine. This opens a new **Autopilot job** tab.
1. The **Autopilot job** panel lists the metric values including the **Objective** metric for each model under **Model name**. The **Best model** is listed at the top of the list under **Model name** and it is highlighted in the **Models** tab.
1. To review model details, select the model that you are interested in and select **View in model details**. This opens a new **Model Details** tab.
1. Choose the **Performance** tab between the **Explainability** and **Artifacts** tab.
1. On the top right section of the tab, select the down arrow on the **Download Performance Reports** button.
1. The down arrow provides two options to view Autopilot performance metrics:
1. You can download a PDF of the performance report to view the metrics graphically.
1. You can view metrics as raw data and download it as a JSON file.
For instructions on how to create and run an AutoML job in SageMaker Studio Classic, see [Create Regression or Classification Jobs for Tabular Data Using the AutoML API](autopilot-automate-model-development-create-experiment.md).
The performance report contains two sections. The first contains details about the Autopilot job that produced the model. The second section contains a model quality report.
## Autopilot Job details
This first section of the report gives some general information about the Autopilot job that produced the model. These job details include the following information:
+ Autopilot candidate name
+ Autopilot job name
+ Problem type
+ Objective metric
+ Optimization direction
## Model quality report
Model quality information is generated by Autopilot model insights. The report's content that is generated depends on the problem type it addressed: regression, binary classification, or multiclass classification. The report specifies the number of rows that were included in the evaluation dataset and the time at which the evaluation occurred.
### Metrics tables
The first part of the model quality report contains metrics tables. These are appropriate for the type of problem that the model addressed.
The following image is an example of a metrics table that Autopilot generates for a regression problem. It shows the metric name, value, and standard deviation.
![\[Amazon SageMaker Autopilot model insights regression metrics report example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-regression-metrics.png)
The following image is an example of a metrics table generated by Autopilot for a multiclass classification problem. It shows the metric name, value, and standard deviation.
![\[Amazon SageMaker Autopilot model insights multiclass classification metrics report example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-multiclass-metrics-report.png)
### Graphical model performance information
The second part of the model quality report contains graphical information to help you evaluate model performance. The contents of this section depend on the problem type used in modeling.
#### The area under the receiver operating characteristic curve
The area under the receiver operating characteristic curve represents the trade-off between true positive and false positive rates. It is an industry-standard accuracy metric used for binary classification models. AUC (area under the curve) measures the ability the model to predict a higher score for positive examples, as compared to negative examples. The AUC metric provides an aggregated measure of the model performance across all possible classification thresholds.
The AUC metric returns a decimal value from 0 to 1. AUC values near 1 indicate that the machine learning model is highly accurate. Values near 0.5 indicate that the model is performing no better than guessing at random. AUC values close to 0 indicate that the model has learned the correct patterns, but is making predictions that are as inaccurate as possible. Values near zero can indicate a problem with the data. For more information about the AUC metric, see the [Receiver operating characteristic](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) article on Wikipedia.
The following is an example of an area under the receiver operating characteristic curve graph to evaluate predictions made by a binary classification model. The dashed thin line represents the area under the receiver operating characteristic curve that a model which classifies no-better-than-random guessing would score, with an AUC score of 0.5. The curves of more accurate classification models lie above this random baseline, where the rate of true positives exceeds the rate of false positives. The area under the receiver operating characteristic curve representing the performance of the binary classification model is the thicker solid line.
![\[Amazon SageMaker Autopilot area under the receiver operating characteristic curve example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-receiver-operating-characteristic-curve.png)
A summary of the graph's components of **false positive rate **(FPR) and **true positive rate **(TPR) are defined as follows.
+ Correct predictions
+ **True positive** (TP): The predicted value is 1, and the true value is 1.
+ **True negative** (TN): The predicted value is 0, and the true value is 0.
+ Erroneous predictions
+ **False positive** (FP): The predicted value is 1, but the true value is 0.
+ **False negative** (FN): The predicted value is 0, but the true value is 1.
The **false positive rate **(FPR) measures the fraction of true negatives (TN) that were falsely predicted as positives (FP), over the sum of FP and TN. The range is 0 to 1. A smaller value indicates better predictive accuracy.
+ FPR = FP/(FP\$1TN)
The **true positive rate **(TPR) measures the fraction true positives that were correctly predicted as positives (TP) over the sum of TP and false negatives (FN). The range is 0 to 1. A larger value indicates better predictive accuracy.
+ TPR = TP/(TP\$1FN)
#### Confusion matrix
A confusion matrix provides a way to visualize the accuracy of the predictions made by a model for binary and multiclass classification for different problems. The confusion matrix in the model quality report contains the following.
+ The number and percentage of correct and incorrect predictions for the actual labels
+ The number and percentage of accurate predictions on the diagonal from the upper-left to the lower-right corner
+ The number and percentage of inaccurate predictions on the diagonal from the upper-right to the lower-left corner
The incorrect predictions on a confusion matrix are the confusion values.
The following diagram is an example of a confusion matrix for a binary classification problem. It contains the following information:
+ The vertical axis is divided into two rows containing true and false actual labels.
+ The horizontal axis is divided into two columns containing true and false labels that were predicted by the model.
+ The color bar assigns a darker tone to a larger number of samples to visually indicate the number of values that were classified in each category.
In this example, the model predicted actual 2817 false values correctly, and 353 actual true values correctly. The model incorrectly predicted 130 actual true values to be false and 33 actual false values to be true. The difference in tone indicates that the dataset is not balanced. The imbalance is because there are many more actual false labels than actual true labels.
![\[Amazon SageMaker Autopilot binary confusion matrix example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-confusion-matrix-binary.png)
The following diagram is an example of a confusion matrix for a multi-class classification problem. The confusion matrix in the model quality report contains the following.
+ The vertical axis is divided into three rows containing three different actual labels.
+ The horizontal axis is divided into three columns containing labels that were predicted by the model.
+ The color bar assigns a darker tone to a larger number of samples to visually indicate the number of values that were classified in each category.
In the example below, the model correctly predicted actual 354 values for label **f**, 1094 values for label **i** and 852 values for label **m**. The difference in tone indicates that the dataset is not balanced because there are many more labels for the value **i** than for **f** or **m**.
![\[Amazon SageMaker Autopilot multiclass confusion matrix example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-confusion-matrix-multiclass.png)
The confusion matrix in the model quality report provided can accommodate a maximum of 15 labels for multiclass classification problem types. If a row corresponding to a label shows a `Nan` value, it means that the validation dataset used to check model predictions does not contain data with that label.
#### Gain curve
In binary classification, a gain curve predicts the cumulative benefit of using a percentage of the dataset to find a positive label. The gain value is calculated during training by dividing the cumulative number of positive observations by the total number of positive observations in the data, at each decile. If the classification model created during training is representative of the unseen data, you can use the gain curve to predict the percentage of data that you must target to obtain a percentage of positive labels. The greater the percentage of the dataset used, the higher the percentage of positive labels found.
In the following example graph, the gain curve is the line with changing slope. The straight line is the percentage of positive labels found by selecting a percentage of data from the dataset at random. Upon targeting 20% of the dataset, you would expect to find larger than 40% of the positive labels. As an example, you might consider using a gain curve to determine your efforts in a marketing campaign. Using our gain curve example, for 83% of people in a neighborhood to purchase cookies, you'd send an advertisement to about 60% of the neighborhood.
![\[Amazon SageMaker Autopilot gain curve example with percentage and gain value.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-gain-curve.png)
#### Lift curve
In binary classification, the lift curve illustrates the uplift of using a trained model to predict the likelihood of finding a positive label compared to a random guess. The lift value is calculated during training using the ratio of percentage gain to the ratio of positive labels at each decile. If the model created during training is representative of the unseen data, use the lift curve to predict the benefit of using the model over randomly guessing.
In the following example graph, the lift curve is the line with changing slope. The straight line is the lift curve associated with selecting the corresponding percentage randomly from the dataset. Upon targeting 40% of the dataset with your model's classification labels, you would expect to find about 1.7 times the number of the positive labels that you would have found by randomly selecting 40% of the unseen data.
![\[Amazon SageMaker Autopilot lift curve example with percentage and lift value.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-lift-curve.png)
#### Precision-recall curve
The precision-recall curve represents the tradeoff between precision and recall for binary classification problems.
**Precision** measures the fraction of actual positives that are predicted as positive (TP) out of all positive predictions (TP and false positive). The range is 0 to 1. A larger value indicates better accuracy in the predicted values.
+ Precision = TP/(TP\$1FP)
**Recall** measures the fraction of actual positives that are predicted as positive (TP) out of all actual positive predictions (TP and false negative). This is also known as the sensitivity or as the true positive rate. The range is 0 to 1. A larger value indicates better detection of positive values from the sample.
+ Recall = TP/(TP\$1FN)
The objective of a classification problem is to correctly label as many elements as possible. A system with high recall but low precision returns a high percentage of false positives.
The following graphic depicts a spam filter that marks every email as spam. It has high recall, but low precision, because recall doesn't measure false positives.
Give more weight to recall over precision if your problem has a low penalty for false positive values, but a high penalty for missing a true positive result. For example, detecting an impending collision in a self-driving vehicle.
![\[Autopilot example of high recall and low precision system, modelling all samples as positives.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-high-recall-low-precision.PNG)
By contrast, a system with high precision, but low recall, returns a high percentage of false negatives. A spam filter that marks every email as desirable (not spam) has high precision but low recall because precision doesn't measure false negatives.
If your problem has a low penalty for false negative values, but a high penalty for missing a true negative results, give more weight to precision over recall. For example, flagging a suspicious filter for a tax audit.
The following graphic depicts a spam filter that has high precision but low recall, because precision doesn't measure false negatives.
![\[Autopilot example of high-precision and low-recall system, modeling all samples as negatives.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-high-precision-low-recall.PNG)
A model that makes predictions with both high precision and high recall produces a high number of correctly labeled results. For more information, see [Precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall) article in Wikipedia.
#### Area under precision-recall curve (AUPRC)
For binary classification problems, Amazon SageMaker Autopilot includes a graph of the area under the precision-recall curve (AUPRC). The AUPRC metric provides an aggregated measure of the model performance across all possible classification thresholds and uses both precision and recall. AUPRC does not take the number of true negatives into account. Therefore, it can be useful to evaluate model performance in cases where there's a large number of true negatives in the data. For example, to model a gene containing a rare mutation.
The following graphic is an example of an AUPRC graph. Precision at its highest value is 1, and recall is at 0. In the lower right corner of the graph, recall is its highest value (1) and precision is 0. In between these two points , the AUPRC curve illustrates the tradeoff between precision and recall at different thresholds.
![\[Precision-recall curve depicts tradeoff between precision and recall at different thresholds.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-binary-precision-recall.png)
#### Actual against predicted plot
The actual against predicted plot shows the difference between actual and predicted model values. In the following example graph, the solid line is a linear line of best fit. If the model were 100% accurate, each predicted point would equal its corresponding actual point and lie on this line of best fit. The distance away from the line of best fit is a visual indication of model error. The larger the distance away from the line of best fit, the higher the model error.
![\[Example with linear line of best fit, differing actual and predicted plot, and model error.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-actual-vs-predicted-plot.png)
#### Standardized residual plot
A standardized residual plot incorporates the following statistical terms:
**`residual`**
A (raw) residual shows the difference between actual and values predicted by your model. The larger the difference, the larger the residual value.
**`standard deviation`**
The standard deviation is a measure of how values vary from an average value. A high standard deviation indicates that many values are very different from their average value. A low standard deviation indicates that many values are close to their average value.
**`standardized residual`**
A standardized residual divides the raw residuals by their standard deviation. Standardized residuals have units of standard deviation and are useful in identifying outliers in data regardless of the difference in scale of the raw residuals. If a standardized residual is much smaller or larger than the other standardized residuals, it indicates that the model is not fitting these observations well.
The standardized residual plot measures the strength of the difference between observed and expected values. The actual predicted value is displayed on the x axis. A point with a value larger than an absolute value of 3 is commonly regarded as an outlier.
The following example graph shows that a large number of standardized residuals are clustered around 0 on the horizontal axis. The values close to zero indicate that the model is fitting these points well. The points towards the top and bottom of the plot are not predicted well by the model.
![\[Amazon SageMaker Autopilot standardized residual plot example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-standardized-residual.png)
#### Residual histogram
A residual histogram incorporates the following statistical terms:
**`residual`**
A (raw) residual shows the difference between actual and values predicted by your model. The larger the difference, the larger the residual value.
**`standard deviation`**
The standard deviation is a measure of how much values vary from an average value. A high standard deviation indicates that many values are very different from their average value. A low standard deviation indicates that many values are close to their average value.
**`standardized residual`**
A standardized residual divides the raw residuals by their standard deviation. Standardized residuals have units of standard deviation. These are useful in identifying outliers in data regardless of the difference in scale of the raw residuals. If a standardized residual is much smaller or larger than the other standardized residuals, it would indicate that the model is not fitting these observations well.
**`histogram`**
A histogram is a graph that shows how often a value occurred.
The residual histogram shows the distribution of standardized residual values. A histogram distributed in a bell shape and centered at zero indicates that the model does not systematically overpredict or underpredict any particular range of target values.
In the following graphic, the standardized residual values indicate that the model is fitting the data well. If the graph showed values far away from the center value, it would indicate that those values don't fit the model well.
![\[Standardized residual value close to zero, indicating that the model fits the data well.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-residual-histogram.png)
# Autopilot notebooks generated to manage AutoML tasks
Amazon SageMaker Autopilot manages the key tasks in an automatic machine learning (AutoML) process using an AutoML job. The AutoML job creates three notebook-based reports that describe the plan that Autopilot follows to generate candidate models.
A candidate model consists of a (pipeline, algorithm) pair. First, there’s a **data exploration** notebook that describes what Autopilot learned about the data that you provided. Second, there’s a **candidate definition** notebook, which uses the information about the data to generate candidates. Third, a **model insights** report that can help detail the performance characteristics of the best model in the leaderboard of an Autopilot experiment.
**Topics**
+ [Autopilot data exploration report](autopilot-data-exploration-report.md)
+ [Find and run the candidate definition notebook](autopilot-candidate-generation-notebook.md)
You can run these notebooks in Amazon SageMaker AI, or locally, if you have installed the [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable). You can share the notebooks just like any other SageMaker Studio Classic notebook. The notebooks are created for you to conduct experiments. For example, you could edit the following items in the notebooks:
+ Preprocessors used on the data
+ Amount of hyperparameter optimization (HPO) runs and their parallelism
+ Algorithms to try
+ Instance types used for the HPO jobs
+ Hyperparameter ranges
Modifications to the candidate definition notebook are encouraged as a learning tool. With this capability, you learn how decisions made during the machine learning process impact your results.
**Note**
When you run the notebooks in your default instance, you incur baseline costs. However, when you run HPO jobs from the candidate notebook, these jobs use additional compute resources that incur additional costs.
# Autopilot data exploration report
Amazon SageMaker Autopilot cleans and pre-processes your dataset automatically. High-quality data improves machine learning efficiency and produces models that make more accurate predictions.
There are issues with customer-provided datasets that cannot be fixed automatically without the benefit of some domain knowledge. Large outlier values in the target column for regression problems, for example, may cause suboptimal predictions for the non-outlier values. Outliers may need to be removed depending on the modeling objective. If a target column is included by accident as one of the input features, the final model will validate well, but be of little value for future predictions.
To help customers discover these sorts of issues, Autopilot provides a data exploration report that contains insights into potential issues with their data. The report also suggests how to handle the issues.
A data exploration notebook containing the report is generated for every Autopilot job. The report is stored in an Amazon S3 bucket and can be accessed from your output path. The path of the data exploration report usually adheres to the following pattern.
```
[s3 output path]/[name of the automl job]/sagemaker-automl-candidates/[name of processing job used for data analysis]/notebooks/SageMaker AIAutopilotDataExplorationNotebook.ipynb
```
The location of the data exploration notebook can be obtained from the Autopilot API using the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html) operation response, which is stored in [DataExplorationNotebookLocation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobArtifacts.html#sagemaker-Type-AutoMLJobArtifacts-DataExplorationNotebookLocation).
When running Autopilot from SageMaker Studio Classic, you can open the data exploration report using the following steps:
1. Choose the **Home** icon ![\[Black square icon representing a placeholder or empty image.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/studio/icons/house.png) from the *left navigation pane* to view the top-level **Amazon SageMaker Studio Classic** navigation menu.
1. Select the **AutoML** card from the main working area. This opens a new **Autopilot** tab.
1. In the **Name** section, select the Autopilot job that has the data exploration notebook that you want to examine. This opens a new **Autopilot job** tab.
1. Select **Open data exploration notebook** from the top right section of the **Autopilot job** tab.
The data exploration report is generated from your data before the training process begins. This allows you to stop Autopilot jobs that might lead to meaningless results. Likewise, you can address any issues or improvements with your dataset before rerunning Autopilot. This way, you can use your domain expertise to improve the data quality manually, before you train a model on a better-curated dataset.
The data report contains only static markdown and can be opened in any Jupyter environment. The notebook that contains the report can be converted to other formats, such as PDF or HTML. For more information about conversions, see [Using the nbconvert script to convert Jupyter notebooks to other formats.](https://nbconvert.readthedocs.io/en/latest/usage.html ).
**Topics**
+ [Dataset Summary](#autopilot-data-exploration-report-dataset-summary)
+ [Target Analysis](#autopilot-data-exploration-report-target-analysis)
+ [Data Sample](#autopilot-data-exploration-report-data-sample)
+ [Duplicate rows](#autopilot-data-exploration-report-duplicate-rows)
+ [Cross column correlations](#autopilot-data-exploration-report-cross-column-correlations)
+ [Anomalous Rows](#autopilot-data-exploration-report-cross-anomolous-rows)
+ [Missing values, cardinality, and descriptive statistics](#autopilot-data-exploration-report-description-statistics-and-values)
## Dataset Summary
This **Dataset Summary** provides key statistics characterizing your dataset including the number of rows, columns, percent duplicate rows and missing target values. It is intended to provide you with a quick alert when there are issue with your dataset that Amazon SageMaker Autopilot has detected and that are likely to require your intervention. The insights are surfaced as warnings that are classified as being of either “high” or “low” severity. The classification depends on the level of confidence that the issue will adversely impact the performance of the model.
The high and low severity insights appear in the summary as pop-ups. For most of the insights, recommendations are offered for how to confirm that there is an issue with the dataset that requires your attention. Proposals are also provided for how to resolve the issues.
Autopilot provides additional statistics about missing or not valid target values in our dataset to help you detect other issues that may not be captured by high severity insights. An unexpected number of columns of a particular type might indicate that some columns that you want to use may be missing from the dataset. It could also indicate that there was an issue with how the data was prepared or stored. Fixing these data problems brought to your attention by Autopilot can improve the performance of the machine learning models trained on your data.
High severity insights are shown in the summary section and in other relevant sections in the report. Examples of high and low-severity insights are usually given depending on the section of the data report.
## Target Analysis
Various high and low-severity insights are shown in this section related to the distribution of values in the target column. Check that target column contains the correct values. Incorrect values in target column will likely result in a machine learning model that doesn't serve the intended business purpose. Several data insights of high and low severity are present in this section. Here are several examples.
+ **Outlier target values **- Skewed or unusual target distribution for regression, such as heavy tailed targets.
+ **High or low target cardinality **- Infrequent number of class labels or a large number of unique classes for classification.
For both regression and classification problem types, not valid values such as numeric infinity, `NaN` or empty space in target column are surfaced. Depending on the problem type, different dataset statistics are presented. A distribution of target column values for a regression problem allows you to verify if the distribution is what you expected.
The following screenshot shows an Autopilot data report, which includes statistics such as the mean, median, minimum, maximum, percentage of outliers in your dataset. The screenshot also includes a histogram showing the distribution of labels in the target column. The histogram shows **Target Column Values** on the horizontal axis and **Count **on the vertical axis. A box highlights the **Outliers Percentage** section of the screenshot to indicate where this statistic appears.
![\[Autopilot data report on the distribution of target column values.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-data-report-target-analysis.png)
Multiple statistics are shown regarding target values and their distribution. If any of the outliers, not valid values, or missing percentages are greater than zero, these values are surfaced so you can investigate why your data contains unusable target values. Some unusable target values are highlighted as a low severity insight warning.
In the following screenshot, a ` symbol was added accidentally to the target column, which prevented the numeric value of the target from being parsed. A **Low severity insight: "Invalid target values"** warning appears. The warning in this example states "0.14% of the labels in the target column could not be converted to numeric values. The most common non-numeric values are: ["-3.8e-05","-9-05","-4.7e-05","-1.4999999999999999e-05","-4.3e-05"]. That usually indicates that there are problems with data collection or processing. Amazon SageMaker Autopilot ignores all observations with invalid target label."
![\[Autopilot data report low severity warning about invalid target values.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-data-report-target-analysis-invalid-target-values.png)
Autopilot also provides a histogram showing the distribution of labels for classification.
The following screenshot shows an example of statistics given for your target column including the number of classes, missing or not valid values. A histogram with **Target Label** on the horizontal axis and **Frequency** on the vertical axis shows the distribution of each label category.
![\[Autopilot data report high cardinality for classification.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-data-report-target-analysis-invalid-classification.png)
**Note**
You can find definitions of all the terms presented in this and other sections in **Definitions** section at the bottom of the report notebook.
## Data Sample
Autopilot presents an actual sample of your data to help you spot issues with your dataset. The sample table scrolls horizontally. Inspect the sample data to verify that all the necessary columns are present in the dataset.
Autopilot also calculates a measure of prediction power, that can be used to identify a linear or nonlinear relationship between a feature and the target variable. A value of `0` indicates that the feature has no predictive value in predicting the target variable. A value of `1` indicates the highest predictive power for the target variable. For more information on predictive power, see the **Definitions** section.
**Note**
It is not recommended that you use prediction power as a substitute for feature importance. Only use it if you're certain that prediction power is an appropriate measure for your use case.
The following screenshot shows example data sample. The top row contains the prediction power of each column in your dataset. The second row contains the column data type. Subsequent rows contain the labels. The columns contain the target column followed by each feature column. Each feature column has an associated prediction power, highlighted in this screenshot, with a box. In this example, the column containing the feature `x51` has a predictive power of `0.68` for the target variable `y`. The feature `x55` is slightly less predictive with a prediction power of `0.59`.
![\[Autopilot data report data sample prediction power.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-data-report-data-sample-prediction.png)
## Duplicate rows
If duplicate rows are present in the dataset, Amazon SageMaker Autopilot displays a sample of them.
**Note**
It is not recommended to balance a dataset by up-sampling before providing it to Autopilot. This may result in inaccurate validation scores for the models trained by Autopilot, and the models that are produced may be unusable.
## Cross column correlations
Autopilot uses the Pearson's correlation coefficient, a measure of linear correlation between two features, to populate a correlation matrix. In the correlation matrix, numeric features are plotted on both the horizontal and vertical axes, with the Pearson's correlation coefficient plotted at their intersections. The higher the correlation between two features, the higher the coefficient, with a maximum value of `|1|`.
+ A value of `-1` indicates that the features are perfectly negatively correlated.
+ A value of `1`, which occurs when a feature is correlated with itself, indicates perfect positive correlation.
You can use the information in the correlation matrix to remove highly correlated features. A smaller number of features reduces chances of overfitting a model and can reduce the costs of production in two ways. It lessens the Autopilot runtime needed and, for some applications, can make data collection procedures cheaper.
The following screenshot shows an example of a correlation matrix between `7` features. Each feature is displayed in a matrix on both the horizontal and vertical axes. The Pearson's correlation coefficient is displayed at the intersection between two features. Each feature intersection has a color tone associated with it. The higher the correlation, the darker the tone. The darkest tones occupy the diagonal of the matrix, where each feature is correlated with itself, representing perfect correlation.
![\[Autopilot data report data cross-correlation matrix.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-data-report-data-cross-column-statistics.png)
## Anomalous Rows
Amazon SageMaker Autopilot detects which rows in your dataset might be anomalous. It then assigns an anomaly score to each row. Rows with negative anomaly scores are considered anomalous.
The following screenshot shows the output from an Autopilot analysis for rows containing anomalies. A column containing an anomalous score appears next to the dataset columns for each row.
![\[Autopilot dataset with anomalous rows, showing negative anomaly scores.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-data-report-data-anomalous-rows.png)
## Missing values, cardinality, and descriptive statistics
Amazon SageMaker Autopilot examines and reports on properties of the individual columns of your dataset. In each section of the data report that presents this analysis, the content is arranged in order. This is so you can check the most “suspicious” values first. Using these statistics you can improve contents of individual columns, and improve the quality of the model produced by Autopilot.
Autopilot calculates several statistics on the categorical values in columns that contain them. These include the number of unique entries and, for text, the number of unique words.
Autopilot calculates several standard statistics on the numerical values in columns that contain them. The following image depicts these statistics, including the mean, median, minimum and maximum values, and the percentages of numerical types and of outlier values.
![\[Autopilot data report statistics on columns with numerical values.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-data-report-data-descriptive-statistics.png)
# Find and run the candidate definition notebook
The candidate definition notebook contains each suggested preprocessing step, algorithm, and hyperparameter ranges.
You can choose which candidate to train and tune in two ways. The first, by running sections of the notebook. The second, by running the entire notebook to optimize all candidates to identify a best candidate. If you run the entire notebook, only the best candidate is displayed after job completion.
To run Autopilot from SageMaker Studio Classic, open the candidate definition notebook by following these steps:
1. Choose the **Home** icon ![\[Black square icon representing a placeholder or empty image.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/studio/icons/house.png) from the left navigation pane to view the top-level **Amazon SageMaker Studio Classic** navigation menu.
1. Select the **AutoML** card from the main working area. This opens a new **Autopilot** tab.
1. In the **Name** section, select the Autopilot job that has the candidate definition notebook that you want to examine. This opens a new **Autopilot job** tab.
1. Choose **Open candidate generation notebook** from the top right section of the **Autopilot job** tab. This opens a new read-only preview of the **Amazon SageMaker Autopilot Candidate Definition Notebook**.
To run the candidate definition notebook, follow these steps:
1. Choose **Import notebook** at the top right of the **Amazon SageMaker Autopilot Candidate Definition Notebook** tab. This opens a tab to set up a new notebook environment to run the notebook.
1. Select an existing SageMaker **Image** or use a **Custom Image**.
1. Select a **Kernel**, an **Instance type**, and an optional **Start-up script**.
You can now run the notebook in this new environment.
# Configure inference output in generated containers
Autopilot generates an ordered [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ContainerDefinition.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ContainerDefinition.html) list. This can be used to build a model to deploy in a machine learning pipeline. This model can be used for online hosting and inference.
Customers can list inference container definitions with the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListCandidateForAutoMLJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListCandidateForAutoMLJob.html) API. The list of inference container definitions that represent the best candidate is also available in the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html) response.
## Inference container definitions for regression and classification problem types
Autopilot generates inference containers specific to the [training mode](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-model-support-validation.html#autopilot-training-mode) and the [problem type](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-datasets-problem-types.html#autopilot-problem-types) of the job.
### Container definitions for hyperparameter optimization (HPO) mode
+ **Regression**: HPO generates two containers:
1. A feature engineering container that transforms the original features into features that the regression algorithms can train on.
1. An algorithm container that transforms features and generates a regression score for the dataset.
+ **Classification**: HPO generates three containers:
1. A feature engineering container that transforms the original features into features that the classification algorithms can train on.
1. An algorithm container that generates the `predicted_label` with the highest probability. This container can also produce the various probabilities associated with the classification outcomes in the inference response.
1. A feature engineering container that performs post-processing of the algorithm prediction. For example, it can perform an inverse transform on the predicted label and change it to the original label.
### Container definitions for ensembling mode
In ensembling mode, both regression and classification problem types have only one inference container. This inference container transforms the features and generates the predictions based on problem type.
## Inference responses per problem type
### Inference responses for classification models
For classification inference containers, you can select the content of the inference response by using four predefined keys:
+ `predicted_label`: The label with the highest probability of predicting the correct label, as determined by Autopilot.
+ `probability`:
+ **HPO models:** The probability of the `True` class for binary classification. The probability of the `predicted_label` for multiclass classification.
+ **Ensemble models:** The probability of the `predicted_label` for binary and multiclass classification.
+ `probabilities`: The list of probabilities for all corresponding classes.
+ `labels`: The list of all labels.
For example, for a binary classification problem, if you pass the inference response keys `['predicted_label', 'probability', 'probabilities', 'labels']` and the output response appears as `[1, 0.1, "[0.9, 0.1]", "['1', '0']"]`, you should interpret it as follows:
1. `predicted_label` equals `1` because label "1" has a higher probability (`0.9` in this case).
1. For HPO models, `probability` equals `0.1` which is the probability of the `positive_class` (`0` in this case) selected by Autopilot.
For Ensemble models, `probability` equals `0.9` which is the probability of the `predicted_label`.
1. `probabilities` lists the `probability` of each label in `labels`.
1. `labels` are the unique labels in the dataset, where the second label ("0" in this case) is the `positive_class` selected by Autopilot.
By default, inference containers are configured to generate only the `predicted_label`. To select additional inference content, you can update the `inference_response_keys` parameter to include up to these three environment variables:
+ `SAGEMAKER_INFERENCE_SUPPORTED`: This is set to provide hints to you about what content each container supports.
+ `SAGEMAKER_INFERENCE_INPUT`: This should be set to the keys that the container expects in input payload.
+ `SAGEMAKER_INFERENCE_OUTPUT`: This should be populated with the set of keys that the container outputs.
### Inference responses for classification models in HPO mode
This section shows how to configure the inference response from classification models using hyperparameter optimization (HPO) mode.
To choose the inference response content in HPO mode: Add the `SAGEMAKER_INFERENCE_INPUT` and `SAGEMAKER_INFERENCE_OUTPUT` variables to the second and third containers that are generated in HPO mode for classification problems.
The keys supported by the second container (algorithm) are predicted\$1label, probability, and probabilities. Note that `labels` is deliberately not added to `SAGEMAKER_INFERENCE_SUPPORTED`.
The keys supported by the third classification model container are `predicted_label`, `labels`, `probability`, and `probabilities`. Therefore, the `SAGEMAKER_INFERENCE_SUPPORTED` environment includes the names of these keys.
To update the definition of the inference containers to receive `predicted_label` and `probability`, use the following code example.
```
containers[1]['Environment'].update({'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label, probability'})
containers[2]['Environment'].update({'SAGEMAKER_INFERENCE_INPUT': 'predicted_label, probability'})
containers[2]['Environment'].update({'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label, probability'})
```
The following code example updates the definition of the inference containers to receive `predicted_label`, `probabilities`, and `labels`. Do not pass the `labels` to the second container (the algorithm container), because it is generated by the third container independently.
```
containers[1]['Environment'].update({'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label,probabilities'})
containers[2]['Environment'].update({'SAGEMAKER_INFERENCE_INPUT': 'predicted_label,probabilities'})
containers[2]['Environment'].update({'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label, probabilities,labels'})
```
The following collapsible sections provide code examples for AWS SDK for Python (Boto3) and for SageMaker SDK for Python. Each section shows how to select the content of the inference responses in HPO mode for the respective code example.
#### AWS SDK for Python (Boto3)
```
import boto3
sm_client = boto3.client('sagemaker', region_name='')
role = ''
input_data = ''
output_path = ''
best_candidate = sm_client.describe_auto_ml_job(AutoMLJobName='')['BestCandidate']
best_candidate_containers = best_candidate['InferenceContainers']
best_candidate_name = best_candidate['CandidateName']
best_candidate_containers[1]['Environment'].update({'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label, probability'})
best_candidate_containers[2]['Environment'].update({'SAGEMAKER_INFERENCE_INPUT': 'predicted_label, probability'})
best_candidate_containers[2]['Environment'].update({'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label, probability'})
# create model
reponse = sm_client.create_model(
ModelName = '',
ExecutionRoleArn = role,
Containers = best_candidate_containers
)
# Lauch Transform Job
response = sm_client.create_transform_job(
TransformJobName='',
ModelName='',
TransformInput={
'DataSource': {
'S3DataSource': {
'S3DataType': 'S3Prefix',
'S3Uri': input_data
}
},
'ContentType': "text/CSV",
'SplitType': 'Line'
},
TransformOutput={
'S3OutputPath': output_path,
'AssembleWith': 'Line',
},
TransformResources={
'InstanceType': 'ml.m4.xlarge',
'InstanceCount': 1,
},
)
```
#### SageMaker SDK for Python
```
from sagemaker import AutoML
aml = AutoML.attach(auto_ml_job_name='')
aml_best_model = aml.create_model(name='',
candidate=None,
inference_response_keys**=['probabilities', 'labels'])
aml_transformer = aml_best_model.transformer(accept='text/csv',
assemble_with='Line',
instance_type='ml.m5.xlarge',
instance_count=1,)
aml_transformer.transform('',
content_type='text/csv',
split_type='Line',
job_name='',
wait=True)
```
### Inference responses for classification models in ensembling mode
This section shows how to configure the inference response from classification models using ensembling mode.
In **ensembling mode**, to choose the content of the inference response, update the `SAGEMAKER_INFERENCE_OUTPUT` environment variable.
The keys supported by the classification model container are `predicted_label`, `labels`, `probability`, and `probabilities`. These keys are included in the `SAGEMAKER_INFERENCE_SUPPORTED` environment.
To update the inference container definition to receive `predicted_label` and `probability`, refer to the following code example.
```
containers[0]['Environment'].update({'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label, probability'})
```
The following collapsible section provides a code example for selecting the content of the inference responses in ensembling mode. The example uses AWS SDK for Python (Boto3).
#### AWS SDK for Python (Boto3)
```
import boto3
sm_client = boto3.client('sagemaker', region_name='')
role = ''
input_data = ''
output_path = ''
best_candidate = sm_client.describe_auto_ml_job(AutoMLJobName='')['BestCandidate']
best_candidate_containers = best_candidate['InferenceContainers']
best_candidate_name = best_candidate['CandidateName']
*best_candidate_containers[0]['Environment'].update({'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label, probability'})
*
# create model
reponse = sm_client.create_model(
ModelName = '',
ExecutionRoleArn = role,
Containers = best_candidate_containers
)
# Lauch Transform Job
response = sm_client.create_transform_job(
TransformJobName='',
ModelName='',
TransformInput={
'DataSource': {
'S3DataSource': {
'S3DataType': 'S3Prefix',
'S3Uri': input_data
}
},
'ContentType': "text/CSV",
'SplitType': 'Line'
},
TransformOutput={
'S3OutputPath': output_path,
'AssembleWith': 'Line',
},
TransformResources={
'InstanceType': 'ml.m4.xlarge',
'InstanceCount': 1,
},
)
```
The following collapsible section provides a code example that is identical to the SageMaker SDK for Python example for HPO. It is included for your convenience.
#### SageMaker SDK for Python
The following HPO code example uses SageMaker SDK for Python.
```
from sagemaker import AutoML
aml = AutoML.attach(auto_ml_job_name='')
aml_best_model = aml.create_model(name='',
candidate=None,
*inference_response_keys**=['probabilities', 'labels'])*
aml_transformer = aml_best_model.transformer(accept='text/csv',
assemble_with='Line',
instance_type='ml.m5.xlarge',
instance_count=1,)
aml_transformer.transform('',
content_type='text/csv',
split_type='Line',
job_name='',
wait=True)
```
# Create an Image Classification Job using the AutoML API
The following instructions show how to create an Amazon SageMaker Autopilot job as a pilot experiment for image classification problem types using SageMaker [API Reference](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-reference.html).
**Note**
Tasks such as text and image classification, time-series forecasting, and fine-tuning of large language models are exclusively available through the version 2 of the [AutoML REST API](autopilot-reference.md). If your language of choice is Python, you can refer to [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_auto_ml_job_v2.html) or the [AutoMLV2 object](https://sagemaker.readthedocs.io/en/stable/api/training/automlv2.html#sagemaker.automl.automlv2.AutoMLV2) of the Amazon SageMaker Python SDK directly.
Users who prefer the convenience of a user interface can use [Amazon SageMaker Canvas](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-getting-started.html) to access pre-trained models and generative AI foundation models, or create custom models tailored for specific text, image classification, forecasting needs, or generative AI.
You can create an Autopilot image classification experiment programmatically by calling the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html) API action in any language supported by Amazon SageMaker Autopilot or the AWS CLI.
For information on how this API action translates into a function in the language of your choice, see the [ See Also](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_SeeAlso) section of `CreateAutoMLJobV2` and choose an SDK. As an example, for Python users, see the full request syntax of `[create\$1auto\$1ml\$1job\$1v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_auto_ml_job_v2)` in AWS SDK for Python (Boto3).
The following is a collection of mandatory and optional input request parameters for the `CreateAutoMLJobV2` API action used in image classification.
## Required parameters
When calling `[CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html)` to create an Autopilot experiment for image classification, you must provide the following values:
+ An `[AutoMLJobName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_RequestSyntax)` to specify the name of your job.
+ At least one `[AutoMLJobChannel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html)` in `[AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig)` to specify your data source.
+ An `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)` of type `[ImageClassificationJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ImageClassificationJobConfig.html)`.
+ An `[OutputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html)` to specify the Amazon S3 output path to store the artifacts of your AutoML job.
+ A `[RoleArn](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-RoleArn)` to specify the ARN of the role used to access your data.
All other parameters are optional.
## Optional parameters
The following sections provide details of some optional parameters that you can pass to your image classification AutoML job.
### How to specify the training and validation datasets of an AutoML job
You can provide your own validation dataset and custom data split ratio, or let Autopilot split the dataset automatically.
Each [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html) object (see the required parameter [AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker-api/src/AWSSageMakerAPIDoc/build/server-root/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig)) has a `ChannelType`, which can be set to either `training` or `validation` values that specify how the data is to be used when building a machine learning model.
At least one data source must be provided and a maximum of two data sources is allowed: one for training data and one for validation data. How you split the data into training and validation datasets depends on whether you have one or two data sources.
How you split the data into training and validation datasets depends on whether you have one or two data sources.
+ If you only have **one data source**, the `ChannelType` is set to `training` by default and must have this value.
+ If the `ValidationFraction` value in [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html) is not set, 0.2 (20%) of the data from this source is used for validation by default.
+ If the `ValidationFraction` is set to a value between 0 and 1, the dataset is split based on the value specified, where the value specifies the fraction of the dataset used for validation.
+ If you have **two data sources**, the `ChannelType` of one of the `AutoMLJobChannel` objects must be set to `training`, the default value. The `ChannelType` of the other data source must be set to `validation`. The two data sources must have the same format, either CSV or Parquet, and the same schema. You must not set the value for the `ValidationFraction` in this case because all of the data from each source is used for either training or validation. Setting this value causes an error.
### How to specify the automatic model deployment configuration for an AutoML job
To enable automatic deployment for the best model candidate of an AutoML job, include a `[ModelDeployConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-ModelDeployConfig)` in the AutoML job request. This will allow the deployment of the best model to a SageMaker AI endpoint. Below are the available configurations for customization.
+ To let Autopilot generate the endpoint name, set `[AutoGenerateEndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents)` to `True`.
+ To provide your own name for the endpoint, set `[AutoGenerateEndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents) to False and provide a name of your choice in [EndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents)`.
# Datasets format and objective metric for image classification
In this section we learn about the available formats for datasets used in image classification as well as the objective metric used to evaluate the predictive quality of machine learning model candidates. The metrics calculated for candidates are specified using an array of [MetricDatum](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_MetricDatum.html) types.
## Datasets formats
Autopilot supports .png, .jpg, and .jpeg image formats. If your dataset contains all .png images use `image/png`, if it contains all .jpg or .jpeg images use `image/jpeg`, and if your dataset contains a mix of image formats use `image/*`.
## Objective metric
The following list contains the names of the metrics that are currently available to measure the performance of models for image classification.
**`Accuracy`**
The ratio of the number of correctly classified items to the total number of (correctly and incorrectly) classified items. Accuracy measures how close the predicted class values are to the actual values. Values for accuracy metrics vary between zero (0) and one (1). A value of 1 indicates perfect accuracy, and 0 indicates perfect inaccuracy.
# Deploy Autopilot models for real-time inference
After you train your Amazon SageMaker Autopilot models, you can set up an endpoint and obtain predictions interactively. The following section describes the steps for deploying your model to a SageMaker AI real-time inference endpoint to get predictions from your model.
## Real-time inferencing
Real-time inference is ideal for inference workloads where you have real-time, interactive, low latency requirements. This section shows how you can use real-time inferencing to obtain predictions interactively from your model.
You can use SageMaker APIs to manually deploy the model that produced the best validation metric in an Autopilot experiment as follows.
Alternatively, you can chose the automatic deployment option when creating your Autopilot experiment. For information on setting up the automatic deployment of models, see `[ModelDeployConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-ModelDeployConfig)` in the request parameters of `[CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_RequestParameters)`. This creates an endpoint automatically.
**Note**
To avoid incurring unnecessary charges, you can delete unneeded endpoint and resources created from model deployment. For information about pricing of instances by Region, see [Amazon SageMaker Pricing](https://aws.amazon.com/sagemaker/pricing/).
1. **Obtain the candidate container definitions**
Obtain the candidate container definitions from [InferenceContainers](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html#sagemaker-Type-AutoMLCandidate-InferenceContainers). A container definition for inference refers to the containerized environment designed for deploying and running your trained SageMaker AI model to make predictions.
The following AWS CLI command example uses the [DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html) API to obtain candidate definitions for the best model candidate.
```
aws sagemaker describe-auto-ml-job-v2 --auto-ml-job-name job-name --region region
```
1. **List candidates**
The following AWS CLI command example uses the [ListCandidatesForAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListCandidatesForAutoMLJob.html) API to list all model candidates.
```
aws sagemaker list-candidates-for-auto-ml-job --auto-ml-job-name --region
```
1. **Create a SageMaker AI model**
Use the container definitions from the previous steps and a candidate of your choice to create a SageMaker AI model by using the [CreateModel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) API. See the following AWS CLI command as an example.
```
aws sagemaker create-model --model-name '' \
--containers [', , ]' \
--execution-role-arn '' --region '
```
1. **Create an endpoint configuration**
The following AWS CLI command example uses the [CreateEndpointConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html) API to create an endpoint configuration.
```
aws sagemaker create-endpoint-config --endpoint-config-name '' \
--production-variants '' \
--region ''
```
1. **Create the endpoint**
The following AWS CLI example uses the [CreateEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpoint.html) API to create the endpoint.
```
aws sagemaker create-endpoint --endpoint-name '' \
--endpoint-config-name '' \
--region ''
```
Check the progress of your endpoint deployment by using the [DescribeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpoint.html) API. See the following AWS CLI command as an example.
```
aws sagemaker describe-endpoint —endpoint-name '' —region
```
After the `EndpointStatus` changes to `InService`, the endpoint is ready to use for real-time inference.
1. **Invoke the endpoint**
The following command structure invokes the endpoint for real-time inferencing.
```
aws sagemaker invoke-endpoint --endpoint-name '' \
--region '' --body '' [--content-type] ''
```
# Explainability report
Amazon SageMaker Autopilot provides an explainability report to help explain how a best model candidate makes predictions for image classification problems. This report can assist ML engineers, product managers, and other internal stakeholders in understanding the characteristics of the model. Both consumers and regulators rely on transparency in machine learning to trust and interpret decisions made on model predictions. You can use these explanations for auditing and meeting regulatory requirements, establishing trust in the model, supporting human decision-making, and debugging and improving model performance.
The Autopilot explanatory functionality for image classification uses a visual class activation map (CAM) approach that produces a heatmap where the distribution and intensity of each color highlights the areas of an image that contribute the most to a specific prediction. This approach relies on principal components derived from an implementation of [Eigen-CAM](https://arxiv.org/ftp/arxiv/papers/2008/2008.00299.pdf).
Autopilot generates the explainability report as a JSON file. The report includes analysis details that are based on the validation dataset. Each image used to generate the report contains the following information:
+ `input_image_uri`: The Amazon S3 URI to the input image taken as input for the heatmap.
+ `heatmap_image_uri`: The Amazon S3 URI to the heatmap image generated by Autopilot.
+ `predicted_label`: The label class predicted by best model trained by Autopilot.
+ `probability`: The confidence with which the `predicted_label` is predicted.
You can find the Amazon S3 prefix to the explainability artifacts generated for the best candidate in the response to `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` at `[BestCandidate.CandidateProperties.CandidateArtifactLocations.Explainability](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateArtifactLocations.html#sagemaker-Type-CandidateArtifactLocations-Explainability)`.
The following examples illustrates what the heatmaps look like on few samples from [Oxford-IIIT Pet Dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/). The heatmap image displays color gradients that indicate the relative importance of different features within the image. The red color represents regions with greater importance in predicting the "predicted\$1label" of the input image compared to the features represented by the blue color.
****
| Input Image | Heatmap Image |
| --- | --- |
| ![\[The original image of a dog.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-image-classification-explainability-img1-input.png) | ![\[A dog with a heatmap highlighting the regions with the greater contribution to the predicted label.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-image-classification-explainability-img1-output.png) |
| ![\[The original image of a cat.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-image-classification-explainability-img2-input.png) | ![\[A cat with a heatmap highlighting the regions with the greater contribution to the predicted label.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-image-classification-explainability-img2-output.png) |
# Model performance report
An Amazon SageMaker AI model quality report (also referred to as performance report) provides insights and quality information for the best model candidate generated by an AutoML job. This includes information about the job details, model problem type, objective function, and various metrics. This section details the content of a performance report for image classification problems and explains how to access the metrics as raw data in a JSON file.
You can find the Amazon S3 prefix to the model quality report artifacts generated for the best candidate in the response to `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` at `[BestCandidate.CandidateProperties.CandidateArtifactLocations.ModelInsights](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateArtifactLocations.html#sagemaker-Type-CandidateArtifactLocations-ModelInsights)`.
The performance report contains two sections:
+ The first section contains details about the Autopilot job that produced the model.
+ The second section contains a model quality report with various performance metrics.
## Autopilot job details
This first section of the report gives some general information about the Autopilot job that produced the model. These details include the following information:
+ Autopilot candidate name: The name of the best model candidate.
+ Autopilot job name: The name of the job.
+ Problem type: The problem type. In our case, *image classification*.
+ Objective metric: The objective metric used to optimize the performance of the model. In our case, *Accuracy*.
+ Optimization direction: Indicates whether to minimize or maximize the objective metric.
## Model quality report
Model quality information is generated by Autopilot model insights. The report's content that is generated depends on the problem type it addressed. The report specifies the number of rows that were included in the evaluation dataset and the time at which the evaluation occurred.
### Metrics tables
The first part of the model quality report contains metrics tables. These are appropriate for the type of problem that the model addressed.
The following image is an example of a metrics table generates by Autopilot for an image or text classification problem. It shows the metric name, value, and standard deviation.
![\[Amazon SageMaker Autopilot model insights image or text classification metrics report example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-multiclass-metrics-report.png)
### Graphical model performance information
The second part of the model quality report contains graphical information to help you evaluate model performance. The contents of this section depend on the selected problem type.
#### Confusion matrix
A confusion matrix provides a way to visualize the accuracy of the predictions made by a model for binary and multiclass classification for different problems.
A summary of the graph's components of **false positive rate **(FPR) and **true positive rate **(TPR) are defined as follows.
+ Correct predictions
+ **True positive** (TP): The predicted the value is 1, and the true value is 1.
+ **True negative** (TN): The predicted the value is 0, and the true value is 0.
+ Erroneous predictions
+ **False positive** (FP): The predicted the value is 1, but the true value is 0.
+ **False negative** (FN): The predicted the value is 0, but the true value is 1.
The confusion matrix in the model quality report contains the following.
+ The number and percentage of correct and incorrect predictions for the actual labels
+ The number and percentage of accurate predictions on the diagonal from the upper-left to the lower-right corner
+ The number and percentage of inaccurate predictions on the diagonal from the upper-right to the lower-left corner
The incorrect predictions on a confusion matrix are the confusion values.
The following diagram is an example of a confusion matrix for a multi-class classification problem. The confusion matrix in the model quality report contains the following.
+ The vertical axis is divided into three rows containing three different actual labels.
+ The horizontal axis is divided into three columns containing labels that were predicted by the model.
+ The color bar assigns a darker tone to a larger number of samples to visually indicate the number of values that were classified in each category.
In the example below, the model correctly predicted actual 354 values for label **f**, 1094 values for label **i** and 852 values for label **m**. The difference in tone indicates that the dataset is not balanced because there are many more labels for the value **i** than for **f** or **m**.
![\[Amazon SageMaker Autopilot multiclass confusion matrix example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-confusion-matrix-multiclass.png)
The confusion matrix in the model quality report provided can accommodate a maximum of 15 labels for multiclass classification problem types. If a row corresponding to a label shows a `Nan` value, it means that the validation dataset used to check model predictions does not contain data with that label.
# Create an AutoML job for text classification using the API
The following instructions show how to create an Amazon SageMaker Autopilot job as a pilot experiment for text classification problem types using SageMaker [API Reference](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-reference.html).
**Note**
Tasks such as text and image classification, time-series forecasting, and fine-tuning of large language models are exclusively available through the version 2 of the [AutoML REST API](autopilot-reference.md). If your language of choice is Python, you can refer to [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_auto_ml_job_v2.html) or the [AutoMLV2 object](https://sagemaker.readthedocs.io/en/stable/api/training/automlv2.html#sagemaker.automl.automlv2.AutoMLV2) of the Amazon SageMaker Python SDK directly.
Users who prefer the convenience of a user interface can use [Amazon SageMaker Canvas](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-getting-started.html) to access pre-trained models and generative AI foundation models, or create custom models tailored for specific text, image classification, forecasting needs, or generative AI.
You can create an Autopilot text classification experiment programmatically by calling the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html) API action in any language supported by Amazon SageMaker Autopilot or the AWS CLI.
For information on how this API action translates into a function in the language of your choice, see the [ See Also](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_SeeAlso) section of `CreateAutoMLJobV2` and choose an SDK. As an example, for Python users, see the full request syntax of `[create\$1auto\$1ml\$1job\$1v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_auto_ml_job_v2)` in AWS SDK for Python (Boto3).
The following is a collection of mandatory and optional input request parameters for the `CreateAutoMLJobV2` API action used in text classification.
## Required parameters
When calling `[CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html)` to create an Autopilot experiment for text classification, you must provide the following values:
+ An `[AutoMLJobName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_RequestSyntax)` to specify the name of your job.
+ At least one `[AutoMLJobChannel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html)` in `[AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig)` to specify your data source.
+ An `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)` of type `[TextClassificationJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TextClassificationJobConfig.html)`.
+ An `[OutputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html)` to specify the Amazon S3 output path to store the artifacts of your AutoML job.
+ A `[RoleArn](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-RoleArn)` to specify the ARN of the role used to access your data.
All other parameters are optional.
## Optional parameters
The following sections provide details of some optional parameters that you can pass to your text classification AutoML job.
### How to specify the training and validation datasets of an AutoML job
You can provide your own validation dataset and custom data split ratio, or let Autopilot split the dataset automatically.
Each [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html) object (see the required parameter [AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker-api/src/AWSSageMakerAPIDoc/build/server-root/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig)) has a `ChannelType`, which can be set to either `training` or `validation` values that specify how the data is to be used when building a machine learning model.
At least one data source must be provided and a maximum of two data sources is allowed: one for training data and one for validation data. How you split the data into training and validation datasets depends on whether you have one or two data sources.
How you split the data into training and validation datasets depends on whether you have one or two data sources.
+ If you only have **one data source**, the `ChannelType` is set to `training` by default and must have this value.
+ If the `ValidationFraction` value in [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html) is not set, 0.2 (20%) of the data from this source is used for validation by default.
+ If the `ValidationFraction` is set to a value between 0 and 1, the dataset is split based on the value specified, where the value specifies the fraction of the dataset used for validation.
+ If you have **two data sources**, the `ChannelType` of one of the `AutoMLJobChannel` objects must be set to `training`, the default value. The `ChannelType` of the other data source must be set to `validation`. The two data sources must have the same format, either CSV or Parquet, and the same schema. You must not set the value for the `ValidationFraction` in this case because all of the data from each source is used for either training or validation. Setting this value causes an error.
### How to specify the automatic model deployment configuration for an AutoML job
To enable automatic deployment for the best model candidate of an AutoML job, include a `[ModelDeployConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-ModelDeployConfig)` in the AutoML job request. This will allow the deployment of the best model to a SageMaker AI endpoint. Below are the available configurations for customization.
+ To let Autopilotgenerate the endpoint name, set `[AutoGenerateEndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents)` to `True`.
+ To provide your own name for the endpoint, set `[AutoGenerateEndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents) to False and provide a name of your choice in [EndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents)`.
# Datasets format and objective metric for text classification
In this section we learn about the available formats for datasets used in text classification as well as the metric used to evaluate the predictive quality of machine learning model candidates. The metrics calculated for candidates are specified using an array of [MetricDatum](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_MetricDatum.html) types.
## Datasets formats
Autopilot supports tabular data formatted as CSV files or as Parquet files. For tabular data, each column contains a feature with a specific data type and each row contains an observation. The properties of these two file formats differ considerably.
+ **CSV** (comma-separated-values) is a row-based file format that stores data in human readable plaintext which a popular choice for data exchange as they are supported by a wide range of applications.
+ **Parquet** is a column-based file format where the data is stored and processed more efficiently than row-based file formats. This makes them a better option for big data problems.
The **data types** accepted for columns include numerical, categorical, text.
Autopilot supports building machine learning models on large datasets up to hundreds of GBs. For details on the default resource limits for input datasets and how to increase them, see [Amazon SageMaker Autopilot quotas](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-quotas.html).
## Objective metric
The following list contains the names of the metrics that are currently available to measure the performance of models for text classification.
**`Accuracy`**
The ratio of the number of correctly classified items to the total number of (correctly and incorrectly) classified items. Accuracy measures how close the predicted class values are to the actual values. Values for accuracy metrics vary between zero (0) and one (1). A value of 1 indicates perfect accuracy, and 0 indicates perfect inaccuracy.
# Deploy Autopilot models for real-time inference
After you train your Amazon SageMaker Autopilot models, you can set up an endpoint and obtain predictions interactively. The following section describes the steps for deploying your model to a SageMaker AI real-time inference endpoint to get predictions from your model.
## Real-time inferencing
Real-time inference is ideal for inference workloads where you have real-time, interactive, low latency requirements. This section shows how you can use real-time inferencing to obtain predictions interactively from your model.
You can use SageMaker APIs to manually deploy the model that produced the best validation metric in an Autopilot experiment as follows.
Alternatively, you can chose the automatic deployment option when creating your Autopilot experiment. For information on setting up the automatic deployment of models, see `[ModelDeployConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-ModelDeployConfig)` in the request parameters of `[CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_RequestParameters)`. This creates an endpoint automatically.
**Note**
To avoid incurring unnecessary charges, you can delete unneeded endpoint and resources created from model deployment. For information about pricing of instances by Region, see [Amazon SageMaker Pricing](https://aws.amazon.com/sagemaker/pricing/).
1. **Obtain the candidate container definitions**
Obtain the candidate container definitions from [InferenceContainers](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html#sagemaker-Type-AutoMLCandidate-InferenceContainers). A container definition for inference refers to the containerized environment designed for deploying and running your trained SageMaker AI model to make predictions.
The following AWS CLI command example uses the [DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html) API to obtain candidate definitions for the best model candidate.
```
aws sagemaker describe-auto-ml-job-v2 --auto-ml-job-name job-name --region region
```
1. **List candidates**
The following AWS CLI command example uses the [ListCandidatesForAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListCandidatesForAutoMLJob.html) API to list all model candidates.
```
aws sagemaker list-candidates-for-auto-ml-job --auto-ml-job-name --region
```
1. **Create a SageMaker AI model**
Use the container definitions from the previous steps and a candidate of your choice to create a SageMaker AI model by using the [CreateModel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) API. See the following AWS CLI command as an example.
```
aws sagemaker create-model --model-name '' \
--containers [', , ]' \
--execution-role-arn '' --region '
```
1. **Create an endpoint configuration**
The following AWS CLI command example uses the [CreateEndpointConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html) API to create an endpoint configuration.
```
aws sagemaker create-endpoint-config --endpoint-config-name '' \
--production-variants '' \
--region ''
```
1. **Create the endpoint**
The following AWS CLI example uses the [CreateEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpoint.html) API to create the endpoint.
```
aws sagemaker create-endpoint --endpoint-name '' \
--endpoint-config-name '' \
--region ''
```
Check the progress of your endpoint deployment by using the [DescribeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpoint.html) API. See the following AWS CLI command as an example.
```
aws sagemaker describe-endpoint —endpoint-name '' —region
```
After the `EndpointStatus` changes to `InService`, the endpoint is ready to use for real-time inference.
1. **Invoke the endpoint**
The following command structure invokes the endpoint for real-time inferencing.
```
aws sagemaker invoke-endpoint --endpoint-name '' \
--region '' --body '' [--content-type] ''
```
# Explainability report
Amazon SageMaker Autopilot provides an explainability report to help explain how a best model candidate makes predictions for text classification problems. This report can assist ML engineers, product managers, and other internal stakeholders in understanding the characteristics of the model. Both consumers and regulators rely on transparency in machine learning to trust and interpret decisions made on model predictions. You can use these explanations for auditing and meeting regulatory requirements, establishing trust in the model, supporting human decision-making, and debugging and improving model performance.
The Autopilot explanatory functionality for text classification uses the axiomatic attribution method *Integrated Gradients*. This approach relies on an implementation of [Axiomatic Attribution for Deep Network](https://arxiv.org/pdf/1703.01365.pdf).
Autopilot generates the explainability report as a JSON file. The report includes analysis details that are based on the validation dataset. Each sample used to generate the report contains the following information:
+ `text`: The input text content explained.
+ `token_scores`: The list of scores for every token in the text.
+
+ `attribution`: The score depicting the importance of the token.
+ `description.partial_text`: The partial substring that represents the token.
+ `predicted_label`: The label class predicted by the best model candidate.
+ `probability`: The confidence with which the `predicted_label` was predicted.
You can find the Amazon S3 prefix to the explainability artifacts generated for the best candidate in the response to `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` at `[BestCandidate.CandidateProperties.CandidateArtifactLocations.Explainability](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateArtifactLocations.html#sagemaker-Type-CandidateArtifactLocations-Explainability)`.
The following is an example of analysis content that you could find in the explainability artifacts.
```
{
"text": "It was a fantastic movie!",
"predicted_label": 2,
"probability": 0.9984835,
"token_scores": [
{
"attribution": 0,
"description": {
"partial_text": "It"
}
},
{
"attribution": -0.022447118861679088,
"description": {
"partial_text": "was"
}
},
{
"attribution": -0.2164326456817965,
"description": {
"partial_text": "a"
}
},
{
"attribution": 0.675,
"description": {
"partial_text": "fantastic"
}
},
{
"attribution": 0.416,
"description": {
"partial_text": "movie!"
}
}
]
}
```
In this sample of the JSON report, the explanatory functionality evaluates the text `It was a fantastic movie!` and scores the contribution of each of its token to the overall predicted label. The predicted label is `2`, which is a strong positive sentiment, with a probability of 99.85%. The JSON sample then details the contribution of each individual token to this prediction. For example, the token `fantastic` has a stronger attribution than the token `was`. It is the token that contributed the most to the final prediction.
# Model performance report
An Amazon SageMaker AI model quality report (also referred to as performance report) provides insights and quality information for the best model candidate generated by an AutoML job. This includes information about the job details, model problem type, objective function, and various metrics. This section details the content of a performance report for text classification problems and explains how to access the metrics as raw data in a JSON file.
You can find the Amazon S3 prefix to the model quality report artifacts generated for the best candidate in the response to `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` at `[BestCandidate.CandidateProperties.CandidateArtifactLocations.ModelInsights](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateArtifactLocations.html#sagemaker-Type-CandidateArtifactLocations-ModelInsights)`.
The performance report contains two sections:
+ The first section contains details about the Autopilot job that produced the model.
+ The second section contains a model quality report with various performance metrics.
## Autopilot job details
This first section of the report gives some general information about the Autopilot job that produced the model. These details include the following information:
+ Autopilot candidate name: The name of the best model candidate.
+ Autopilot job name: The name of the job.
+ Problem type: The problem type. In our case, *text classification*.
+ Objective metric: The objective metric used to optimize the performance of the model. In our case, *Accuracy*.
+ Optimization direction: Indicates whether to minimize or maximize the objective metric.
## Model quality report
Model quality information is generated by Autopilot model insights. The report's content that is generated depends on the problem type it addressed. The report specifies the number of rows that were included in the evaluation dataset and the time at which the evaluation occurred.
### Metrics tables
The first part of the model quality report contains metrics tables. These are appropriate for the type of problem that the model addressed.
The following image is an example of a metrics table generated by Autopilot for an image or text classification problem. It shows the metric name, value, and standard deviation.
![\[Amazon SageMaker Autopilot model insights image or text classification metrics report example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-multiclass-metrics-report.png)
### Graphical model performance information
The second part of the model quality report contains graphical information to help you evaluate model performance. The contents of this section depend on the selected problem type.
#### Confusion matrix
A confusion matrix provides a way to visualize the accuracy of the predictions made by a model for binary and multiclass classification for different problems.
A summary of the graph's components of **false positive rate **(FPR) and **true positive rate **(TPR) are defined as follows.
+ Correct predictions
+ **True positive** (TP): The predicted the value is 1, and the true value is 1.
+ **True negative** (TN): The predicted the value is 0, and the true value is 0.
+ Erroneous predictions
+ **False positive** (FP): The predicted the value is 1, but the true value is 0.
+ **False negative** (FN): The predicted the value is 0, but the true value is 1.
The confusion matrix in the model quality report contains the following.
+ The number and percentage of correct and incorrect predictions for the actual labels
+ The number and percentage of accurate predictions on the diagonal from the upper-left to the lower-right corner
+ The number and percentage of inaccurate predictions on the diagonal from the upper-right to the lower-left corner
The incorrect predictions on a confusion matrix are the confusion values.
The following diagram is an example of a confusion matrix for a multi-class classification problem. The confusion matrix in the model quality report contains the following.
+ The vertical axis is divided into three rows containing three different actual labels.
+ The horizontal axis is divided into three columns containing labels that were predicted by the model.
+ The color bar assigns a darker tone to a larger number of samples to visually indicate the number of values that were classified in each category.
In the example below, the model correctly predicted actual 354 values for label **f**, 1094 values for label **i** and 852 values for label **m**. The difference in tone indicates that the dataset is not balanced because there are many more labels for the value **i** than for **f** or **m**.
![\[Amazon SageMaker Autopilot multiclass confusion matrix example.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-model-insights-confusion-matrix-multiclass.png)
The confusion matrix in the model quality report provided can accommodate a maximum of 15 labels for multiclass classification problem types. If a row corresponding to a label shows a `Nan` value, it means that the validation dataset used to check model predictions does not contain data with that label.
# Create an AutoML job for time-series forecasting using the API
Forecasting in machine learning refers to the process of predicting future outcomes or trends based on historical data and patterns. By analyzing past time-series data and identifying underlying patterns, machine learning algorithms can make predictions and provide valuable insights into future behavior. In forecasting, the goal is to develop models that can accurately capture the relationship between input variables and the target variable over time. This involves examining various factors such as trends, seasonality, and other relevant patterns within the data. The collected information is then used to train a machine learning model. The trained model is capable of generating predictions by taking new input data and applying the learned patterns and relationships. It can provide forecasts for a wide range of use cases, such as sales projections, stock market trends, weather forecasts, demand forecasting, and many more.
The following instructions show how to create an Amazon SageMaker Autopilot job as a pilot experiment for time-series forecasting problem types using SageMaker [API Reference](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-reference.html).
**Note**
Tasks such as text and image classification, time-series forecasting, and fine-tuning of large language models are exclusively available through the version 2 of the [AutoML REST API](autopilot-reference.md). If your language of choice is Python, you can refer to [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_auto_ml_job_v2.html) or the [AutoMLV2 object](https://sagemaker.readthedocs.io/en/stable/api/training/automlv2.html#sagemaker.automl.automlv2.AutoMLV2) of the Amazon SageMaker Python SDK directly.
Users who prefer the convenience of a user interface can use [Amazon SageMaker Canvas](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-getting-started.html) to access pre-trained models and generative AI foundation models, or create custom models tailored for specific text, image classification, forecasting needs, or generative AI.
You can create an Autopilot time-series forecasting experiment programmatically by calling the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html) API in any language supported by Amazon SageMaker Autopilot or the AWS CLI.
For information on how this API action translates into a function in the language of your choice, see the [ See Also](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_SeeAlso) section of `CreateAutoMLJobV2` and choose an SDK. As an example, for Python users, see the full request syntax of `[create\$1auto\$1ml\$1job\$1v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_auto_ml_job_v2)` in AWS SDK for Python (Boto3).
Autopilot trains several model candidates with your target time-series, then selects an optimal forecasting model for a given objective metric. When your model candidates have been trained, you can find the best candidate metrics in the response to `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` at `[BestCandidate](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateProperties.html#sagemaker-Type-CandidateProperties-CandidateMetrics)`.
The following sections define the mandatory and optional input request parameters for the `CreateAutoMLJobV2` API used in time-series forecasting.
**Note**
Refer to the notebook [Time-Series Forecasting with Amazon SageMaker Autopilot](https://github.com/aws/amazon-sagemaker-examples/blob/main/autopilot/autopilot_time_series.ipynb) for a practical, hands-on time-series forecasting example. In this notebook, you use Amazon SageMaker Autopilot to train a time-series model and produce predictions using the trained model. The notebook provides instructions for retrieving a ready-made dataset of tabular historical data on Amazon S3.
## Prerequisites
Before using Autopilot to create a time-series forecasting experiment in SageMaker AI, make sure to:
+ Prepare your time-series dataset. Dataset preparation involves collecting relevant data from various sources, cleaning and filtering it to remove noise and inconsistencies, and organizing it into a structured format. See [Time-series datasets format and missing values filling methods](timeseries-forecasting-data-format.md) to learn more about time-series formats requirements in Autopilot. Optionally, you can supplement your dataset with the public holiday calendar of the country of your choice to capture associated patterns. For more information on holiday calendars, see [National holiday calendars](autopilot-timeseries-forecasting-holiday-calendars.md).
**Note**
We recommend providing at least 3-5 historical data points for each 1 future data point you want to predict. For example, to forecast 7 days ahead (horizon of 1 week) based on daily data, train your model on a minimum of 21-35 days of historical data. Make sure to provide enough data to capture seasonal and recurrent patterns.
+ Place your time-series data in an Amazon S3 bucket.
+ Grant full access to the Amazon S3 bucket containing your input data for the SageMaker AI execution role used to run your experiment. Once this is done, you can use the ARN of this execution role in Autopilot API requests.
+ For information on retrieving your SageMaker AI execution role, see [Get your execution role](sagemaker-roles.md#sagemaker-roles-get-execution-role).
+ For information on granting your SageMaker AI execution role permissions to access one or more specific buckets in Amazon S3, see * Add Additional Amazon S3 Permissions to a SageMaker AI Execution Role* in [Create execution role](sagemaker-roles.md#sagemaker-roles-create-execution-role).
## Required parameters
When calling `[CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html)` to create an Autopilot experiment for time-series forecasting, you must provide the following values:
+ An `[AutoMLJobName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_RequestSyntax)` to specify the name of your job. The name should be of type `string`, and should have a minimum length of 1 character and a maximum length of 32.
+ At least one `[AutoMLJobChannel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html)` in `[AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig)` in which you specify the name of the Amazon S3 bucket that contains your data. Optionally, you can specify the content (CSV or Parquet files) and compression (GZip) types.
+ An `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)` of type `[TimeSeriesForecastingJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesForecastingJobConfig.html)` to configure the settings of your time-series forecasting job. In particular, you must specify:
+ The **frequency** of predictions, which refers to the desired granularity (hourly, daily, monthly, etc) of your forecast.
Valid intervals are an integer followed by `Y` (Year), `M` (Month), `W` (Week), `D` (Day), `H` (Hour), and `min` (Minute). For example, `1D` indicates every day and `15min` indicates every 15 minutes. The value of a frequency must not overlap with the next larger frequency. For example, you must use a frequency of `1H` instead of `60min`.
The valid values for each frequency are the following:
+ Minute - 1-59
+ Hour - 1-23
+ Day - 1-6
+ Week - 1-4
+ Month - 1-11
+ Year - 1
+ The **horizon** of predictions in your forecast, which refers to the number of time-steps that the model predicts. The forecast horizon is also called the prediction length. The maximum forecast horizon is the lesser of 500 time-steps or 1/4 of the time-steps in the dataset.
+ A [TimeSeriesConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesConfig.html) in which you define the schema of your dataset to map the column headers to your forecast by specifying:
+ A `TargetAttributeName`: The column that contains historical data of the target field to forecast.
+ A `TimestampAttributeName`: The column that contains a point in time at which the target value of a given item is recorded.
+ A `ItemIdentifierAttributeName`: The column that contains the item identifiers for which you want to predict the target value.
The following is an example of those request parameters. In this example, you are setting up a daily forecast for the expected quantity or level of demand of specific items over a period of 20 days.
```
"AutoMLProblemTypeConfig": {
"ForecastFrequency": "D",
"ForecastHorizon": 20,
"TimeSeriesConfig": {
"TargetAttributeName": "demand",
"TimestampAttributeName": "timestamp",
"ItemIdentifierAttributeName": "item_id"
},
```
+ An `[OutputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html)` to specify the Amazon S3 output path to store the artifacts of your AutoML job.
+ A `[RoleArn](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-RoleArn)` to specify the ARN of the role used to access your data. You can use the ARN of the execution role to which you have granted access to your data.
All other parameters are optional. For example, you can set specific forecast quantiles, choose a filling method for missing values in the dataset, or define how to aggregate data that does not align with forecast frequency. To learn how to set those additional parameters, see [Optional parameters](#timeseries-forecasting-api-optional-params).
## Optional parameters
The following sections provide details of some optional parameters that you can pass to your time-series forecasting AutoML job.
### How to specify algorithms
By default, your Autopilot job trains a pre-defined list of algorithms on your dataset. However, you can provide a subset of the default selection of algorithms.
For time-series forecasting, you must choose `[TimeSeriesForecastingJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesForecastingJobConfig.html)` as the type of `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)`.
Then, you can specify an array of selected `AutoMLAlgorithms` in the `AlgorithmsConfig` attribute of [CandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateGenerationConfig.html).
The following is an example of an `AlgorithmsConfig` attribute listing exactly three algorithms ("cnn-qr", "prophet", "arima") in its `AutoMLAlgorithms` field.
```
{
"[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)": {
"[TimeSeriesForecastingJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesForecastingJobConfig.html)": {
"[CandidateGenerationConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateGenerationConfig.html)": {
"[AlgorithmsConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateGenerationConfig.html#sagemaker-Type-CandidateGenerationConfig-AlgorithmsConfig)":[
{"[AutoMLAlgorithms](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLAlgorithmConfig.html)":["cnn-qr", "prophet", "arima"]}
]
},
},
},
}
```
For the list of available algorithms for time-series forecasting, see [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLAlgorithmConfig.html#sagemaker-Type-AutoMLAlgorithmConfig-AutoMLAlgorithms](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLAlgorithmConfig.html#sagemaker-Type-AutoMLAlgorithmConfig-AutoMLAlgorithms). For details on each algorithm, see [Algorithms support for time-series forecasting](timeseries-forecasting-algorithms.md).
### How to specify custom quantiles
Autopilot trains 6 models candidates with your target time-series, then combines these models using a stacking ensemble method to create an optimal forecasting model for a given objective metric. Each Autopilot forecasting model generates a probabilistic forecast by producing forecasts at quantiles between P1 and P99. These quantiles are used to account for forecast uncertainty. By default, forecasts will be generated for the 0.1 (`p10`), 0.5 (`p50`), and 0.9 (`p90`). You can choose to specify your own quantiles.
In Autopilot, you can specify up to five forecast quantiles from 0.01 (`p1`) to 0.99 (`p99`), by increments of 0.01 or higher in the `ForecastQuantiles` attribute of [TimeSeriesForecastingJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesForecastingJobConfig.html).
In the following example, you are setting up a daily 10th, 25th, 50th, 75th, and 90th percentile forecast for the expected quantity or level of demand of specific items over a period of 20 days.
```
"AutoMLProblemTypeConfig": {
"ForecastFrequency": "D",
"ForecastHorizon": 20,
"ForecastQuantiles": ["p10", "p25", "p50", "p75", "p90"],
"TimeSeriesConfig": {
"TargetAttributeName": "demand",
"TimestampAttributeName": "timestamp",
"ItemIdentifierAttributeName": "item_id"
},
```
### How to aggregate data for different forecast frequencies
To create a forecast model (also referred to as the best model candidate from your experiment), you must specify a forecast frequency. The forecast frequency determines the frequency of predictions in your forecasts. For example, monthly sales forecasts. Autopilot best model can generate forecasts for data frequencies that are higher than the frequency at which your data is recorded.
During training, Autopilot aggregates any data that does not align with the forecast frequency you specify. For example, you might have some daily data but specify a weekly forecast frequency. Autopilot aligns the daily data based on the week that it belongs in. Autopilot then combines it into single record for each week.
During aggregation, the default transformation method is to sum the data. You can configure the aggregation when you create your AutoML job in the `Transformations` attribute of [TimeSeriesForecastingJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesForecastingJobConfig.html). The supported aggregation methods are `sum` (default), `avg`, `first`, `min`, `max`. Aggregation is only supported for the target column.
In the following example, you configure the aggregation to calculate the average of the individual promo forecasts to provide the final aggregated forecast values.
```
"Transformations": {
"Aggregation": {
"promo": "avg"
}
}
```
### How to handle missing values in your input datasets
Autopilot provides a number of filling methods to handle missing values in the target and other numeric columns of your time-series datasets. For information on the list of supported filling methods and their available filling logic, see [Handle missing values](timeseries-forecasting-data-format.md#timeseries-missing-values).
You configure your filling strategy in the `Transformations` attribute of [TimeSeriesForecastingJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesForecastingJobConfig.html) when creating your AutoML job.
To set a filling method, you need to provide a key-value pair:
+ The key is the name of the column for which you want to specify the filling method.
+ The value associated with the key is an object that defines the filling strategy for that column.
You can specify multiple filling methods for a single column.
To set a specific value for the filling method, you should set the fill parameter to the desired filling method value (for example `"backfill" : "value"`), and define the actual filling value in an additional parameter suffixed with "\$1value". For example, to set `backfill` to a value of `2`, you must include two parameters: `"backfill": "value"` and `"backfill_value":"2"`.
In the following example, you specify the filling strategy for the incomplete data column, "price" as follows: All missing values between the first data point of an item and the last are set to `0` after which all missing values are filled with the value `2` until the end date of the dataset.
```
"Transformations": {
"Filling": {
"price": {
"middlefill" : "zero",
"backfill" : "value",
"backfill_value": "2"
}
}
}
```
### How to specify an objective metric
Autopilot produces accuracy metrics to evaluate the model candidates and help you choose which to use to generate forecasts. When you run a time-series forecasting experiment, you can either choose AutoML to let Autopilot optimize the predictor for you, or you can manually choose an algorithm for your predictor.
By default, Autopilot uses the Average Weighted Quantile Loss. However, you can configure the objective metric when you create your AutoML job in the `MetricName` attribute of [AutoMLJobObjective](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobObjective.html).
For the list of available algorithms, see [Algorithms support for time-series forecasting](timeseries-forecasting-algorithms.md).
### How to incorporate national holiday information to your dataset
In Autopilot, you can incorporate a feature-engineered dataset of national holiday information to your time-series. Autopilot provide native support for the holiday calendars of over 250 countries. After you choose a country, Autopilot applies that country’s holiday calendar to every item in your dataset during training. This allows the model to identify patterns associated with specific holidays.
You can enable the holiday featurization when you create your AutoML job by passing an [HolidayConfigAttributes](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_HolidayConfigAttributes.html) object to the `HolidayConfig` attribute of [TimeSeriesForecastingJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesForecastingJobConfig.html). The `HolidayConfigAttributes` object contains the two letters `CountryCode` attribute that determines the country of the public national holiday calendar used to augment your time-series dataset.
Refer to [Country Codes](autopilot-timeseries-forecasting-holiday-calendars.md#holiday-country-codes) for the list of supported calendars and their corresponding country code.
### How to enable automatic deployment
Autopilot allows you to automatically deploy your forecast model to an endpoint. To enable automatic deployment for the best model candidate of an AutoML job, include a `[ModelDeployConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-ModelDeployConfig)` in the AutoML job request. This allows the deployment of the best model to a SageMaker AI endpoint. Below are the available configurations for customization.
+ To let Autopilotgenerate the endpoint name, set `[AutoGenerateEndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents)` to `True`.
+ To provide your own name for the endpoint, set `[AutoGenerateEndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents) to False and provide a name of your choice in [EndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents)`.
### How to configure AutoML to initiate a remote job on EMR Serverless for large datasets
You can configure your AutoML job V2 to automatically initiate a remote job on Amazon EMR Serverless when additional compute resources are needed to process large datasets. By seamlessly transitioning to EMR Serverless when required, the AutoML job can handle datasets that would otherwise exceed the initially provisioned resources, without any manual intervention from you. EMR Serverless is available for the tabular and time series problem types. We recommend setting up this option for time-series datasets larger than 30 GB.
To allow your AutoML job V2 to automatically transition to EMR Serverless for large dataset, you need to provide an `EmrServerlessComputeConfig` object, which includes an `ExecutionRoleARN` field, to the `AutoMLComputeConfig` of the AutoML job V2 input request.
The `ExecutionRoleARN` is the ARN of the IAM role granting the AutoML job V2 the necessary permissions to run EMR Serverless jobs.
This role should have the following trust relationship:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "emr-serverless.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
------
And grant the permissions to:
+ Create, list, and update EMR Serverless applications.
+ Start, list, get, or cancel job runs on an EMR Serverless application.
+ Tag EMR Serverless resources.
+ Pass an IAM role to the EMR Serverless service for execution.
By granting the `iam:PassRole` permission, the AutoML job V2 can temporarily assume the `EMRServerlessRuntimeRole-*` role and pass it to the EMR Serverless service. These are the IAM roles used by the EMR Serverless job execution environments to access other AWS services and resources needed during runtime, such as Amazon S3 for data access, CloudWatch for logging, access to the AWS Glue Data Catalog or other services based on your workload requirements.
See [Job runtime roles for Amazon EMR Serverless](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/security-iam-runtime-role.html) for details on this role permissions.
The IAM policy defined in the provided JSON document grants those permissions:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [{
"Sid": "EMRServerlessCreateApplicationOperation",
"Effect": "Allow",
"Action": "emr-serverless:CreateApplication",
"Resource": "arn:aws:emr-serverless:*:*:/*",
"Condition": {
"StringEquals": {
"aws:RequestTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessListApplicationOperation",
"Effect": "Allow",
"Action": "emr-serverless:ListApplications",
"Resource": "arn:aws:emr-serverless:*:*:/*",
"Condition": {
"StringEquals": {
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessApplicationOperations",
"Effect": "Allow",
"Action": [
"emr-serverless:UpdateApplication",
"emr-serverless:GetApplication"
],
"Resource": "arn:aws:emr-serverless:*:*:/applications/*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessStartJobRunOperation",
"Effect": "Allow",
"Action": "emr-serverless:StartJobRun",
"Resource": "arn:aws:emr-serverless:*:*:/applications/*",
"Condition": {
"StringEquals": {
"aws:RequestTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessListJobRunOperation",
"Effect": "Allow",
"Action": "emr-serverless:ListJobRuns",
"Resource": "arn:aws:emr-serverless:*:*:/applications/*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessJobRunOperations",
"Effect": "Allow",
"Action": [
"emr-serverless:GetJobRun",
"emr-serverless:CancelJobRun"
],
"Resource": "arn:aws:emr-serverless:*:*:/applications/*/jobruns/*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "EMRServerlessTagResourceOperation",
"Effect": "Allow",
"Action": "emr-serverless:TagResource",
"Resource": "arn:aws:emr-serverless:*:*:/*",
"Condition": {
"StringEquals": {
"aws:RequestTag/sagemaker:is-canvas-resource": "True",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
},
{
"Sid": "IAMPassOperationForEMRServerless",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/EMRServerlessRuntimeRole-*",
"Condition": {
"StringEquals": {
"iam:PassedToService": "emr-serverless.amazonaws.com",
"aws:ResourceAccount": "${aws:PrincipalAccount}"
}
}
}
]
}
```
------
# Time-series datasets format and missing values filling methods
Time-series data refers to a collection of observations or measurements recorded over regular intervals of time. In this type of data, each observation is associated with a specific timestamp or time period, creating a sequence of data points ordered chronologically.
The specific columns you include in your time-series dataset depend on the goals of your analysis and the data available to you. At a minimum, the time-series data is composed of a 3-column table where:
+ One column contains unique identifiers assigned to individual items to refer to their value at a specific moment.
+ Another column represents the point-in-time value or **target** to log the value of a given item at a specific moment. After the model is trained on those target values, this target column contains the values that the model predicts at a specified frequency within a defined horizon.
+ And a timestamp column is included to record the date and time when the value was measured.
+ Additional columns can contain other factors that may influence the forecast performance. For example, in a time-series dataset for retail where the target is the sales or revenue, you might include features that provide information about units sold, product ID, store location, customer count, inventory levels, as well as covariate indicators such as weather data or demographic information.
**Note**
You can add a feature-engineered dataset of national holiday information to your time-series. By including holidays in your time series model, you can capture the periodic patterns that holidays create. This helps your forecasts better reflect the underlying seasonality of your data. For information on the available calendars per country, see [National holiday calendars](autopilot-timeseries-forecasting-holiday-calendars.md)
## Datasets format for time-series forecasting
Autopilot supports numeric, categorical, text, and datetime data types. The data type of the target column must be numeric.
Autopilot supports time-series data formatted as CSV (default) files or as Parquet files.
+ **CSV** (comma-separated-values) is a row-based file format that stores data in human readable plaintext which a popular choice for data exchange as they are supported by a wide range of applications.
+ **Parquet** is a column-based file format where the data is stored and processed more efficiently than row-based file formats. This makes them a better option for big data problems.
For more information about the resource limits on time-series datasets for forecasting in Autopilot, see [Time-series forecasting resource limits for Autopilot](timeseries-forecasting-limits.md).
## Handle missing values
A common issue in time-series forecasting data is the presence of missing values. Your data might contain missing values for a number of reasons, including measurement failures, formatting problems, human errors, or a lack of information to record. For instance, if you are forecasting product demand for a retail store and an item is sold out or unavailable, there would be no sales data to record while that item is out of stock. If prevalent enough, missing values can significantly impact a model's accuracy.
Autopilot provides a number of filling methods to handle missing values, with distinct approaches for the target column and other additional columns. Filling is the process of adding standardized values to missing entries in your dataset.
Refer to [How to handle missing values in your input datasets](autopilot-create-experiment-timeseries-forecasting.md#timeseries-forecasting-fill-missing-values) to learn how to set the method for filling missing values in your time-series dataset.
Autopilot supports the following filling methods:
+ **Front filling:** Fills any missing values between the earliest recorded data point among all items and the starting point of each item (each item can start at a different time). This ensures that the data for each item is complete and spans from the earliest recorded data point to its respective starting point.
+ **Middle filling:** Fills any missing values between the start and end dates of the items in the dataset.
+ **Back filling:** Fills any missing values between the last data point of each item (each item can stop at a different time) and the last recorded data point among all items.
+ **Future filling:** Fills any missing values between the last recorded data point among all items and the end of the forecast horizon.
The following image provides a visual representation of the different filling methods.
![\[The different filling methods for time series forecasting in Amazon SageMaker Autopilot.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/autopilot/autopilot-forecast-filling-methods.png)
### Choose a filling logic
When choosing a filling logic, you should consider how the logic will be interpreted by your model. For instance, in a retail scenario, recording 0 sales of an available item is different from recording 0 sales of an unavailable item, as the latter does not imply a lack of customer interest in the item. Because of this, `0` filling in the target column of the time-series might cause the predictor to be under-biased in its predictions, while `NaN` filling might ignore actual occurrences of 0 available items being sold and cause the predictor to be over-biased.
### Filling logic
You can perform filling on the target column and other numeric columns in your datasets. Target columns have different filling guidelines and restrictions than the rest of the numeric columns.
Filling Guidelines
| Column type | Filling by default? | Supported filling methods | Default filling logic | Accepted filling logic |
| --- | --- | --- | --- | --- |
| Target column | Yes | Middle and back filling | 0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sagemaker/latest/dg/timeseries-forecasting-data-format.html) |
| Other numeric columns | No | Middle, back, and future filling | No default | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sagemaker/latest/dg/timeseries-forecasting-data-format.html) |
**Note**
For both the target and other numeric columns, `mean`, `median`, `min`, and `max` are calculated based on a rolling window of the 64 most recent data entries before the missing values.
# National holiday calendars
Autopilot supports a feature-engineered dataset of national holiday information that provides access to the holiday calendars of over 250 countries. Holiday calendar features are especially useful in the retail domain, where public holidays can significantly affect demand. The following section lists the country codes that you can use to access the holiday calendars of each supported country.
Refer to [How to incorporate national holiday information to your dataset](autopilot-create-experiment-timeseries-forecasting.md#timeseries-forecasting-add-holiday-calendar) to learn how to add a calendar to your dataset.
## Country Codes
Autopilot provides native support for the public holiday calendars of the following countries. Use the **Country Code** when specifying a country with the API.
| Country | Country Code |
| --- | --- |
| Afghanistan | AF |
| Åland Islands | AX |
| Albania | AL |
| Algeria | DZ |
| American Samoa | AS |
| Andorra | AD |
| Angola | AO |
| Anguilla | AI |
| Antartica | AQ |
| Antigua and Barbuda | AG |
| Argentina | AR |
| Armenia | AM |
| Aruba | AW |
| Australia | AU |
| Austria | AT |
| Azerbaijan | AZ |
| Bahamas | BS |
| Bahrain | BH |
| Bangladesh | BD |
| Barbados | BB |
| Belarus | BY |
| Belgium | BE |
| Belize | BZ |
| Benin | BJ |
| Bermuda | BM |
| Bhutan | BT |
| Bolivia | BO |
| Bosnia and Herzegovina | BA |
| Botswana | BW |
| Bouvet Island | BV |
| Brazil | BR |
| British Indian Ocean Territory | IO |
| British Virgin Islands | VG |
| Brunei Darussalam | BN |
| Bulgaria | BG |
| Burkina Faso | BF |
| Burundi | BI |
| Cambodia | KH |
| Cameroon | CM |
| Canada | CA |
| Cape Verde | CV |
| Caribbean Netherlands | BQ |
| Cayman Islands | KY |
| Central African Republic | CF |
| Chad | TD |
| Chile | CL |
| China | CN |
| Christmas Island | CX |
| Cocos (Keeling) Islands | CC |
| Colombia | CO |
| Comoros | KM |
| Cook Islands | CK |
| Costa Rica | CR |
| Croatia | HR |
| Cuba | CU |
| Curaçao | CW |
| Cyprus | CY |
| Czechia | CZ |
| Democratic Republic of the Congo | CD |
| Denmark | DK |
| Djibouti | DJ |
| Dominica | DM |
| Dominican Republic | DO |
| Ecuador | EC |
| Egypt | EG |
| El Salvador | SV |
| Equatorial Guinea | GQ |
| Eritrea | ER |
| Estonia | EE |
| Eswatini | SZ |
| Ethiopia | ET |
| Falkland Islands | FK |
| Faroe Islands | FO |
| Fiji | FJ |
| Finland | FI |
| France | FR |
| French Guiana | GF |
| French Polynesia | PF |
| French Southern Territories | TF |
| Gabon | GA |
| Gambia | GM |
| Georgia | GE |
| Germany | DE |
| Ghana | GH |
| Gibraltar | GI |
| Greece | GR |
| Greenland | GL |
| Grenada | GD |
| Guadeloupe | GP |
| Guam | GU |
| Guatemala | GT |
| Guernsey | GG |
| Guinea | GN |
| Guinea-Bissau | GW |
| Guyana | GY |
| Haiti | HT |
| Heard Island and McDonald Islands | HM |
| Honduras | HN |
| Hong Kong | HK |
| Hungary | HU |
| Iceland | IS |
| India | IN |
| Indonesia | ID |
| Iran | IR |
| Iraq | IQ |
| Ireland | IE |
| Isle of Man | IM |
| Israel | IL |
| Italy | IT |
| Ivory Coast | CI |
| Jamaica | JM |
| Japan | JP |
| Jersey | JE |
| Jordan | JO |
| Kazakhstan | KZ |
| Kenya | KE |
| Kiribati | KI |
| Kosovo | XK |
| Kuwait | KW |
| Kyrgyzstan | KG |
| Laos | LA |
| Latvia | LV |
| Lebanon | LB |
| Lesotho | LS |
| Liberia | LR |
| Libya | LY |
| Liechtenstein | LI |
| Lithuania | LT |
| Luxembourg | LU |
| Macao | MO |
| Madagascar | MG |
| Malawi | MW |
| Malaysia | MY |
| Maldives | MV |
| Mali | ML |
| Malta | MT |
| Marshall Islands | MH |
| Martinique | MQ |
| Mauritania | MR |
| Mauritius | MU |
| Mayotte | YT |
| Mexico | MX |
| Micronesia | FM |
| Moldova | MD |
| Monaco | MC |
| Mongolia | MN |
| Montenegro | ME |
| Montserrat | MS |
| Morocco | MA |
| Mozambique | MZ |
| Myanmar | MM |
| Namibia | NA |
| Nauru | NR |
| Nepal | NP |
| Netherlands | NL |
| New Caledonia | NC |
| New Zealand | NZ |
| Nicaragua | NI |
| Niger | NE |
| Nigeria | NG |
| Niue | NU |
| Norfolk Island | NF |
| North Korea | KP |
| North Macedonia | MK |
| Northern Mariana Islands | MP |
| Norway | NO |
| Oman | OM |
| Pakistan | PK |
| Palau | PW |
| Palestine | PS |
| Panama | PA |
| Papua New Guinea | PG |
| Paraguay | PY |
| Peru | PE |
| Philippines | PH |
| Pitcairn Islands | PN |
| Poland | PL |
| Portugal | PT |
| Puerto Rico | PR |
| Qatar | QA |
| Republic of the Congo | CG |
| Réunion | RE |
| Romania | RO |
| Russian Federation | RU |
| Rwanda | RW |
| Saint Barthélemy | BL |
| "Saint Helena, Ascension and Tristan da Cunha " | SH |
| Saint Kitts and Nevis | KN |
| Saint Lucia | LC |
| Saint Martin | MF |
| Saint Pierre and Miquelon | PM |
| Saint Vincent and the Grenadines | VC |
| Samoa | WS |
| San Marino | SM |
| Sao Tome and Principe | ST |
| Saudi Arabia | SA |
| Senegal | SN |
| Serbia | RS |
| Seychelles | SC |
| Sierra Leone | SL |
| Singapore | SG |
| Sint Maarten | SX |
| Slovakia | SK |
| Slovenia | SI |
| Solomon Islands | SB |
| Somalia | SO |
| South Africa | ZA |
| South Georgia and the South Sandwich Islands | GS |
| South Korea | KR |
| South Sudan | SS |
| Spain | ES |
| Sri Lanka | LK |
| Sudan | SD |
| Suriname | SR |
| Svalbard and Jan Mayen | SJ |
| Sweden | SE |
| Switzerland | CH |
| Syrian Arab Republic | SY |
| Taiwan | TW |
| Tajikistan | TJ |
| Tanzania | TZ |
| Thailand | TH |
| Timor-Leste | TL |
| Togo | TG |
| Tokelau | TK |
| Tonga | TO |
| Trinidad and Tobago | TT |
| Tunisia | TN |
| Turkey | TR |
| Turkmenistan | TM |
| Turks and Caicos Islands | TC |
| Tuvalu | TV |
| Uganda | UG |
| Ukraine | UA |
| United Arab Emirates | AE |
| United Kingdom | UK |
| United Nations | UN |
| United States | US |
| United States Minor Outlying Islands | UM |
| United States Virgin Islands | VI |
| Uruguay | UY |
| Uzbekistan | UZ |
| Vanuatu | VU |
| Vatican City | VA |
| Venezuela | VE |
| Vietnam | VN |
| Wallis and Futuna | WF |
| Western Sahara | EH |
| Yemen | YE |
| Zambia | ZM |
| Zimbabwe | ZW |
# Objective metrics
Autopilot produces accuracy metrics to evaluate the model candidates and help you choose which to use to generate forecasts. You can either let Autopilot optimize the predictor for you, or you can manually choose an algorithm for your predictor. By default, Autopilot uses the Average Weighted Quantile Loss.
The following list contains the names of the metrics that are currently available to measure the performance of models for time-series forecasting.
**`RMSE`**
Root mean squared error (RMSE) – Measures the square root of the squared difference between predicted and actual values, and is averaged over all values. It's an important metric to indicate the presence of large model errors and outliers. Values range from zero (0) to infinity, with smaller numbers indicating a better model fit to the data. RMSE is dependent on scale, and should not be used to compare datasets of different sizes.
**`wQL`**
Weighted Quantile Loss (wQL) – Assess the accuracy of the forecast by measuring the weighted absolute differences between predicted and actual P10, P50, and P90 quantiles with lower values indicating better performance.
**`Average wQL (default)`**
Average Weighted Quantile Loss (Average wQL) – Evaluates the forecast by averaging the accuracy at the P10, P50, and P90 quantiles. A lower value indicates a more accurate model.
**`MASE`**
Mean Absolute Scaled Error (MASE) – The mean absolute error of the forecast normalized by the mean absolute error of a simple baseline forecasting method. A lower value indicates a more accurate model, where MASE < 1 is estimated to be better than the baseline and MASE > 1 is estimated to be worse than the baseline.
**`MAPE`**
Mean Absolute Percent Error (MAPE) – The percentage error (percent difference of the mean forecasted value versus the actual value) averaged over all time points. A lower value indicates a more accurate model, where MAPE = 0 is a model with no errors.
**`WAPE`**
Weighted Absolute Percent Error (WAPE) – The sum of the absolute error normalized by the sum of the absolute target, which measure the overall deviation of forecasted values from observed values. A lower value indicates a more accurate model.
# Algorithms support for time-series forecasting
Autopilot trains the following six built-in algorithms with your target time-series. Then, using a stacking ensemble method, it combines these model candidates to create an optimal forecasting model for a given objective metric.
+ **Convolutional Neural Network - Quantile Regression (CNN-QR)** – CNN-QR is a proprietary machine learning algorithm for forecasting time-series using causal convolutional neural networks (CNNs). CNN-QR works best with large datasets containing hundreds of time-series.
+ **DeepAR\$1** – DeepAR\$1 is a proprietary machine learning algorithm for forecasting time-series using recurrent neural networks (RNNs). DeepAR\$1 works best with large datasets containing hundreds of feature time-series.
+ **Prophet** – [Prophet](https://facebook.github.io/prophet/) is a popular local Bayesian structural time series model based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality. The Autopilot Prophet algorithm uses the [Prophet class](https://facebook.github.io/prophet/docs/quick_start.html#python-ap) of the Python implementation of Prophet. It works best with time-series with strong seasonal effects and several seasons of historical data.
+ **Non-Parametric Time Series (NPTS)** – The NPTS proprietary algorithm is a scalable, probabilistic baseline forecaster. It predicts the future value distribution of a given time-series by sampling from past observations. NPTS is especially useful when working with sparse or intermittent time series.
+ **Autoregressive Integrated Moving Average (ARIMA)** – ARIMA is a commonly used statistical algorithm for time-series forecasting. The algorithm captures standard temporal structures (patterned organizations of time) in the input dataset. It is especially useful for simple datasets with under 100 time series.
+ **Exponential Smoothing (ETS)** – ETS is a commonly used statistical algorithm for time-series forecasting. The algorithm is especially useful for simple datasets with under 100 time series, and datasets with seasonality patterns. ETS computes a weighted average over all observations in the time series dataset as its prediction, with exponentially decreasing weights over time.
# Forecast a deployed Autopilot model
After training your models using the AutoML API, you can deploy them for real-time or batch-based forecasting.
The AutoML API trains several model candidates for your time-series data and selects an optimal forecasting model based on your target objective metric. Once your model candidates have been trained, you can find the best candidate in the response [DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html) at [BestCandidate](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html#sagemaker-Type-AutoMLCandidate-CandidateName).
To get predictions using this best performing model, you can either set up an endpoint to obtain forecasts interactively or use batch forecasting to make predictions on a batch of observations.
**Considerations**
+ When providing input data for forecasting, the schema of your data should remain the same as the one used to train your model, including the number of columns, column headers, and data types. You can forecast for existing or new item IDs within the same or different timestamp range to predict for a different time period.
+ Forecasting models predict for the forecast horizon points in the future specified in the input request at training, which is from the *target end date* to the *target end date \$1 forecast horizon*. To use the model for predicting specific dates, you should provide the data in the same format as the original input data, extending up to a specified *target end date*. In this scenario, the model will start predicting from the new target end date.
For example, if your dataset had monthly data from January to June with a Forecast horizon of 2, then the model would predict the target value for the next 2 months, which would be July and August. If in August, you want to predict for the next 2 months, this time your input data should be from January to August and the model will predict for the next 2 months (September, October).
+ When forecasting future data points, there is no set minimum for the amount of historical data to provide. Include enough data to capture seasonal and recurrent patterns in your time-series.
**Topics**
+ [Real-time forecasting](timeseries-forecasting-realtime.md)
+ [Batch forecasting](timeseries-forecasting-batch.md)
# Real-time forecasting
Real-time forecasting is useful when you need to generate predictions on-the-fly, such as for applications that require immediate responses or when forecasting for individual data points.
By deploying your AutoML model as a real-time endpoint, you can generate forecasts on-demand and minimize the latency between receiving new data and obtaining predictions. This makes real-time forecasting well-suited for applications that require immediate, personalized, or event-driven forecasting capabilities.
For real time forecasting, the dataset should be a subset of the input dataset. The real time endpoint has an input data size of approximately 6MB and a response timeout limitation of 60 seconds. We recommend bringing in one or few items at a time.
You can use SageMaker APIs to retrieve the best candidate of an AutoML job and then create a SageMaker AI endpoint using that candidate.
Alternatively, you can chose the automatic deployment option when creating your Autopilot experiment. For information on setting up the automatic deployment of models, see [How to enable automatic deployment](autopilot-create-experiment-timeseries-forecasting.md#timeseries-forecasting-auto-model-deployment).
**To create a SageMaker AI endpoint using your best model candidate:**
1.
**Retrieve the details of the AutoML job.**
The following AWS CLI command example uses the [DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html) API to obtain details of the AutoML job, including the information about the best model candidate.
```
aws sagemaker describe-auto-ml-job-v2 --auto-ml-job-name job-name --region region
```
1.
**Extract the container definition from [InferenceContainers](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html#sagemaker-Type-AutoMLCandidate-InferenceContainers) for the best model candidate.**
A container definition is the containerized environment used to host the trained SageMaker AI model for making predictions.
```
BEST_CANDIDATE=$(aws sagemaker describe-auto-ml-job-v2 \
--auto-ml-job-name job-name
--region region \
--query 'BestCandidate.InferenceContainers[0]' \
--output json
```
This command extracts the container definition for the best model candidate and stores it in the `BEST_CANDIDATE` variable.
1.
**Create a SageMaker AI model using the best candidate container definition.**
Use the container definitions from the previous steps to create a SageMaker AI model by using the [CreateModel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) API.
```
aws sagemaker create-model \
--model-name 'your-candidate-name>' \
--primary-container "$BEST_CANDIDATE"
--execution-role-arn 'execution-role-arn>' \
--region 'region>
```
The `--execution-role-arn` parameter specifies the IAM role that SageMaker AI assumes when using the model for inference. For details on the permissions required for this role, see [CreateModel API: Execution Role Permissions](https://docs.aws.amazon.com/).
1.
**Create a SageMaker AI endpoint configuration using the model.**
The following AWS CLI command uses the [CreateEndpointConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html) API to create an endpoint configuration.
```
aws sagemaker create-endpoint-config \
--production-variants file://production-variants.json \
--region 'region'
```
Where the `production-variants.json` file contains the model configuration, including the model name and instance type.
**Note**
We recommend using [m5.12xlarge](https://aws.amazon.com/ec2/instance-types/m5/) instances for real-time forecasting.
```
[
{
"VariantName": "variant-name",
"ModelName": "model-name",
"InitialInstanceCount": 1,
"InstanceType": "m5.12xlarge"
}
]
}
```
1.
**Create the SageMaker AI endpoint using the endpoint configuration.**
The following AWS CLI example uses the [CreateEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpoint.html) API to create the endpoint.
```
aws sagemaker create-endpoint \
--endpoint-name 'endpoint-name>' \
--endpoint-config-name 'endpoint-config-name' \
--region 'region'
```
Check the progress of your real-time inference endpoint deployment by using the [DescribeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpoint.html) API. See the following AWS CLI command as an example.
```
aws sagemaker describe-endpoint \
--endpoint-name 'endpoint-name' \
--region 'region'
```
After the `EndpointStatus` changes to `InService`, the endpoint is ready to use for real-time inference.
1.
**Invoke the SageMaker AI endpoint to make predictions.**
```
aws sagemaker invoke-endpoint \
--endpoint-name 'endpoint-name' \
--region 'region' \
--body file://input-data-in-bytes.json \
--content-type 'application/json' outfile
```
Where the `input-data-in-bytes.json` file contains the input data for the prediction.
# Batch forecasting
Batch forecasting, also known as offline inferencing, generates model predictions on a batch of observations. Batch inference is a good option for large datasets or if you don't need an immediate response to a model prediction request.
By contrast, online inference (real-time inferencing) generates predictions in real time.
You can use SageMaker APIs to retrieve the best candidate of an AutoML job and then submit a batch of input data for inference using that candidate.
1.
**Retrieve the details of the AutoML job.**
The following AWS CLI command example uses the [DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html) API to obtain details of the AutoML job, including the information about the best model candidate.
```
aws sagemaker describe-auto-ml-job-v2 --auto-ml-job-name job-name --region region
```
1.
**Extract the container definition from [InferenceContainers](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html#sagemaker-Type-AutoMLCandidate-InferenceContainers) for the best model candidate.**
A container definition is the containerized environment used to host the trained SageMaker AI model for making predictions.
```
BEST_CANDIDATE=$(aws sagemaker describe-auto-ml-job-v2 \
--auto-ml-job-name job-name
--region region \
--query 'BestCandidate.InferenceContainers[0]' \
--output json
```
This command extracts the container definition for the best model candidate and stores it in the `BEST_CANDIDATE` variable.
1.
**Create a SageMaker AI model using the best candidate container definition.**
Use the container definitions from the previous steps to create a SageMaker AI model by using the [CreateModel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) API.
```
aws sagemaker create-model \
--model-name 'model-name' \
--primary-container "$BEST_CANDIDATE"
--execution-role-arn 'execution-role-arn>' \
--region 'region>
```
The `--execution-role-arn` parameter specifies the IAM role that SageMaker AI assumes when using the model for inference. For details on the permissions required for this role, see [CreateModel API: Execution Role Permissions](https://docs.aws.amazon.com/).
1.
**Create a batch transform job.**
The following example creates a transform job using the [CreateTransformJob](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/create-transform-job.html) API.
```
aws sagemaker create-transform-job \
--transform-job-name 'transform-job-name' \
--model-name 'model-name'\
--transform-input file://transform-input.json \
--transform-output file://transform-output.json \
--transform-resources file://transform-resources.json \
--region 'region'
```
The input, output, and resource details are defined in separate JSON files:
+ `transform-input.json`:
```
{
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "s3://my-input-data-bucket/path/to/input/data"
}
},
"ContentType": "text/csv",
"SplitType": "None"
}
```
+ `transform-output.json`:
```
{
"S3OutputPath": "s3://my-output-bucket/path/to/output",
"AssembleWith": "Line"
}
```
+ `transform-resources.json`:
**Note**
We recommend using [m5.12xlarge](https://aws.amazon.com/ec2/instance-types/m5/) instances for general-purpose workloads and `m5.24xlarge` instances for big data forecasting tasks.
```
{
"InstanceType": "instance-type",
"InstanceCount": 1
}
```
1.
**Monitor the progress of your transform job using the [DescribeTransformJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeTransformJob.html) API.**
See the following AWS CLI command as an example.
```
aws sagemaker describe-transform-job \
--transform-job-name 'transform-job-name' \
--region region
```
1.
**Retrieve the batch transform output.**
After the job is finished, the predicted result is available in the `S3OutputPath`.
The output file name has the following format: `input_data_file_name.out`. As an example, if your input file is `text_x.csv`, the output name will be `text_x.csv.out`.
```
aws s3 ls s3://my-output-bucket/path/to/output/
```
The following code examples illustrate the use of the AWS SDK for Python (boto3) and the AWS CLI for batch forecasting.
------
#### [ AWS SDK for Python (boto3) ]
The following example uses **AWS SDK for Python (boto3)** to make predictions in batches.
```
import sagemaker
import boto3
session = sagemaker.session.Session()
sm_client = boto3.client('sagemaker', region_name='us-west-2')
role = 'arn:aws:iam::1234567890:role/sagemaker-execution-role'
output_path = 's3://test-auto-ml-job/output'
input_data = 's3://test-auto-ml-job/test_X.csv'
best_candidate = sm_client.describe_auto_ml_job_v2(AutoMLJobName=job_name)['BestCandidate']
best_candidate_containers = best_candidate['InferenceContainers']
best_candidate_name = best_candidate['CandidateName']
# create model
reponse = sm_client.create_model(
ModelName = best_candidate_name,
ExecutionRoleArn = role,
Containers = best_candidate_containers
)
# Lauch Transform Job
response = sm_client.create_transform_job(
TransformJobName=f'{best_candidate_name}-transform-job',
ModelName=model_name,
TransformInput={
'DataSource': {
'S3DataSource': {
'S3DataType': 'S3Prefix',
'S3Uri': input_data
}
},
'ContentType': "text/csv",
'SplitType': 'None'
},
TransformOutput={
'S3OutputPath': output_path,
'AssembleWith': 'Line',
},
TransformResources={
'InstanceType': 'ml.m5.2xlarge',
'InstanceCount': 1,
},
)
```
The batch inference job returns a response in the following format.
```
{'TransformJobArn': 'arn:aws:sagemaker:us-west-2:1234567890:transform-job/test-transform-job',
'ResponseMetadata': {'RequestId': '659f97fc-28c4-440b-b957-a49733f7c2f2',
'HTTPStatusCode': 200,
'HTTPHeaders': {'x-amzn-requestid': '659f97fc-28c4-440b-b957-a49733f7c2f2',
'content-type': 'application/x-amz-json-1.1',
'content-length': '96',
'date': 'Thu, 11 Aug 2022 22:23:49 GMT'},
'RetryAttempts': 0}}
```
------
#### [ AWS Command Line Interface (AWS CLI) ]
1. **Obtain the best candidate container definitions**.
```
aws sagemaker describe-auto-ml-job-v2 --auto-ml-job-name 'test-automl-job' --region us-west-2
```
1. **Create the model**.
```
aws sagemaker create-model --model-name 'test-sagemaker-model'
--containers '[{
"Image": "348316444620.dkr.ecr.us-west-2.amazonaws.com/sagemaker-sklearn-automl:2.5-1-cpu-py3",
"ModelDataUrl": "s3://amzn-s3-demo-bucket/out/test-job1/data-processor-models/test-job1-dpp0-1-e569ff7ad77f4e55a7e549a/output/model.tar.gz",
"Environment": {
"AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF": "1",
"AUTOML_TRANSFORM_MODE": "feature-transform",
"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "application/x-recordio-protobuf",
"SAGEMAKER_PROGRAM": "sagemaker_serve",
"SAGEMAKER_SUBMIT_DIRECTORY": "/opt/ml/model/code"
}
}, {
"Image": "348316444620.dkr.ecr.us-west-2.amazonaws.com/sagemaker-xgboost:1.3-1-cpu-py3",
"ModelDataUrl": "s3://amzn-s3-demo-bucket/out/test-job1/tuning/flicdf10v2-dpp0-xgb/test-job1E9-244-7490a1c0/output/model.tar.gz",
"Environment": {
"MAX_CONTENT_LENGTH": "20971520",
"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv",
"SAGEMAKER_INFERENCE_OUTPUT": "predicted_label",
"SAGEMAKER_INFERENCE_SUPPORTED": "predicted_label,probability,probabilities"
}
}, {
"Image": "348316444620.dkr.ecr.us-west-2.amazonaws.com/sagemaker-sklearn-automl:2.5-1-cpu-py3",
"ModelDataUrl": "s3://amzn-s3-demo-bucket/out/test-job1/data-processor-models/test-job1-dpp0-1-e569ff7ad77f4e55a7e549a/output/model.tar.gz",
"Environment": {
"AUTOML_TRANSFORM_MODE": "inverse-label-transform",
"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv",
"SAGEMAKER_INFERENCE_INPUT": "predicted_label",
"SAGEMAKER_INFERENCE_OUTPUT": "predicted_label",
"SAGEMAKER_INFERENCE_SUPPORTED": "predicted_label,probability,labels,probabilities",
"SAGEMAKER_PROGRAM": "sagemaker_serve",
"SAGEMAKER_SUBMIT_DIRECTORY": "/opt/ml/model/code"
}
}]' \
--execution-role-arn 'arn:aws:iam::1234567890:role/sagemaker-execution-role' \
--region 'us-west-2'
```
1. **Create a transform job**.
```
aws sagemaker create-transform-job --transform-job-name 'test-tranform-job'\
--model-name 'test-sagemaker-model'\
--transform-input '{
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "s3://amzn-s3-demo-bucket/data.csv"
}
},
"ContentType": "text/csv",
"SplitType": "None"
}'\
--transform-output '{
"S3OutputPath": "s3://amzn-s3-demo-bucket/output/",
"AssembleWith": "Line"
}'\
--transform-resources '{
"InstanceType": "ml.m5.2xlarge",
"InstanceCount": 1
}'\
--region 'us-west-2'
```
1. **Check the progress of the transform job**.
```
aws sagemaker describe-transform-job --transform-job-name 'test-tranform-job' --region us-west-2
```
The following is the response from the transform job.
```
{
"TransformJobName": "test-tranform-job",
"TransformJobArn": "arn:aws:sagemaker:us-west-2:1234567890:transform-job/test-tranform-job",
"TransformJobStatus": "InProgress",
"ModelName": "test-model",
"TransformInput": {
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "s3://amzn-s3-demo-bucket/data.csv"
}
},
"ContentType": "text/csv",
"CompressionType": "None",
"SplitType": "None"
},
"TransformOutput": {
"S3OutputPath": "s3://amzn-s3-demo-bucket/output/",
"AssembleWith": "Line",
"KmsKeyId": ""
},
"TransformResources": {
"InstanceType": "ml.m5.2xlarge",
"InstanceCount": 1
},
"CreationTime": 1662495635.679,
"TransformStartTime": 1662495847.496,
"DataProcessing": {
"InputFilter": "$",
"OutputFilter": "$",
"JoinSource": "None"
}
}
```
After the `TransformJobStatus` changes to `Completed`, you can check the inference result in the `S3OutputPath`.
------
# Amazon SageMaker Autopilot data exploration notebook
Amazon SageMaker Autopilot cleans and pre-processes your dataset automatically. To help users understand their data, uncover patterns, relationships, and anomalies about the time-series, Amazon SageMaker Autopilot generates a **data exploration** static report in the form of a notebook for users to reference.
The data exploration notebook is generated for every Autopilot job. The report is stored in an Amazon S3 bucket and can be accessed from the job output path.
You can find the Amazon S3 prefix to the data exploration notebook in the response to `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` at `[AutoMLJobArtifacts.DataExplorationNotebookLocation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html#sagemaker-DescribeAutoMLJobV2-response-AutoMLJobArtifacts)`.
# Reports generated by Amazon SageMaker Autopilot
In addition to the data exploration notebook, Autopilot generates various reports for the best model candidate of each experiment.
+ An explainability report provides insights into how the model makes forecasts.
+ A performance report provides a quantitative assessment of the model's forecasting capabilities.
+ A backtest results report is generated after testing the model's performance on historical data.
## Explainability report
Autopilot explainability report helps you better understand how the attributes in your datasets impact forecasts for specific time-series (item and dimension combinations) and time points. Autopilot uses a metric called *Impact scores* to quantify the relative impact of each attribute and determine whether they increase or decrease forecast values.
For example, consider a forecasting scenario where the target is `sales` and there are two related attributes: `price` and `color`. Autopilot may find that the item’s color has a high impact on sales for certain items, but a negligible effect for other items. It may also find that a promotion in the summer has a high impact on sales, but a promotion in the winter has little effect.
The explainability report is generated only when:
+ The time series dataset includes additional feature columns or is associated with a holiday calendar.
+ The base models CNN-QR and DeepAR\$1 are included in the final ensemble.
### Interpret Impact scores
Impact scores measure the relative impact attributes have on forecast values. For example, if the `price` attribute has an impact score that is twice as large as the `store location` attribute, you can conclude that the price of an item has twice the impact on forecast values than the store location.
Impact scores also provide information on whether attributes increase or decrease forecast values.
The Impact scores range from -1 to 1, where the sign denotes the direction of the impact. A score of 0 indicates no impact, while scores close to 1 or -1 indicate a significant impact.
It is important to note that Impact scores measure the relative impact of attributes, not the absolute impact. Therefore, Impact scores cannot be used to determine whether particular attributes improve model accuracy. If an attribute has a low Impact score, that does not necessarily mean that it has a low impact on forecast values; it means that it has a lower impact on forecast values than other attributes used by the predictor.
### Find the explainability report
You can find the Amazon S3 prefix to the explainability artifacts generated for the best candidate in the response to `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` at `[BestCandidate.CandidateProperties.CandidateArtifactLocations.Explainability](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateArtifactLocations.html#sagemaker-Type-CandidateArtifactLocations-Explainability)`.
## Model performance report
Autopilot model quality report (also referred to as performance report) provides insights and quality information for the best model candidate (best predictor) generated by an AutoML job. This includes information about the job details, objective function, and accuracy metrics (`wQL`, `MAPE`, `WAPE`, `RMSE`, `MASE`).
You can find the Amazon S3 prefix to the model quality report artifacts generated for the best candidate in the response to `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` at `[BestCandidate.CandidateProperties.CandidateArtifactLocations.ModelInsights](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateArtifactLocations.html#sagemaker-Type-CandidateArtifactLocations-ModelInsights)`.
## Backtests results report
Backtests results provide insights into the performance of a time-series forecasting model by evaluating its predictive accuracy and reliability. It helps analysts and data scientists assess its performance on historical data and assists in understanding its potential performance on future, unseen data.
Autopilot uses backtesting to tune parameters and produce accuracy metrics. During backtesting, Autopilot automatically splits your time-series data into two sets, a training set and a testing set. The training set is used to train a model which is then used to generate forecasts for data points in the testing set. Autopilot uses this testing dataset to evaluate the model's accuracy by comparing forecasted values with observed values in the testing set.
You can find the Amazon S3 prefix to the model quality report artifacts generated for the best candidate in the response to `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` at `[BestCandidate.CandidateProperties.CandidateArtifactLocations.BacktestResults](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateArtifactLocations.html#sagemaker-Type-CandidateArtifactLocations-BacktestResults)`.
# Time-series forecasting resource limits for Autopilot
The following table lists the resource limits for time-series forecasting jobs in Amazon SageMaker Autopilot and whether or not you can adjust each limit.
| **Resource limits** | **Default limit** | **Adjustable** |
| --- | --- | --- |
| Size of input dataset | 30 GB | Yes |
| Size of a single Parquet file | 2 GB | No |
| Maximum number of rows in a dataset | 3 billion | Yes |
| Maximum number of grouping columns | 5 | No |
| Maximum number of numerical features | 13 | No |
| Maximum number of categorical features | 10 | No |
| Maximum number of time-series (unique combinations of item and grouping columns) per dataset | 5,000,000 | Yes |
| Maximum Forecast horizon | 500 | Yes |
# Create an AutoML job to fine-tune text generation models using the API
Large language models (LLMs) excel in multiple generative tasks, including text generation, summarization, completion, question answering, and more. Their performance can be attributed to their significant size and extensive training on diverse datasets and various tasks. However, specific domains, such as healthcare and financial services, may require customized fine-tuning to adapt to unique data and use cases. By tailoring their training to their particular domain, LLMs can improve their performance and provide more accurate outputs for targeted applications.
Autopilot offers the capability to fine-tune a selection of pre-trained generative text models. In particular, Autopilot supports the **instruction-based fine tuning** of a selection of general-purpose large language models (LLMs) powered by JumpStart.
**Note**
The text generation models that support fine-tuning in Autopilot are currently accessible exclusively in Regions supported by SageMaker Canvas. See the documentation of SageMaker Canvas for the [full list of its supported Regions](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas.html).
Fine-tuning a pre-trained model requires a specific dataset of clear instructions that guide the model on how to generate output or behave for that task. The model learns from the dataset, adjusting its parameters to conform to the provided instructions. Instruction-based fine-tuning involves using labeled examples formatted as prompt-response pairs and phrased as instructions. For more information about fine-tuning, see [Fine-tune a foundation model](https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-fine-tuning.html).
The following guidelines outline the process of creating an Amazon SageMaker Autopilot job as a pilot experiment to fine-tune text generation LLMs using the SageMaker [API Reference](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-reference.html).
**Note**
Tasks such as text and image classification, time-series forecasting, and fine-tuning of large language models are exclusively available through the version 2 of the [AutoML REST API](autopilot-reference.md). If your language of choice is Python, you can refer to [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_auto_ml_job_v2.html) or the [AutoMLV2 object](https://sagemaker.readthedocs.io/en/stable/api/training/automlv2.html#sagemaker.automl.automlv2.AutoMLV2) of the Amazon SageMaker Python SDK directly.
Users who prefer the convenience of a user interface can use [Amazon SageMaker Canvas](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-getting-started.html) to access pre-trained models and generative AI foundation models, or create custom models tailored for specific text, image classification, forecasting needs, or generative AI.
To create an Autopilot experiment programmatically for fine-tuning an LLM, you can call the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html) API in any language supported by Amazon SageMaker Autopilot or the AWS CLI.
For information about how this API action translates into a function in the language of your choice, see the [ See Also](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_SeeAlso) section of `CreateAutoMLJobV2` and choose an SDK. As an example, for Python users, see the full request syntax of `[create\$1auto\$1ml\$1job\$1v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_auto_ml_job_v2)` in AWS SDK for Python (Boto3).
**Note**
Autopilot fine-tunes large language models without requiring multiple candidates to be trained and evaluated. Instead, using your dataset, Autopilot directly fine-tunes your target model to enhance a default objective metric, the cross-entropy loss. Fine-tuning language models in Autopilot does not require setting the `AutoMLJobObjective` field.
Once your LLM is fine-tuned, you can evaluate its performance by accessing various ROUGE scores through the `[BestCandidate](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateProperties.html#sagemaker-Type-CandidateProperties-CandidateMetrics)` when making a `[DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)` API call. The model also provides information about its training and validation loss as well as perplexity. For a comprehensive list of metrics for evaluating the quality of the text generated by the fine-tuned models, see [Metrics for fine-tuning large language models in Autopilot](autopilot-llms-finetuning-metrics.md).
## Prerequisites
Before using Autopilot to create a fine-tuning experiment in SageMaker AI, make sure to take the following steps:
+ (Optional) Choose the pre-trained model you want to fine-tune.
For the list of pre-trained models available for fine-tuning in Amazon SageMaker Autopilot, see [Supported large language models for fine-tuning](autopilot-llms-finetuning-models.md). The selection of a model is not mandatory; if no model is specified, Autopilot automatically defaults to the model *Falcon7BInstruct*.
+ Create a dataset of instructions. See [Dataset file types and input data format](autopilot-llms-finetuning-data-format.md) to learn about the format requirements for your instruction-based dataset.
+ Place your dataset in an Amazon S3 bucket.
+ Grant full access to the Amazon S3 bucket containing your input data for the SageMaker AI execution role used to run your experiment.
+ For information on retrieving your SageMaker AI execution role, see [Get your execution role](sagemaker-roles.md#sagemaker-roles-get-execution-role).
+ For information on granting your SageMaker AI execution role permissions to access one or more specific buckets in Amazon S3, see * Add Additional Amazon S3 Permissions to a SageMaker AI Execution Role* in [Create execution role](sagemaker-roles.md#sagemaker-roles-create-execution-role).
+ Additionally, you should provide your execution role with the necessary permissions to access the default storage Amazon S3 bucket used by JumpStart. This access is required for storing and retrieving pre-trained model artifacts in JumpStart. To grant access to this Amazon S3 bucket, you must create a new inline custom policy on your execution role.
Here's an example policy that you can use in your JSON editor when configuring AutoML fine-tuning jobs in `us-west-2`:
*JumpStart's bucket names follow a predetermined pattern that depends on the AWS Regions. You must adjust the name of the bucket accordingly.*
```
{
"Sid": "Statement1",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::jumpstart-cache-prod-us-west-2",
"arn:aws:s3:::jumpstart-cache-prod-us-west-2/*"
]
}
```
Once this is done, you can use the ARN of this execution role in Autopilot API requests.
## Required parameters
When calling `[CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html)` to create an Autopilot experiment for LLM fine-tuning, you must provide the following values:
+ An `[AutoMLJobName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#API_CreateAutoMLJobV2_RequestSyntax)` to specify the name of your job. The name should be of type `string`, and should have a minimum length of 1 character and a maximum length of 32.
+ At least one `[AutoMLJobChannel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html)` of the `training` type within the `[AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig)`. This channel specifies the name of the Amazon S3 bucket where your fine-tuning dataset is located. You have the option to define a `validation` channel. If no validation channel is provided, and a `ValidationFraction` is configured in the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html), this fraction is utilized to randomly divide the training dataset into training and validation sets. Additionally, you can specify the type of content (CSV or Parquet files) for the dataset.
+ An `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)` of type `[TextGenerationJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TextGenerationJobConfig.html)` to configure the settings of your training job.
In particular, you can specify the name of the base model to fine-tune in the `BaseModelName` field. For the list of pre-trained models available for fine-tuning in Amazon SageMaker Autopilot, see [Supported large language models for fine-tuning](autopilot-llms-finetuning-models.md).
+ An `[OutputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html)` to specify the Amazon S3 output path to store the artifacts of your AutoML job.
+ A `[RoleArn](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html#sagemaker-CreateAutoMLJob-request-RoleArn)` to specify the ARN of the role used to access your data.
The following is an example of the full request format used when making an API call to `CreateAutoMLJobV2` for fine-tuning a (`Falcon7BInstruct`) model.
```
{
"AutoMLJobName": "",
"AutoMLJobInputDataConfig": [
{
"ChannelType": "training",
"CompressionType": "None",
"ContentType": "text/csv",
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "s3:///.csv"
}
}
}
],
"OutputDataConfig": {
"S3OutputPath": "s3:///output",
"KmsKeyId": "arn:aws:kms:::key/"
},
"RoleArn":"arn:aws:iam:::role/",
"AutoMLProblemTypeConfig": {
"TextGenerationJobConfig": {
"BaseModelName": "Falcon7BInstruct"
}
}
}
```
All other parameters are optional.
## Optional parameters
The following sections provide details of some optional parameters that you can pass to your fine-tuning AutoML job.
### How to specify the training and validation datasets of an AutoML job
You can provide your own validation dataset and custom data split ratio, or let Autopilot split the dataset automatically.
Each [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html) object (see the required parameter [AutoMLJobInputDataConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLJobInputDataConfig)) has a `ChannelType`, which can be set to either `training` or `validation` values that specify how the data is to be used when building a machine learning model.
At least one data source must be provided and a maximum of two data sources is allowed: one for training data and one for validation data. How you split the data into training and validation datasets depends on whether you have one or two data sources.
+ If you only have **one data source**, the `ChannelType` is set to `training` by default and must have this value.
+ If the `ValidationFraction` value in [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html) is not set, 0.2 (20%) of the data from this source is used for validation by default.
+ If the `ValidationFraction` is set to a value between 0 and 1, the dataset is split based on the value specified, where the value specifies the fraction of the dataset used for validation.
+ If you have **two data sources**, the `ChannelType` of one of the `AutoMLJobChannel` objects must be set to `training`, the default value. The `ChannelType` of the other data source must be set to `validation`. The two data sources must have the same format, either CSV or Parquet, and the same schema. You must not set the value for the `ValidationFraction` in this case because all of the data from each source is used for either training or validation. Setting this value causes an error.
### How to enable automatic deployment
With Autopilot, you can automatically deploy your fine-tuned model to an endpoint. To enable automatic deployment for your fine-tuned model, include a `[ModelDeployConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-ModelDeployConfig)` in the AutoML job request. This allows the deployment of your fine-tuned model to a SageMaker AI endpoint. Below are the available configurations for customization.
+ To let Autopilot generate the endpoint name, set `[AutoGenerateEndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents)` to `True`.
+ To provide your own name for the endpoint, set `[AutoGenerateEndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents) to False and provide a name of your choice in [EndpointName](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html#API_ModelDeployConfig_Contents)`.
### How to set the EULA acceptance when fine-tuning a model using the AutoML API
For models requiring the acceptance of an end-user license agreement before fine-tuning, you can accept the EULA by setting the `AcceptEula` attribute of the `[ModelAccessConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelAccessConfig.html)` to `True` in `[TextGenerationJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TextGenerationJobConfig.html)` when configuring your `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)`.
### How to set hyperparameters to optimize the learning process of a model
You can optimize the learning process of your text generation model by setting hyperparameter values in the `TextGenerationHyperParameters` attribute of `[TextGenerationJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TextGenerationJobConfig.html)` when configuring your `[AutoMLProblemTypeConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html#sagemaker-CreateAutoMLJobV2-request-AutoMLProblemTypeConfig)`.
Autopilot allows for the setting of four common hyperparameters across all models.
+ `epochCount`: Its value should be a string containing an integer value within the range of `1` to `10`.
+ `batchSize`: Its value should be a string containing an integer value within the range of `1` to `64`.
+ `learningRate`: Its value should be a string containing a floating-point value within the range of `0` to `1`.
+ `learningRateWarmupSteps`: Its value should be a string containing an integer value within the range of `0` to `250`.
For more details on each hyperparameter, see [Hyperparameters for optimizing the learning process of your text generation models](autopilot-llms-finetuning-hyperparameters.md).
The following JSON example shows a `TextGenerationHyperParameters` field passed to the TextGenerationJobConfig where all four hyperparameters are configured.
```
"AutoMLProblemTypeConfig": {
"TextGenerationJobConfig": {
"BaseModelName": "Falcon7B",
"TextGenerationHyperParameters": {"epochCount":"5", "learningRate":"0.000001", "batchSize": "32", "learningRateWarmupSteps": "10"}
}
}
```
# Supported large language models for fine-tuning
Using Autopilot API, users can fine-tune large language models (LLMs) that are powered by Amazon SageMaker JumpStart.
**Note**
For fine-tuning models that require the acceptance of an end-user license agreement, you must explicitly declare EULA acceptance when creating your AutoML job. Note that after fine-tuning a pretrained model, the weights of the original model are changed, so you do not need to later accept a EULA when deploying the fine-tuned model.
For information on how to accept the EULA when creating a fine-tuning job using the AutoML API, see [How to set the EULA acceptance when fine-tuning a model using the AutoML API](autopilot-create-experiment-finetune-llms.md#autopilot-llms-finetuning-set-eula).
You can find the full details of each model by searching for your **JumpStart Model ID** in the following [model table](https://sagemaker.readthedocs.io/en/stable/doc_utils/pretrainedmodels.html#built-in-algorithms-with-pre-trained-model-table), and then following the link in the **Source** column. These details might include the languages supported by the model, biases it may exhibit, the datasets employed for fine-tuning, and more.
The following table lists the supported JumpStart models that you can fine-tune with an AutoML job.
| JumpStart Model ID | `BaseModelName` in API request | Description |
| --- | --- | --- |
| huggingface-textgeneration-dolly-v2-3b-bf16 | Dolly3B | Dolly 3B is a 2.8 billion parameter instruction-following large language model based on [pythia-2.8b](https://huggingface.co/EleutherAI/pythia-2.8b#pythia-28b). It is trained on the instruction/response fine tuning dataset [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) and can perform tasks including brainstorming, classification, questions and answers, text generation, information extraction, and summarization. |
| huggingface-textgeneration-dolly-v2-7b-bf16 | Dolly7B | Dolly 7B is a 6.9 billion parameter instruction-following large language model based on [pythia-6.9b](https://huggingface.co/EleutherAI/pythia-6.9b). It is trained on the instruction/response fine tuning dataset [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) and can perform tasks including brainstorming, classification, questions and answers, text generation, information extraction, and summarization. |
| huggingface-textgeneration-dolly-v2-12b-bf16 | Dolly12B | Dolly 12B is a 12 billion parameter instruction-following large language model based on [pythia-12b](https://huggingface.co/EleutherAI/pythia-12b). It is trained on the instruction/response fine tuning dataset [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) and can perform tasks including brainstorming, classification, questions and answers, text generation, information extraction, and summarization. |
| huggingface-llm-falcon-7b-bf16 | Falcon7B | Falcon 7B is a 7 billion parameter causal large language model trained on 1,500 billion tokens enhanced with curated corpora. Falcon-7B is trained on English and French data only, and does not generalize appropriately to other languages. Because the model was trained on large amounts of web data, it carries the stereotypes and biases commonly found online. |
| huggingface-llm-falcon-7b-instruct-bf16 | Falcon7BInstruct | Falcon 7B Instruct is a 7 billion parameter causal large language model built on Falcon 7B and fine-tuned on a 250 million tokens mixture of chat/instruct datasets. Falcon 7B Instruct is mostly trained on English data, and does not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it carries the stereotypes and biases commonly encountered online. |
| huggingface-llm-falcon-40b-bf16 | Falcon40B | Falcon 40B is a 40 billion parameter causal large language model trained on 1,000 billion tokens enhanced with curated corpora. It is trained mostly on English, German, Spanish, and French, with limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, and Swedish. It does not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it carries the stereotypes and biases commonly encountered online. |
| huggingface-llm-falcon-40b-instruct-bf16 | Falcon40BInstruct | Falcon 40B Instruct is a 40 billion parameter causal large language model built on Falcon40B and fine-tuned on a mixture of Baize. It is mostly trained on English and French data, and does not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it carries the stereotypes and biases commonly encountered online. |
| huggingface-text2text-flan-t5-large | FlanT5L | The [https://huggingface.co/docs/transformers/model_doc/t5](https://huggingface.co/docs/transformers/model_doc/t5) model family is a set of large language models that are fine-tuned on multiple tasks and can be further trained. These models are well-suited for tasks such as language translation, text generation, sentence completion, word sense disambiguation, summarization, or question answering. Flan T5 L is a 780 million parameter large language model trained on numerous languages. You can find the list of the languages supported by Flan T5 L in the details of the model retrieved from your search by model ID in JumpStart's [model table](https://sagemaker.readthedocs.io/en/stable/doc_utils/pretrainedmodels.html#built-in-algorithms-with-pre-trained-model-table). |
| huggingface-text2text-flan-t5-xl | FlanT5XL | The [https://huggingface.co/docs/transformers/model_doc/t5](https://huggingface.co/docs/transformers/model_doc/t5) model family is a set of large language models that are fine-tuned on multiple tasks and can be further trained. These models are well-suited for tasks such as language translation, text generation, sentence completion, word sense disambiguation, summarization, or question answering. Flan T5 XL is a 3 billion parameter large language model trained on numerous languages. You can find the list of the languages supported by Flan T5 XL in the details of the model retrieved from your search by model ID in JumpStart's [model table](https://sagemaker.readthedocs.io/en/stable/doc_utils/pretrainedmodels.html#built-in-algorithms-with-pre-trained-model-table). |
| huggingface-text2text-flan-t5-xxll | FlanT5XXL | The [https://huggingface.co/docs/transformers/model_doc/t5](https://huggingface.co/docs/transformers/model_doc/t5) model family is a set of large language models that are fine-tuned on multiple tasks and can be further trained. These models are well-suited for tasks such as language translation, text generation, sentence completion, word sense disambiguation, summarization, or question answering. Flan T5 XXL is a 11 billion parameter model. You can find the list of the languages supported by Flan T5 XXL in the details of the model retrieved from your search by model ID in JumpStart's [model table](https://sagemaker.readthedocs.io/en/stable/doc_utils/pretrainedmodels.html#built-in-algorithms-with-pre-trained-model-table). |
| meta-textgeneration-llama-2-7b | Llama2-7B | Llama 2 is a collection of pretrained and fine-tuned generative text models, ranging in scale from 7 billion to 70 billion parameters. Llama2-7B is the 7 billion parameter model that is intended for English use and can be adapted for a variety of natural language generation tasks. |
| meta-textgeneration-llama-2-7b-f | Llama2-7BChat | Llama 2 is a collection of pretrained and fine-tuned generative text models, ranging in scale from 7 billion to 70 billion parameters. Llama2-7B is the 7 billion parameter chat model that is optimized for dialogue use cases. |
| meta-textgeneration-llama-2-13b | Llama2-13B | Llama 2 is a collection of pretrained and fine-tuned generative text models, ranging in scale from 7 billion to 70 billion parameters. Llama2-13B is the 13 billion parameter model that is intended for English use and can be adapted for a variety of natural language generation tasks. |
| meta-textgeneration-llama-2-13b-f | Llama2-13BChat | Llama 2 is a collection of pretrained and fine-tuned generative text models, ranging in scale from 7 billion to 70 billion parameters. Llama2-13B is the 13 billion parameter chat model that is optimized for dialogue use cases. |
| huggingface-llm-mistral-7b | Mistral7B | Mistral 7B is a seven billion parameters code and general purpose English text generation model. It can be used in a variety of use cases including text summarization, classification, text completion, or code completion. |
| huggingface-llm-mistral-7b-instruct | Mistral7BInstruct | Mistral 7B Instruct is the fine-tuned version of Mistral 7B for conversational use cases. It was specialized using a variety of publicly available conversation datasets in English. |
| huggingface-textgeneration1-mpt-7b-bf16 | MPT7B | MPT 7B is a decoder-style transformer large language model with 6.7 billion parameters, pre-trained from scratch on 1 trillion tokens of English text and code. It is prepared to handle long context lengths. |
| huggingface-textgeneration1-mpt-7b-instruct-bf16 | MPT7BInstruct | MPT 7B Instruct is a model for short-form instruction following tasks. It is built by fine-tuning MPT 7B on a dataset derived from [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) and the [Anthropic Helpful and Harmless (HH-RLHF)](https://huggingface.co/datasets/Anthropic/hh-rlhf) datasets. |
# Dataset file types and input data format
Instruction-based fine-tuning uses labeled datasets to improve the performance of pre-trained LLMs on specific natural language processing (NLP) tasks. The labeled examples are formatted as prompt-response pairs and phrased as instructions.
To learn about the supported dataset file types, see [Supported dataset file types](#autopilot-llms-finetuning-dataset-format).
To learn about input data format, see [Input data format for instruction-based fine-tuning](#autopilot-llms-finetuning-input-format).
## Supported dataset file types
Autopilot supports instruction-based fine-tuning datasets formatted as CSV files (default) or as Parquet files.
+ **CSV** (comma separated values) is a row-based file format that stores data in human readable plaintext, which is a popular choice for data exchange as it is supported by a wide range of applications.
+ **Parquet** is a binary, column-based file format where the data is stored and processed more efficiently than in human readable file formats such as CSV. This makes it a better option for big data problems.
**Note**
The dataset may consist of multiple files, each of which must adhere to a specific template. For information on how to format your input data, see [Input data format for instruction-based fine-tuning](#autopilot-llms-finetuning-input-format).
## Input data format for instruction-based fine-tuning
Each file in the dataset must adhere to the following format:
+ The dataset must contain exactly two comma-separated and named columns, `input` and `output`. Autopilot does not allow any additional columns.
+ The `input` columns contain the prompts, and their corresponding `output` contains the expected answer. Both the `input` and `output` are in string format.
The following example illustrates the input data format for instruction-based fine-tuning in Autopilot.
```
input,output
"",""
```
**Note**
We recommend using datasets with a minimum of 1000 rows to ensure optimal learning and performance of the model.
Additionally, Autopilot sets a maximum limit on the number of rows in the dataset and the context length based on the type of model being used.
+ The limits on the number of rows in a dataset apply to the cumulative count of rows across all files within the dataset, including multiple files. If there are two [channel types](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLChannel.html) defined (one for training and one for validation), the limit applies to the total number of rows across all datasets within both channels. When the number of rows exceeds the threshold, the job fails with a validation error.
+ When the length of the input or output of a row in the dataset exceeds the limit set on the context of the language model, it is automatically truncated. If more than 60% of the rows in the dataset are truncated, whether in their input or output, Autopilot fails the job with a validation error.
The following table presents those limits for each model.
| JumpStart Model ID | `BaseModelName` in API request | Row Limit | Context Length Limit |
| --- | --- | --- | --- |
| huggingface-textgeneration-dolly-v2-3b-bf16 | Dolly3B | 10,000 rows | 1024 tokens |
| huggingface-textgeneration-dolly-v2-7b-bf16 | Dolly7B | 10,000 rows | 1024 tokens |
| huggingface-textgeneration-dolly-v2-12b-bf16 | Dolly12B | 10,000 rows | 1024 tokens |
| huggingface-llm-falcon-7b-bf16 | Falcon7B | 1,000 rows | 1024 tokens |
| huggingface-llm-falcon-7b-instruct-bf16 | Falcon7BInstruct | 1,000 rows | 1024 tokens |
| huggingface-llm-falcon-40b-bf16 | Falcon40B | 10,000 rows | 1024 tokens |
| huggingface-llm-falcon-40b-instruct-bf16 | Falcon40BInstruct | 10,000 rows | 1024 tokens |
| huggingface-text2text-flan-t5-large | FlanT5L | 10,000 rows | 1024 tokens |
| huggingface-text2text-flan-t5-xl | FlanT5XL | 10,000 rows | 1024 tokens |
| huggingface-text2text-flan-t5-xxll | FlanT5XXL | 10,000 rows | 1024 tokens |
| meta-textgeneration-llama-2-7b | Llama2-7B | 10,000 rows | 2048 tokens |
| meta-textgeneration-llama-2-7b-f | Llama2-7BChat | 10,000 rows | 2048 tokens |
| meta-textgeneration-llama-2-13b | Llama2-13B | 7,000 rows | 2048 tokens |
| meta-textgeneration-llama-2-13b-f | Llama2-13BChat | 7,000 rows | 2048 tokens |
| huggingface-llm-mistral-7b | Mistral7B | 10,000 rows | 2048 tokens |
| huggingface-llm-mistral-7b-instruct | Mistral7BInstruct | 10,000 rows | 2048 tokens |
| huggingface-textgeneration1-mpt-7b-bf16 | MPT7B | 10,000 rows | 1024 tokens |
| huggingface-textgeneration1-mpt-7b-instruct-bf16 | MPT7BInstruct | 10,000 rows | 1024 tokens |
# Hyperparameters for optimizing the learning process of your text generation models
You can optimize the learning process of your base model by adjusting any combination of the following hyperparameters. These parameters are available for all models.
+ **Epoch Count**: The `epochCount` hyperparameter determines how many times the model goes through the entire training dataset. It influences the training duration and can prevent overfitting when set appropriately. Large number of epochs may increase the overall runtime of fine-tuning jobs. We recommend setting a large `MaxAutoMLJobRuntimeInSeconds` within the `CompletionCriteria` of the `[TextGenerationJobConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TextGenerationJobConfig.html)` to avoid fine-tuning jobs from stopping prematurely.
+ **Batch Size**: The `batchSize` hyperparameter defines the number of data samples used in each iteration of training. It can affect the convergence speed and memory usage. With large batch size, the risk of out of memory (OOM) errors increases, which may surface as an internal server error in Autopilot. To check for such error, check the `/aws/sagemaker/TrainingJobs` log group for the training jobs launched by your Autopilot job. You can access those logs in CloudWatch from in the AWS management console. Choose **Logs**, and then choose the `/aws/sagemaker/TrainingJobs` **log group**. To remedy OOM errors, reduce the batch size.
We recommend starting with a batch size of 1, then incrementally increase it until an out of memory error occurs. As a reference, 10 epochs typically takes up to 72h to complete.
+ **Learning Rate**: The `learningRate` hyperparameter controls the step size at which a model's parameters are updated during training. It determines how quickly or slowly the model's parameters are updated during training. A high learning rate means that the parameters are updated by a large step size, which can lead to faster convergence but may also cause the optimization process to overshoot the optimal solution and become unstable. A low learning rate means that the parameters are updated by a small step size, which can lead to more stable convergence but at the cost of slower learning.
+ **Learning Rate Warmup Steps**: The `learningRateWarmupSteps` hyperparameter specifies the number of training steps during which the learning rate gradually increases before reaching its target or maximum value. This helps the model converge more effectively and avoid issues like divergence or slow convergence that can occur with an initially high learning rate.
To learn about how to adjust hyperparameters for your fine-tuning experiment in Autopilot and discover their possible values, see [How to set hyperparameters to optimize the learning process of a model](autopilot-create-experiment-finetune-llms.md#autopilot-llms-finetuning-set-hyperparameters).
# Metrics for fine-tuning large language models in Autopilot
The following section describes the metrics that you can use to understand your fine-tuned large language models (LLMs). Using your dataset, Autopilot directly fine-tunes a target LLM to enhance a default objective metric, the cross-entropy loss.
Cross-entropy loss is a widely used metric to assess the dissimilarity between the predicted probability distribution and the actual distribution of words in the training data. By minimizing cross-entropy loss, the model learns to make more accurate and contextually relevant predictions, particularly in tasks related to text generation.
After fine-tuning an LLM you can evaluate the quality of its generated text using a range of ROUGE scores. Additionally, you can analyze the perplexity and cross-entropy training and validation losses as part of the evaluation process.
+ Perplexity loss measures how well the model can predict the next word in a sequence of text, with lower values indicating a better understanding of the language and context.
+ Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of metrics used in the field of natural language processing (NLP) and machine learning to evaluate the quality of machine-generated text, such as text summarization or text generation. It primarily assesses the similarities between the generated text and the ground truth reference (human-written) text of a validation dataset. ROUGE measures are designed to assess various aspects of text similarity, including the precision and recall of n-grams (contiguous sequences of words) in the system-generated and reference texts. The goal is to assess how well a model captures the information present in the reference text.
There are several variants of ROUGE metrics, depending on the type of n-grams used and the specific aspects of text quality being evaluated.
The following list contains the name and description of the ROUGE metrics available after the fine-tuning of large language models in Autopilot.
**`ROUGE-1`, `ROUGE-2`**
ROUGE-N, the primary ROUGE metric, measures the overlap of n-grams between the system-generated and reference texts. ROUGE-N can be adjusted to different values of `n` (here `1` or `2`) to evaluate how well the system-generated text captures the n-grams from the reference text.
**`ROUGE-L`**
ROUGE-L (ROUGE-Longest Common Subsequence) calculates the longest common subsequence between the system-generated text and the reference text. This variant considers word order in addition to content overlap.
**`ROUGE-L-Sum`**
ROUGE-L-SUM (Longest Common Subsequence for Summarization) is designed for the evaluation of text summarization systems. It focuses on measuring the longest common subsequence between the machine-generated summary and the reference summary. ROUGE-L-SUM takes into account the order of words in the text, which is important in text summarization tasks.
# Autopilot model deployment and predictions
After fine-tuning a large language model (LLM), you can deploy the model for real-time text generation by setting up an endpoint to obtain interactive predictions.
**Note**
We recommend running real-time inference jobs on `ml.g5.12xlarge` for better performances. Alternatively, `ml.g5.8xlarge` instances are suitable for Falcon-7B-Instruct and MPT-7B-Instruct text generation tasks.
You can find the specifics of these instances within the [Accelerated Computing](https://aws.amazon.com/ec2/instance-types/) category in the selection of instance types provided by Amazon EC2.
## Real-time text generation
You can use SageMaker APIs to manually deploy your fine-tuned model to a SageMaker AI Hosting [real-time inference endpoint](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html), then begin making predictions by invoking the endpoint as follows.
**Note**
Alternatively, you can chose the automatic deployment option when creating your fine-tuning experiment in Autopilot. For information on setting up the automatic deployment of models, see [How to enable automatic deployment](autopilot-create-experiment-finetune-llms.md#autopilot-llms-finetuning-auto-model-deployment).
You can also use the SageMaker Python SDK and the `JumpStartModel` class to perform inferences with models fine-tuned by Autopilot. This can be done by specifying a custom location for the model's artifact in Amazon S3. For information on defining your model as a JumpStart model and deploying your model for inference, see [Low-code deployment with the JumpStartModel class](https://sagemaker.readthedocs.io/en/stable/overview.html#deploy-a-pre-trained-model-directly-to-a-sagemaker-endpoint).
1. **Obtain the candidate inference container definitions**
You can find the `InferenceContainerDefinitions` within the `BestCandidate` object retrieved from the response to the [DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html#API_DescribeAutoMLJobV2_ResponseSyntax) API call. A container definition for inference refers to the containerized environment designed for deploying and running your trained model to make predictions.
The following AWS CLI command example uses the [DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html) API to obtain recommended container definitions for your job name.
```
aws sagemaker describe-auto-ml-job-v2 --auto-ml-job-name job-name --region region
```
1. **Create a SageMaker AI model**
Use the container definitions from the previous step to create a SageMaker AI model by using the [CreateModel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) API. See the following AWS CLI command as an example. Use the `CandidateName` for your model name.
```
aws sagemaker create-model --model-name '' \
--primary-container '' --region '
```
1. **Create an endpoint configuration**
The following AWS CLI command example uses the [CreateEndpointConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html) API to create an endpoint configuration.
**Note**
To prevent the endpoint creation from timing out due to a lengthy model download, we recommend setting `ModelDataDownloadTimeoutInSeconds = 3600` and `ContainerStartupHealthCheckTimeoutInSeconds = 3600`.
```
aws sagemaker create-endpoint-config --endpoint-config-name '' \
--production-variants '' ModelDataDownloadTimeoutInSeconds=3600 ContainerStartupHealthCheckTimeoutInSeconds=3600 \
--region ''
```
1. **Create the endpoint**
The following AWS CLI example uses the [CreateEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpoint.html) API to create the endpoint.
```
aws sagemaker create-endpoint --endpoint-name '' \
--endpoint-config-name '' \
--region ''
```
Check the progress of your endpoint deployment by using the [DescribeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpoint.html) API. See the following AWS CLI command as an example.
```
aws sagemaker describe-endpoint —endpoint-name '' —region
```
After the `EndpointStatus` changes to `InService`, the endpoint is ready to use for real-time inference.
1. **Invoke the endpoint**
The following command invokes the endpoint for real-time inferencing. Your prompt needs to be encoded in bytes.
**Note**
The format of your input prompt depends on the language model. For more information on the format of text generation prompts, see [Request format for text generation models real-time inference](#autopilot-llms-finetuning-realtime-prompt-examples).
```
aws sagemaker invoke-endpoint --endpoint-name '' \
--region '' --body '' [--content-type] 'application/json'
```
## Request format for text generation models real-time inference
Different large language models (LLMs) may have specific software dependencies, runtime environments, and hardware requirements influencing Autopilot's recommended container to host the model for inference. Additionally, each model dictates the required input data format and the expected format for predictions and outputs.
Here are example inputs for some models and recommended containers.
+ For Falcon models with the recommended container `huggingface-pytorch-tgi-inference:2.0.1-tgi1.0.3-gpu-py39-cu118-ubuntu20.04`:
```
payload = {
"inputs": "Large language model fine-tuning is defined as",
"parameters": {
"do_sample": false,
"top_p": 0.9,
"temperature": 0.1,
"max_new_tokens": 128,
"stop": ["<|endoftext|>", ""]
}
}
```
+ For all other models with the recommended container `djl-inference:0.22.1-fastertransformer5.3.0-cu118`:
```
payload= {
"text_inputs": "Large language model fine-tuning is defined as"
}
```
# Create a Regression or Classification Autopilot experiment for tabular data using the Studio Classic UI
**Important**
As of November 30, 2023, Autopilot's UI is migrating to [Amazon SageMaker Canvas](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas.html) as part of the updated [Amazon SageMaker Studio](studio-updated.md) experience. SageMaker Canvas provides analysts and citizen data scientists no-code capabilities for tasks such as data preparation, feature engineering, algorithm selection, training and tuning, inference, and more. Users can leverage built-in visualizations and what-if analysis to explore their data and different scenarios, with automated predictions enabling them to easily productionize their models. Canvas supports a variety of use cases, including computer vision, demand forecasting, intelligent search, and generative AI.
Users of [Amazon SageMaker Studio Classic](studio.md), the previous experience of [Studio](studio-updated.md), can continue using the Autopilot UI in Studio Classic. Users with coding experience can continue using all [API references](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-reference.html) in any supported SDK for technical implementation.
If you have been using Autopilot in Studio Classic until now and want to migrate to SageMaker Canvas, you might have to grant additional permissions to your user profile or IAM role so that you can create and use the SageMaker Canvas application. For more information, see [(Optional) Migrate from Autopilot in Studio Classic to SageMaker Canvas](studio-updated-migrate-ui.md#studio-updated-migrate-autopilot).
All UI-related instructions in this guide pertain to Autopilot's standalone features before migrating to [Amazon SageMaker Canvas](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas.html). Users following these instructions should use [Studio Classic](studio.md).
You can use the Amazon SageMaker Studio Classic UI to create Autopilot experiments for classification or regression problems on tabular data. The UI helps you specify the name of your experiment, provide locations for the input and output data, and specify which target data to predict. Optionally, you can also specify the type of problem that you want to solve (regression, classification, multiclass classification), choose your modeling strategy (*stacked ensembles* or *hyperparameters optimization*), select the list of algorithms used by the Autopilot job to train the data, and more.
The UI has descriptions, toggle switches, dropdown menus, radio buttons, and more to help you navigate creating your model candidates. After the experiment runs, you can compare trials and delve into the details of the pre-processing steps, algorithms, and hyperparameter ranges of each model. Optionally, you can download their [explainability](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-explainability.html) and [performance](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-model-insights.html) reports. Use the provided [ notebooks](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-automate-model-development-notebook-output.html ) to see the results of the automated data exploration or the candidate model definitions.
Alternatively, you can use Autopilot AutoML API in [Create Regression or Classification Jobs for Tabular Data Using the AutoML API](autopilot-automate-model-development-create-experiment.md).
# Configure the default parameters of an Autopilot experiment (for administrators)
Autopilot supports setting default values to simplify the configuration of Amazon SageMaker Autopilot when you create an Autopilot experiment using the Studio Classic UI. Administrators can use Studio Classic [lifecycle configurations](studio-lcc.md) (LCC) to set infrastructure, networking, and security values in configuration files and pre-populate the [advanced settings](autopilot-automate-model-development-create-experiment-ui.md#advanced-settings) of `AutoML` jobs.
By doing so, they can fully control network connectivity and access permissions for the resources associated with Amazon SageMaker Studio Classic, including SageMaker AI instances, data sources, output data, and other related services. Specifically, administrators can configure a desired network architecture, such as Amazon VPC, subnets, and security groups, for a Studio Classic domain or individual user profiles. Data scientists can focus on data science specific parameters when creating their Autopilot experiments using the Studio Classic UI. Furthermore, administrators can manage the encryption of data on the instance in which Autopilot experiments run by setting default encryption keys.
**Note**
This feature is currently not available in the Asia Pacific (Hong Kong) and Middle East (Bahrain) opt-in Regions.
In the following sections, you can find the full list of parameters supporting the setting of defaults when creating an Autopilot experiment using the Studio Classic UI, and learn how to set those default values.
**Topics**
+ [List of default parameters supported](#autopilot-list-default-parameters-create-experiment)
+ [Set default Autopilot experiment parameters](#autopilot-set-default-parameters-create-experiment-howto)
## List of default parameters supported
The following parameters support setting default values with a configuration file for creating an Autopilot experiment using the Studio Classic UI. Once set, the values automatically fill in their corresponding field in the Autopilot' **Create Experiment** tab in the Studio Classic UI. See [Advanced settings (optional)](autopilot-automate-model-development-create-experiment-ui.md#advanced-settings) for a full description of each field.
+ **Security:** Amazon VPC, subnets, and security groups.
+ **Access:** AWS IAM role ARNs.
+ **Encryption:** AWS KMS key IDs.
+ **Tags:** Key-value pairs used to label and organize SageMaker AI resources.
## Set default Autopilot experiment parameters
Administrators can set default values in a configuration file, then manually place the file in a recommended location within the Studio Classic environment of specific users, or they can pass the file to a lifecycle configuration script (LCC) to automate the customization of the Studio Classic environment for a given domain or user profile.
+ To set up the configuration file, start by filling in its default parameters.
To configure any or all default values listed in [List of default parameters supported](#autopilot-list-default-parameters-create-experiment), administrators can create a configuration file named `config.yaml`, the structure of which should adhere to this [sample configuration file](https://sagemaker.readthedocs.io/en/stable/overview.html#configuration-file-structure). The following snippet shows a sample configuration file with all the supported `AutoML` parameters. For more information on the format of this file, refer to the [full schema](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/config/config_schema.py).
```
SchemaVersion: '1.0'
SageMaker:
AutoMLJob:
# https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html
AutoMLJobConfig:
SecurityConfig:
EnableInterContainerTrafficEncryption: true
VolumeKmsKeyId: 'kms-key-id'
VpcConfig:
SecurityGroupIds:
- 'security-group-id-1'
- 'security-group-id-2'
Subnets:
- 'subnet-1'
- 'subnet-2'
OutputDataConfig:
KmsKeyId: 'kms-key-id'
RoleArn: 'arn:aws:iam::111222333444:role/Admin'
Tags:
- Key: 'tag_key'
Value: 'tag_value'
```
+ Then, place the configuration file in the recommended location by either [manually copying the file](#autopilot-intelligent-defaults-manual-setup) to its recommended paths or using a [lifecycle configuration](#autopilot-intelligent-defaults-lcc-setup) (LCC).
The configuration file needs to be present in at least one of the following locations in the user's Studio Classic environment. By default, SageMaker AI searches for a configuration file in two locations:
+ First, in `/etc/xdg/sagemaker/config.yaml`. We refer to this file as the *administrator configuration file*.
+ Then, in `/root/.config/sagemaker/config.yaml`. We refer to this file as the *user configuration file*.
Using the *administrator* configuration file, administrators can define a set of default values. Optionally, they can use the *user* configuration file to override values set in the *administrator* configuration file, or set additional default parameter values.
The following snippet shows a sample script which writes the default parameters configuration file to the *administrator* location in the user's Studio Classic environment. You can replace `/etc/xdg/sagemaker` with `/root/.config/sagemaker` to write the file to the *user* location.
```
## Sample script with AutoML intelligent defaults
#!/bin/bash
sudo mkdir -p /etc/xdg/sagemaker
echo "SchemaVersion: '1.0'
CustomParameters:
AnyStringKey: 'AnyStringValue'
SageMaker:
AutoMLJob:
# https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html
AutoMLJobConfig:
SecurityConfig:
EnableInterContainerTrafficEncryption: true
VolumeKmsKeyId: 'kms-key-id'
VpcConfig:
SecurityGroupIds:
- 'security-group-id-1'
- 'security-group-id-2'
Subnets:
- 'subnet-1'
- 'subnet-2'
OutputDataConfig:
KmsKeyId: 'kms-key-id'
RoleArn: 'arn:aws:iam::111222333444:role/Admin'
Tags:
- Key: 'tag_key'
Value: 'tag_value'
" | sudo tee /etc/xdg/sagemaker/config.yaml
```
+ **Copy the files manually** – To copy the configuration files manually, run the [script](#autopilot-intelligent-defaults-manual-setup) created in the previous step from a Studio Classic terminal. In this case, the user profile that executed the script can create Autopilot experiments with the default values applicable only to them.
+ **Create a SageMaker AI lifecycle configuration** – Alternatively, you can use a [lifecycle configuration](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-lcc.html) (LCC) to automate the customization of your Studio Classic environment. LCC are shell scripts triggered by Amazon SageMaker Studio Classic lifecycle events such as starting a Studio Classic application. This customization includes installing custom packages, configuring notebook extensions, pre-loading datasets, setting up source code repositories, or, in our case, pre-populating default parameters. Administrators can attach the LCC to a Studio Classic domain to automate the configuration of default values for each user profile within that domain.
The following sections detail how to create a lifecycle configuration so users can load Autopilot default parameters automatically when launching Studio Classic. You can choose to create an LCC using the SageMaker AI Console or the AWS CLI.
------
#### [ Create a LCC from the SageMaker AI Console ]
Use the following steps to create an LCC containing your default parameters, attach the LCC to a domain or a user profile, then launch a Studio Classic application pre-populated with the default parameters set by the LCC using the SageMaker AI Console.
+ **To create a lifecycle configuration that runs the [script](#autopilot-intelligent-defaults-script) containing your default values using the SageMaker AI Console**
+ Open the SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/).
+ On the left side, navigate to **Admin configurations**, then **Lifecycle configurations**.
+ From the **Lifecycle configurations** page, navigate to the Studio Classic tab, then choose **Create configuration**.
+ For **Name**, type a name using alphanumeric characters and "-", but no spaces. The name can have a maximum of 63 characters.
+ Paste your [script](#autopilot-intelligent-defaults-script) in the **Scripts** section.
+ Choose **Create configuration** to create the lifecycle configuration. This creates an LCC of type `Kernel gateway app`.
+ **To attach the lifecycle configuration to a Studio Classic domain, a space, or a user profile**
Follow the steps in [Attach the lifecycle configuration to Studio Classic domain or user profile ](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-lcc-create-console.html#studio-lcc-create-console-step2) to attach your LCC to a Studio Classic domain or a specific user profile.
+ **To launch your Studio Classic application with the lifecycle configuration**
Once the LCC is attached to a domain or a user profile, impacted users can start a Studio Classic application from the landing page of Studio Classic in Studio to pick up the defaults set by the LCC automatically. This auto-populates the Studio Classic UI when creating an Autopilot experiment.
------
#### [ Create a LCC from the AWS CLI ]
Use the following snippets to launch a Studio Classic application that runs your [script](#autopilot-intelligent-defaults-manual-setup) using the AWS CLI. Note that `lifecycle_config.sh` is the name given to your script in this example.
Before getting started:
+ Ensure that you have updated and configured AWS CLI by completing the prerequisites described in [Create a lifecycle configuration from the AWS CLI](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-lcc-create-cli.html).
+ Install [OpenSSL](https://www.openssl.org/source/) documentation. The AWS CLI command uses the open-source library *OpenSSL* to encode your script in Base64 format. This requirement prevents errors that occur from spacing and line break encoding.
You can now follow these three steps:
+ **Create a new lifecycle configuration referencing the configuration script `lifecycle_config.sh`**
```
LCC_CONTENT=`openssl base64 -A -in lifecycle_config.sh`
## Create a new lifecycle config
aws sagemaker create-studio-lifecycle-config --region region \
--studio-lifecycle-config-name lcc-name \
--studio-lifecycle-config-content $LCC_CONTENT \
--studio-lifecycle-config-app-type default
```
Note the ARN of the newly created lifecycle configuration that is returned. This ARN is required to attach the lifecycle configuration to your application.
+ **Attach the lifecycle configuration to your `JupyterServerApp`**
The following example shows how to create a new user profile with a lifecycle configuration attached. To update an existing user profile, use the AWS CLI [update-user-profile](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sagemaker/update-user-profile.html) command. To create or update a domain, see [create-domain](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sagemaker/create-domain.html) and [update-domain](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sagemaker/update-domain.html). Add the lifecycle configuration ARN from the previous step to the settings of the `JupyterServerAppSettings` application type. You can add multiple lifecycle configurations at the same time by using a list of lifecycle configurations.
```
# Create a new UserProfile
aws sagemaker create-user-profile --domain-id domain-id \
--user-profile-name user-profile-name \
--region region \
--user-settings '{
"JupyterServerAppSettings": {
"LifecycleConfigArns":
["lifecycle-configuration-arn"]
}
}'
```
Once the LCC is attached to a domain or a user profile, impacted users can shut down and update their existing Studio Classic application by following the steps in [Shut down and Update Amazon SageMaker Studio Classic](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-tasks-update-studio.html), or start a new Studio Classic application from the AWS Console to pick up the defaults set by the LCC automatically. This auto-populates the Studio Classic UI when creating an Autopilot experiment. Alternatively, they can launch a new Studio Classic application using the AWS CLI as follows.
+ **Launch your Studio Classic application with the lifecycle configuration using the AWS CLI**
```
# Create a Jupyter Server application
aws sagemaker create-app --domain-id domain-id \
--user-profile-name user-profile-name \
--region region \
--app-type JupyterServer \
--resource-spec LifecycleConfigArn=lifecycle-configuration-arn \
--app-name default
```
For more information on creating a lifecycle configuration using the AWS CLI, see [Create a Lifecycle Configuration from the AWS CLI](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-lcc-create-cli.html).
------
**To create an Autopilot experiment using Studio Classic UI**
1. Sign in at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/), choose **Studio** from the left navigation pane, select your Domain and user profile, then **Open Studio**.
1. In Studio, choose the Studio Classic icon in the top left navigation pane. This opens a Studio Classic app.
1. Run or open a Studio Classic application from the space of your choice, or **Create Studio Classic space. **. On the **Home** tab, choose the **AutoML** card. This opens a new **AutoML** tab.
1. Choose **Create an AutoML experiment**. This opens a new **Create experiment** tab.
1. In the **Experiment and data details** section, enter the following information:
1. **Experiment name** – Must be unique to your account in the current AWS Region and contain a maximum of 63 alphanumeric characters. Can include hyphens (-) but not spaces.
1. **Input data** – Provide the Amazon Simple Storage Service (Amazon S3) bucket location of your input data. This S3 bucket must be in your current AWS Region. The URL must be in an `s3://` format where Amazon SageMaker AI has write permissions. The file must be in CSV or Parquet format and contain at least 500 rows. Select **Browse** to scroll through available paths and **Preview** to see a sample of your input data.
1. **Is your S3 input a manifest file?** – A manifest file includes metadata with your input data. The metadata specifies the location of your data in Amazon S3. It also specifies how the data is formatted and which attributes from the dataset to use when training your model. You can use a manifest file as an alternative to preprocessing when your labeled data is being streamed in `Pipe` mode.
1. **Auto split data?** – Autopilot can split your data into an 80-20% split for training and validation data. If you prefer a custom split, you can choose the **Specify split ratio**. To use a custom dataset for validation, choose **Provide a validation set**.
1. **Output data location (S3 bucket)** – The name of the S3 bucket location where you want to store the output data. The URL for this bucket must be in an Amazon S3 format where Amazon SageMaker AI has write permissions. The S3 bucket must be in the current AWS Region. Autopilot can also create this for you in the same location as your input data.
1. Choose **Next: Target and features**. The **Target and features** tab opens.
1. In the **Target and features** section:
+ Select a column to set as a target for model predictions.
+ Optionally, you can pass the name of a sample weights column in the **Sample weight** section to request your dataset rows to be weighted during training and evaluation. For more information on the available objective metrics, see [Autopilot weighted metrics](autopilot-metrics-validation.md#autopilot-weighted-metrics).
**Note**
Support for sample weights is available in [ensembling mode](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-model-support-validation.html#autopilot-training-mode) only.
+ You can also select features for training and change their data type. The following data types are available: `Text`, `Numerical`, `Categorical`, `Datetime`, `Sequence`, and `Auto`. All features are selected by default.
1. Choose **Next: Training method**. The **Training method** tab opens.
1. In the **Training method** section, select your training option: **Ensembling**, **Hyperparameter optimization (HPO)**, or **Auto** to let Autopilot choose the training method automatically based on the dataset size. Each training mode runs a pre-defined set of algorithms on your dataset to train model candidates. By default, Autopilot pre-selects all the available algorithms for the given training mode. You can run an Autopilot training experiment with all the algorithms or choose your own subset.
For more information on the training modes and the available algorithms, see the **Autopilot training modes** section in the [Training modes and algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-model-support-validation.html) page.
1. Choose **Next: Deployment and advanced settings** to open the **Deployment and advanced settings** tab. Settings include the auto-display endpoint name, machine learning problem type, and additional choices for running your experiment.
1. **Deployment settings** – Autopilot can automatically create an endpoint and deploy your model for you.
To auto-deploy to an automatically generated endpoint, or to provide an endpoint name for custom deployment, set the toggle to **Yes** under **Auto deploy?** If you are importing data from Amazon SageMaker Data Wrangler, you have additional options to auto-deploy the best model with or without the transforms from Data Wrangler.
**Note**
If your Data Wrangler flow contains multi-row operations such as `groupby`, `join`, or `concatenate`, you can't auto-deploy with these transforms. For more information, see [Automatically Train Models on Your Data Flow](https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-autopilot.html).
1. **Advanced settings (optional)** – Autopilot provides additional controls to manually set experimental parameters such as defining your problem type, time constraints on your Autopilot job and trials, security, and encryption settings.
**Note**
Autopilot supports the setting of default values to simplify the configuration of Autopilot experiments using Studio Classic UI. Administrators can use Studio Classic [lifecycle configurations](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-lcc.html) (LCC) to set infrastructure, networking, and security values in configuration files and pre-populate the *advanced settings* of `AutoML` jobs.
To learn about how administrators can automate the customization of an Autopilot experiment, see [Configure the default parameters of an Autopilot experiment (for administrators)](autopilot-set-default-parameters-create-experiment.md).
1. **Machine learning problem type** – Autopilot can automatically infer the type of supervised learning problem from your dataset. If you prefer to choose it manually, you can use the **Select the machine learning problem type** dropdown menu. Note that it defaults to **Auto**. In some cases, SageMaker AI is unable to infer accurately. When that happens, you must provide the value for the job to succeed. In particular, you can choose from the following types:
+ **Binary classification**– Binary classification assigns input data to one of two predefined and mutually exclusive classes, based on their attributes, such as medical diagnosis based on results of diagnostic tests that determine if someone has a disease.
+ **Regression** – Regression establishes a relationship between the input variables (also known as independent variables or features) and the target variable (also known as the dependent variable). This relationship is captured through a mathematical function or model that maps the input variables to a continuous output. It is commonly used for tasks such as predicting house prices based on features like square footage and the number of bathrooms, stock market trends, or estimating sales figures.
+ **Multiclass classification** – Multiclass classification assigns input data to one of several classes based on their attributes, like the prediction of the topic most relevant to a text document, such as politics, finance, or philosophy.
1. **Runtime** – You can define a maximum time limit. Upon reaching the time limit, trials and jobs that exceed the time constraint automatically stop.
1. **Access** – You can choose the role that Amazon SageMaker Studio Classic assumes to gain temporary access to AWS services (in particular, SageMaker AI and Amazon S3) on your behalf. If no role is explicitly defined, Studio Classic automatically uses the default SageMaker AI execution role attached to your user profile.
1. **Encryption** – To enhance the security of your data at rest and protect it against unauthorized access, you can specify encryption keys to encrypt data in your Amazon S3 buckets and in the Amazon Elastic Block Store (Amazon EBS) volume attached to your Studio Classic domain.
1. **Security** – You can choose the virtual private cloud (Amazon VPC) in which your SageMaker AI job runs. Ensure that the Amazon VPC has access to your input and output Amazon S3 buckets.
1. **Project** – Specify the name of the SageMaker AI project to associate with this Autopilot experiment and model outputs. When you specify a project, Autopilot tags the project to an experiment. This lets you know which model outputs are associated with this project.
1. **Tags** – Tags are an array of key-value pairs. Use tags to categorize your resources from AWS services, such as their purpose, owner, or environment.
1. Choose **Next: Review and create** to get a summary of your Autopilot experiment before you create it.
1. Select **Create experiment**.The creation of the experiment starts an Autopilot job in SageMaker AI. Autopilot provides the status of the experiment, information on the data exploration process and model candidates in notebooks, a list of generated models and their reports, and the job profile used to create them.
For information on the notebooks generated by an Autopilot job, see [Autopilot notebooks generated to manage AutoML tasks](autopilot-automate-model-development-notebook-output.md). For information on the details of each model candidate and their reports, see [View model details](autopilot-models-details.md) and [View an Autopilot model performance report](autopilot-model-insights.md).
**Note**
To avoid incurring unnecessary charges: If you deploy a model that is no longer needed, delete the endpoints and resources that were created during that deployment. Information about pricing instances by Region is available at [Amazon SageMaker Pricing](https://aws.amazon.com/sagemaker/pricing/).
# Amazon SageMaker Autopilot example notebooks
The following notebooks serve as practical, hands-on examples that address various use cases of Autopilot.
You can find all of Autopilot's notebooks in the [https://github.com/aws/amazon-sagemaker-examples/tree/main/autopilot](https://github.com/aws/amazon-sagemaker-examples/tree/main/autopilot) directory of SageMaker AI GitHub examples repository.
We recommend cloning the full Git repository within Studio Classic to access and run the notebooks directly. For information on how to clone a Git repository in Studio Classic, see [Clone a Git Repository in Amazon SageMaker Studio Classic](studio-tasks-git.md).
| **Use case** | **Description** |
| --- | --- |
| [Serverless inference](https://github.com/aws/amazon-sagemaker-examples/tree/main/autopilot/autopilot-serverless-inference) | By default, Autopilot allows deploying generated models to real-time inference endpoints. In this repository, the notebook illustrates how to deploy Autopilot models trained with `ENSEMBLING` and `HYPERPARAMETER OPTIMIZATION (HPO)` modes to serverless endpoints. Serverless endpoints automatically launch compute resources and scale them in and out depending on traffic, eliminating the need to choose instance types or manage scaling policies. |
| [Custom feature selection](https://github.com/aws/amazon-sagemaker-examples/tree/main/autopilot/custom-feature-selection) | Autopilot inspects your data set, and runs a number of candidates to figure out the optimal combination of data preprocessing steps, machine learning algorithms, and hyperparameters. You can easily deploy either on a real-time endpoint or for batch processing. In some cases, you might want to have the flexibility to bring custom data processing code to Autopilot. For example, your datasets might contain a large number of independent variables, and you may wish to incorporate a custom feature selection step to remove irrelevant variables first. The resulting smaller dataset can then be used to launch an Autopilot job. Ultimately, you would also want to include both the custom processing code and models from Autopilot for real-time or batch processing. |
| [Pipeline example](https://github.com/aws/amazon-sagemaker-examples/tree/main/autopilot/sagemaker-autopilot-pipelines) | While Autopilot streamlines the process of building ML models, MLOps engineers are still responsible for creating, automating, and managing end-to-end ML workflows in production. SageMaker Pipelines can assist in automating various steps of the ML lifecycle, such as data preprocessing, model training, hyperparameter tuning, model evaluation, and deployment. This notebook serves as a demonstration of how to incorporate Autopilot into a SageMaker Pipelines end-to-end AutoML training workflow. To launch an Autopilot experiment within Pipelines, you must create a model-building workflow by writing custom integration code using Pipelines [Lambda](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda) or [Processing](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) steps. For more information, refer to [Move Amazon SageMaker Autopilot ML models from experimentation to production using Amazon SageMaker Pipelines](https://aws.amazon.com/blogs/machine-learning/move-amazon-sagemaker-autopilot-ml-models-from-experimentation-to-production-using-amazon-sagemaker-pipelines/). Alternatively, when using Autopilot in [Ensembling mode](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-model-support-validation.html), you can refer to the notebook example that demonstrates how to use native AutoML step in [SageMaker Pipeline's native AutoML step](https://github.com/aws/amazon-sagemaker-examples/blob/main/autopilot/sagemaker-autopilot-pipelines/autopilot_pipelines_demo_notebook.ipynb). With Autopilot supported as a native step within Pipelines, you can now add an automated training step ([AutoMLStep](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-automl)) to your Pipelines and invoke an Autopilot experiment in Ensembling mode. |
| [ Direct marketing with Amazon SageMaker Autopilot](https://sagemaker-examples.readthedocs.io/en/latest/autopilot/sagemaker_autopilot_direct_marketing.html) | This notebook demonstrates how uses the [Bank Marketing Data Set](https://archive.ics.uci.edu/ml/datasets/bank+marketing) to predict whether a customer will enroll for a term deposit at a bank. You can use Autopilot on this dataset to get the most accurate ML pipeline by exploring options contained in various candidate pipelines. Autopilot generates each candidate in a two-step procedure. The first step performs automated feature engineering on the dataset. The second step trains and tunes an algorithm to produce a model. The notebook contains instructions on how to train the model and how to deploy the model to perform batch inference using the best candidate. |
| [Customer Churn Prediction with Amazon SageMaker Autopilot](https://sagemaker-examples.readthedocs.io/en/latest/autopilot/autopilot_customer_churn.html) | This notebook describes using machine learning for the automated identification of unhappy customers, also known as customer churn prediction. The example shows how to analyze a publicly available dataset and perform feature engineering on it. Next it shows how to tune a model by selecting the best performing pipeline along with the optimal hyperparameters for the training algorithm. Finally, it shows how to deploy the model to a hosted endpoint and how to evaluate its predictions against ground truth. However, ML models rarely give perfect predictions. That's why this notebook also shows how to incorporate the relative costs of prediction mistakes when determining the financial outcome of using ML. |
| [Top Candidates Customer Churn Prediction with Amazon SageMaker Autopilot and Batch Transform (Python SDK)](https://sagemaker-examples.readthedocs.io/en/latest/autopilot/autopilot_customer_churn_high_level_with_evaluation.html) | This notebook also describes using machine learning for the automated identification of unhappy customers, also known as customer churn prediction. This notebook demonstrates how to configure the model to obtain the inference probability, select the top N models, and make Batch Transform on a hold-out test set for evaluation. This notebook works with SageMaker Python SDK >= 1.65.1 released on 6/19/2020. |
| [Bringing your own data processing code to Amazon SageMaker Autopilot](https://sagemaker-examples.readthedocs.io/en/latest/autopilot/custom-feature-selection/Feature_selection_autopilot.html) | This notebook demonstrates how to incorporate and deploy custom data processing code when using Amazon SageMaker Autopilot. It adds a custom feature selection step to remove irrelevant variables to an Autopilot job. It then shows how to deploy both the custom processing code and models generated by Autopilot on a real-time endpoint and, alternatively, for batch processing. |
| More notebooks | You can find more notebooks illustrating other use cases such as [batch transform](https://github.com/aws/amazon-sagemaker-examples/blob/main/autopilot/ap-batch-transform.ipynb), [time-series forecasting](https://github.com/aws/amazon-sagemaker-examples/blob/main/autopilot/autopilot_time_series.ipynb) and more in the root directory. |
# Videos: Use Autopilot to automate and explore the machine learning process
Here is a video series that provides a tour of Amazon SageMaker Autopilot capabilities using Studio Classic. They show how to start an AutoML job, analyze and preprocess data, how to do feature engineering and hyperparameter optimization on candidate models, and how to visualize and compare the resulting model metrics.
**Topics**
+ [Start an AutoML job with Amazon SageMaker Autopilot](#autopilot-video-start-automl-job)
+ [Review data exploration and feature engineering automated in Autopilot.](#autopilot-video-generated-notebooks)
+ [Tune models to optimize performance](#autopilot-video-optimizing-model-performance)
+ [Choose and deploy the best model](#autopilot-video-choose-and-deploy-the-best-model)
+ [Amazon SageMaker Autopilot tutorial](#autopilot-walkthrough)
## Start an AutoML job with Amazon SageMaker Autopilot
This video shows you to how to start an AutoML job with Autopilot. (Length: 8:41)
[](http://www.youtube.com/watch?v=https://www.youtube.com/embed/qMEtqJPhqpA)
## Review data exploration and feature engineering automated in Autopilot.
This video shows you how to review the data exploration and candidate definition notebooks generated by Amazon SageMaker Autopilot. (Length: 10:04)
[](http://www.youtube.com/watch?v=https://www.youtube.com/embed/WsfRAeGzgm8)
## Tune models to optimize performance
This video shows you how to optimize model performance during training using hyperparameter tuning. (Length: 4:59)
[](http://www.youtube.com/watch?v=https://www.youtube.com/embed/KZSTsWrDGXs)
## Choose and deploy the best model
This video shows you how to use job metrics to choose the best model and then how to deploy it. (Length: 5:20)
[](http://www.youtube.com/watch?v=https://www.youtube.com/embed/vRHyX3kDstI)
## Amazon SageMaker Autopilot tutorial
This video walks you through an end to end demo where we first build a binary classification model automatically with Amazon SageMaker Autopilot. We see how candidate models have been built and optimized using auto-generated notebooks. We also look at the top candidates with Amazon SageMaker Experiments. Finally, we deploy the top candidate (based on XGBoost), and configure data capture with SageMaker Model Monitor.
[](http://www.youtube.com/watch?v=https://www.youtube.com/embed/DRjOOaR2prQ)
# Autopilot quotas
There are quotas that limit the resources available to you when using Amazon SageMaker Autopilot. Some of these limits are increasable and some are not.
**Note**
The resource quotas documented in the following sections are valid for versions of Amazon SageMaker Studio Classic 3.22.2 and higher. For information on updating your version of SageMaker Studio Classic, see [Shut Down and Update Amazon SageMaker Studio Classic and Apps](studio-tasks-update.md).
**Topics**
+ [Quotas that you can increase](#autopilot-quotas-limits-increasable)
+ [Resource quotas](#autopilot-quotas-resource-limits)
## Quotas that you can increase
The following table contains the resource limits for quotas you can increase:
| Resource | Regions | Default limits | Can be increased up to |
| --- | --- | --- | --- |
| Size of input dataset | All | 100 GB | Hundreds of GBs |
| Size of a single Parquet file\$1 | All | 2 GB | N/A |
| Target dataset size for subsampling\$1\$1 | All | 5 GB | Hundreds of GBs |
| Number of concurrent Autopilot jobs | us-east-1, us-east-2,us-west-2, ap-northeast-1, eu-west-1, eu-central-1 | 4 | Hundreds |
| Number of concurrent Autopilot jobs | ap-northeast-2, ap-southeast-2, eu-west-2, ap-southeast-1 | 2 | Hundreds |
| Number of concurrent Autopilot jobs | All other Regions | 1 | Tens |
**Note**
\$1This 2 GB size limit is for a single compressed Parquet file. You can provide a Parquet dataset that includes multiple compressed Parquet files up to the input dataset maximum size. After the files are decompressed, they may each expand to a larger size.
\$1\$1Autopilot automatically subsamples input datasets that are larger than the target dataset size while accounting for class imbalance and preserving rare class labels.
**To request a quota increase:**
1. Open the [ Service Quotas console](https://console.aws.amazon.com/servicequotas/home/services/sagemaker/quotas).
1. Select your quota increase, then choose **Request increase at account level**.
1. In the **Increase quota value**, enter the new limit value that you are requesting.
1. Choose **Request**.
## Resource quotas
The following table contains the runtime resource limits for an Amazon SageMaker Autopilot job in an AWS Region.
| Resource | Limit per Autopilot job |
| --- | --- |
| Maximum runtime for an Autopilot job | 30 days |
# API Reference guide for Autopilot
This section provides a subset of the HTTP service REST APIs for creating and managing Amazon SageMaker Autopilot resources (AutoML jobs) programmatically.
If your language of choice is Python, you can refer to [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html) or the [AutoMLV2 object](https://sagemaker.readthedocs.io/en/stable/api/training/automlv2.html#sagemaker.automl.automlv2.AutoMLV2) of the Amazon SageMaker Python SDK directly.
## AutoML API Actions
This list details the operations available in the Reference API to manage AutoML jobs programmatically.
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListAutoMLJobs.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListAutoMLJobs.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListCandidatesForAutoMLJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ListCandidatesForAutoMLJob.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_StopAutoMLJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_StopAutoMLJob.html)
**Note**
[CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJobV2.html) and [DescribeAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html) are new versions of [CreateAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateAutoMLJob.html) and [DescribeAutoMLJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html) which offer backward compatibility.
We recommend using the `CreateAutoMLJobV2`. `CreateAutoMLJobV2` can manage tabular problem types identical to those of its previous version `CreateAutoMLJob`, as well as non-tabular problem types such as image or text classification, or time-series forecasting.
Find guidelines about how to migrate a `CreateAutoMLJob` to `CreateAutoMLJobV2` in [Migrate a CreateAutoMLJob to CreateAutoMLJobV2](https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-automate-model-development-create-experiment.html#autopilot-create-experiment-api-migrate-v1-v2).
## AutoML API Data Types
This list details the API AutoML objects used by the actions above as inbound requests or outbound responses.
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLAlgorithmConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLAlgorithmConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidateGenerationConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidateGenerationConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidateStep.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidateStep.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLChannel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLChannel.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLContainerDefinition.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLContainerDefinition.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSource.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSource.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLDataSplitConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLInferenceContainerDefinitions.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLInferenceContainerDefinitions.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobArtifacts.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobArtifacts.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobChannel.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobCompletionCriteria.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobCompletionCriteria.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobInputDataConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobInputDataConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobObjective.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobObjective.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobStepMetadata.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobStepMetadata.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobSummary.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobSummary.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLProblemTypeConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLProblemTypeConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobCompletionCriteria.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobCompletionCriteria.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobSummary.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLJobSummary.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLOutputDataConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLPartialFailureReason.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLPartialFailureReason.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLProblemTypeConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLProblemTypeConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLProblemTypeResolvedAttributes.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLProblemTypeResolvedAttributes.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLResolvedAttributes.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLResolvedAttributes.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLSecurityConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLSecurityConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLS3DataSource.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLS3DataSource.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateArtifactLocations.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateArtifactLocations.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateGenerationConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateGenerationConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateProperties.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CandidateProperties.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_FinalAutoMLJobObjectiveMetric.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_FinalAutoMLJobObjectiveMetric.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_HolidayConfigAttributes.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_HolidayConfigAttributes.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ImageClassificationJobConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ImageClassificationJobConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_MetricDatum.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_MetricDatum.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployResult.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ModelDeployResult.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ResolvedAttributes.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ResolvedAttributes.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularJobConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularResolvedAttributes.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TabularResolvedAttributes.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TextGenerationJobConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TextGenerationJobConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TextGenerationResolvedAttribute.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TextGenerationResolvedAttribute.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesForecastingJobConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesForecastingJobConfig.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesTransformations.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TimeSeriesTransformations.html)
+ [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TuningJobCompletionCriteria.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_TuningJobCompletionCriteria.html)