# Hosting options The following topics describe available SageMaker AI realtime hosting options along with how to set up, invoke, and delete each hosting option. **Topics** + [Single-model endpoints](realtime-single-model.md) + [Multi-model endpoints](multi-model-endpoints.md) + [Multi-container endpoints](multi-container-endpoints.md) + [Inference pipelines in Amazon SageMaker AI](inference-pipelines.md) + [Delete Endpoints and Resources](realtime-endpoints-delete-resources.md) # Single-model endpoints You can create, update, and delete real-time inference endpoints that host a single model with Amazon SageMaker Studio, the AWS SDK for Python (Boto3), the SageMaker Python SDK, or the AWS CLI. For procedures and code examples, see [Deploy models for real-time inference](realtime-endpoints-deploy-models.md). # Multi-model endpoints Multi-model endpoints provide a scalable and cost-effective solution to deploying large numbers of models. They use the same fleet of resources and a shared serving container to host all of your models. This reduces hosting costs by improving endpoint utilization compared with using single-model endpoints. It also reduces deployment overhead because Amazon SageMaker AI manages loading models in memory and scaling them based on the traffic patterns to your endpoint. The following diagram shows how multi-model endpoints work compared to single-model endpoints. ![\[Diagram that shows how multi-model versus how single-model endpoints host models.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/multi-model-endpoints-diagram.png) Multi-model endpoints are ideal for hosting a large number of models that use the same ML framework on a shared serving container. If you have a mix of frequently and infrequently accessed models, a multi-model endpoint can efficiently serve this traffic with fewer resources and higher cost savings. Your application should be tolerant of occasional cold start-related latency penalties that occur when invoking infrequently used models. Multi-model endpoints support hosting both CPU and GPU backed models. By using GPU backed models, you can lower your model deployment costs through increased usage of the endpoint and its underlying accelerated compute instances. Multi-model endpoints also enable time-sharing of memory resources across your models. This works best when the models are fairly similar in size and invocation latency. When this is the case, multi-model endpoints can effectively use instances across all models. If you have models that have significantly higher transactions per second (TPS) or latency requirements, we recommend hosting them on dedicated endpoints. You can use multi-model endpoints with the following features: + [AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-services-overview.html) and VPCs + [Auto scaling](multi-model-endpoints-autoscaling.md) + [Serial inference pipelines](https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipelines.html) (but only one multi-model enabled container can be included in an inference pipeline) + A/B testing You can use the AWS SDK for Python (Boto) or the SageMaker AI console to create a multi-model endpoint. For CPU backed multi-model endpoints, you can create your endpoint with custom-built containers by integrating the [Multi Model Server](https://github.com/awslabs/multi-model-server) library. **Topics** + [How multi-model endpoints work](#how-multi-model-endpoints-work) + [Sample notebooks for multi-model endpoints](#multi-model-endpoint-sample-notebooks) + [Supported algorithms, frameworks, and instances for multi-model endpoints](multi-model-support.md) + [Instance recommendations for multi-model endpoint deployments](multi-model-endpoint-instance.md) + [Create a Multi-Model Endpoint](create-multi-model-endpoint.md) + [Invoke a Multi-Model Endpoint](invoke-multi-model-endpoint.md) + [Add or Remove Models](add-models-to-endpoint.md) + [Build Your Own Container for SageMaker AI Multi-Model Endpoints](build-multi-model-build-container.md) + [Multi-Model Endpoint Security](multi-model-endpoint-security.md) + [CloudWatch Metrics for Multi-Model Endpoint Deployments](multi-model-endpoint-cloudwatch-metrics.md) + [Set SageMaker AI multi-model endpoint model caching behavior](multi-model-caching.md) + [Set Auto Scaling Policies for Multi-Model Endpoint Deployments](multi-model-endpoints-autoscaling.md) ## How multi-model endpoints work SageMaker AI manages the lifecycle of models hosted on multi-model endpoints in the container's memory. Instead of downloading all of the models from an Amazon S3 bucket to the container when you create the endpoint, SageMaker AI dynamically loads and caches them when you invoke them. When SageMaker AI receives an invocation request for a particular model, it does the following: 1. Routes the request to an instance behind the endpoint. 1. Downloads the model from the S3 bucket to that instance's storage volume. 1. Loads the model to the container's memory (CPU or GPU, depending on whether you have CPU or GPU backed instances) on that accelerated compute instance. If the model is already loaded in the container's memory, invocation is faster because SageMaker AI doesn't need to download and load it. SageMaker AI continues to route requests for a model to the instance where the model is already loaded. However, if the model receives many invocation requests, and there are additional instances for the multi-model endpoint, SageMaker AI routes some requests to another instance to accommodate the traffic. If the model isn't already loaded on the second instance, the model is downloaded to that instance's storage volume and loaded into the container's memory. When an instance's memory utilization is high and SageMaker AI needs to load another model into memory, it unloads unused models from that instance's container to ensure that there is enough memory to load the model. Models that are unloaded remain on the instance's storage volume and can be loaded into the container's memory later without being downloaded again from the S3 bucket. If the instance's storage volume reaches its capacity, SageMaker AI deletes any unused models from the storage volume. To delete a model, stop sending requests and delete it from the S3 bucket. SageMaker AI provides multi-model endpoint capability in a serving container. Adding models to, and deleting them from, a multi-model endpoint doesn't require updating the endpoint itself. To add a model, you upload it to the S3 bucket and invoke it. You don’t need code changes to use it. **Note** When you update a multi-model endpoint, initial invocation requests on the endpoint might experience higher latencies as Smart Routing in multi-model endpoints adapt to your traffic pattern. However, once it learns your traffic pattern, you can experience low latencies for most frequently used models. Less frequently used models may incur some cold start latencies since the models are dynamically loaded to an instance. ## Sample notebooks for multi-model endpoints To learn more about how to use multi-model endpoints, you can try the following sample notebooks: + Examples for multi-model endpoints using CPU backed instances: + [Multi-Model Endpoint XGBoost Sample Notebook](https://sagemaker-examples.readthedocs.io/en/latest/advanced_functionality/multi_model_xgboost_home_value/xgboost_multi_model_endpoint_home_value.html) – This notebook shows how to deploy multiple XGBoost models to an endpoint. + [Multi-Model Endpoints BYOC Sample Notebook](https://sagemaker-examples.readthedocs.io/en/latest/advanced_functionality/multi_model_bring_your_own/multi_model_endpoint_bring_your_own.html) – This notebook shows how to set up and deploy a customer container that supports multi-model endpoints in SageMaker AI. + Example for multi-model endpoints using GPU backed instances: + [Run multiple deep learning models on GPUs with Amazon SageMaker AI Multi-model endpoints (MME)](https://github.com/aws/amazon-sagemaker-examples/blob/main/multi-model-endpoints/mme-on-gpu/cv/resnet50_mme_with_gpu.ipynb) – This notebook shows how to use an NVIDIA Triton Inference container to deploy ResNet-50 models to a multi-model endpoint. For instructions on how to create and access Jupyter notebook instances that you can use to run the previous examples in SageMaker AI, see [Amazon SageMaker notebook instances](nbi.md). After you've created a notebook instance and opened it, choose the **SageMaker AI Examples** tab to see a list of all the SageMaker AI samples. The multi-model endpoint notebooks are located in the **ADVANCED FUNCTIONALITY** section. To open a notebook, choose its **Use** tab and choose **Create copy**. For more information about use cases for multi-model endpoints, see the following blogs and resources: + Video: [Hosting thousands of models on SageMaker AI](https://www.youtube.com/watch?v=XqCNTWmHsLc&t=751s) + Video: [SageMaker AI ML for SaaS](https://www.youtube.com/watch?v=BytpYlJ3vsQ) + Blog: [How to scale machine learning inference for multi-tenant SaaS use cases](https://aws.amazon.com/blogs/machine-learning/how-to-scale-machine-learning-inference-for-multi-tenant-saas-use-cases/) + Case study: [Veeva Systems](https://aws.amazon.com/partners/success/advanced-clinical-veeva/) # Supported algorithms, frameworks, and instances for multi-model endpoints For information about the algorithms, frameworks, and instance types that you can use with multi-model endpoints, see the following sections. ## Supported algorithms, frameworks, and instances for multi-model endpoints using CPU backed instances The inference containers for the following algorithms and frameworks support multi-model endpoints: + [XGBoost algorithm with Amazon SageMaker AI](xgboost.md) + [K-Nearest Neighbors (k-NN) Algorithm](k-nearest-neighbors.md) + [Linear Learner Algorithm](linear-learner.md) + [Random Cut Forest (RCF) Algorithm](randomcutforest.md) + [Resources for using TensorFlow with Amazon SageMaker AI](tf.md) + [Resources for using Scikit-learn with Amazon SageMaker AI](sklearn.md) + [Resources for using Apache MXNet with Amazon SageMaker AI](mxnet.md) + [Resources for using PyTorch with Amazon SageMaker AI](pytorch.md) To use any other framework or algorithm, use the SageMaker AI inference toolkit to build a container that supports multi-model endpoints. For information, see [Build Your Own Container for SageMaker AI Multi-Model Endpoints](build-multi-model-build-container.md). Multi-model endpoints support all of the CPU instance types. ## Supported algorithms, frameworks, and instances for multi-model endpoints using GPU backed instances Hosting multiple GPU backed models on multi-model endpoints is supported through the [SageMaker AI Triton Inference server](https://docs.aws.amazon.com/sagemaker/latest/dg/triton.html). This supports all major inference frameworks such as NVIDIA® TensorRT™, PyTorch, MXNet, Python, ONNX, XGBoost, scikit-learn, RandomForest, OpenVINO, custom C\$1\$1, and more. To use any other framework or algorithm, you can use Triton backend for Python or C\$1\$1 to write your model logic and serve any custom model. After you have the server ready, you can start deploying 100s of Deep Learning models behind one endpoint. Multi-model endpoints support the following GPU instance types: | Instance family | Instance type | vCPUs | GiB of memory per vCPU | GPUs | GPU memory | | --- | --- | --- | --- | --- | --- | | p2 | ml.p2.xlarge | 4 | 15.25 | 1 | 12 | | p3 | ml.p3.2xlarge | 8 | 7.62 | 1 | 16 | | g5 | ml.g5.xlarge | 4 | 4 | 1 | 24 | | g5 | ml.g5.2xlarge | 8 | 4 | 1 | 24 | | g5 | ml.g5.4xlarge | 16 | 4 | 1 | 24 | | g5 | ml.g5.8xlarge | 32 | 4 | 1 | 24 | | g5 | ml.g5.16xlarge | 64 | 4 | 1 | 24 | | g4dn | ml.g4dn.xlarge | 4 | 4 | 1 | 16 | | g4dn | ml.g4dn.2xlarge | 8 | 4 | 1 | 16 | | g4dn | ml.g4dn.4xlarge | 16 | 4 | 1 | 16 | | g4dn | ml.g4dn.8xlarge | 32 | 4 | 1 | 16 | | g4dn | ml.g4dn.16xlarge | 64 | 4 | 1 | 16 | # Instance recommendations for multi-model endpoint deployments There are several items to consider when selecting a SageMaker AI ML instance type for a multi-model endpoint: + Provision sufficient [ Amazon Elastic Block Store (Amazon EBS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html) capacity for all of the models that need to be served. + Balance performance (minimize cold starts) and cost (don’t over-provision instance capacity). For information about the size of the storage volume that SageMaker AI attaches for each instance type for an endpoint and for a multi-model endpoint, see [Instance storage volumes](host-instance-storage.md). + For a container configured to run in `MultiModel` mode, the storage volume provisioned for its instances are larger than the default `SingleModel` mode. This allows more models to be cached on the instance storage volume than in `SingleModel` mode. When choosing a SageMaker AI ML instance type, consider the following: + Multi-model endpoints are currently supported for all CPU instances types and on single-GPU instance types. + For the traffic distribution (access patterns) to the models that you want to host behind the multi-model endpoint, along with the model size (how many models could be loaded in memory on the instance), keep the following information in mind: + Think of the amount of memory on an instance as the cache space for models to be loaded, and think of the number of vCPUs as the concurrency limit to perform inference on the loaded models (assuming that invoking a model is bound to CPU). + For CPU backed instances, the number of vCPUs impacts your maximum concurrent invocations per instance (assuming that invoking a model is bound to CPU). A higher amount of vCPUs enables you to invoke more unique models concurrently. + For GPU backed instances, a higher amount of instance and GPU memory enables you to have more models loaded and ready to serve inference requests. + For both CPU and GPU backed instances, have some "slack" memory available so that unused models can be unloaded, and especially for multi-model endpoints with multiple instances. If an instance or an Availability Zone fails, the models on those instances will be rerouted to other instances behind the endpoint. + Determine your tolerance to loading/downloading times: + d instance type families (for example, m5d, c5d, or r5d) and g5s come with an NVMe (non-volatile memory express) SSD, which offers high I/O performance and might reduce the time it takes to download models to the storage volume and for the container to load the model from the storage volume. + Because d and g5 instance types come with an NVMe SSD storage, SageMaker AI does not attach an Amazon EBS storage volume to these ML compute instances that hosts the multi-model endpoint. Auto scaling works best when the models are similarly sized and homogenous, that is when they have similar inference latency and resource requirements. You can also use the following guidance to help you optimize model loading on your multi-model endpoints: **Choosing an instance type that can't hold all of the targeted models in memory** In some cases, you might opt to reduce costs by choosing an instance type that can't hold all of the targeted models in memory at once. SageMaker AI dynamically unloads models when it runs out of memory to make room for a newly targeted model. For infrequently requested models, you sacrifice dynamic load latency. In cases with more stringent latency needs, you might opt for larger instance types or more instances. Investing time up front for performance testing and analysis helps you to have successful production deployments. **Evaluating your model cache hits** Amazon CloudWatch metrics can help you evaluate your models. For more information about metrics you can use with multi-model endpoints, see [CloudWatch Metrics for Multi-Model Endpoint Deployments](multi-model-endpoint-cloudwatch-metrics.md). You can use the `Average` statistic of the `ModelCacheHit` metric to monitor the ratio of requests where the model is already loaded. You can use the `SampleCount` statistic for the `ModelUnloadingTime` metric to monitor the number of unload requests sent to the container during a time period. If models are unloaded too frequently (an indicator of *thrashing*, where models are being unloaded and loaded again because there is insufficient cache space for the working set of models), consider using a larger instance type with more memory or increasing the number of instances behind the multi-model endpoint. For multi-model endpoints with multiple instances, be aware that a model might be loaded on more than 1 instance. # Create a Multi-Model Endpoint You can use the SageMaker AI console or the AWS SDK for Python (Boto) to create a multi-model endpoint. To create either a CPU or GPU backed endpoint through the console, see the console procedure in the following sections. If you want to create a multi-model endpoint with the AWS SDK for Python (Boto), use either the CPU or GPU procedure in the following sections. The CPU and GPU workflows are similar but have several differences, such as the container requirements. **Topics** + [Create a multi-model endpoint (console)](#create-multi-model-endpoint-console) + [Create a multi-model endpoint using CPUs with the AWS SDK for Python (Boto3)](#create-multi-model-endpoint-sdk-cpu) + [Create a multi-model endpoint using GPUs with the AWS SDK for Python (Boto3)](#create-multi-model-endpoint-sdk-gpu) ## Create a multi-model endpoint (console) You can create both CPU and GPU backed multi-model endpoints through the console. Use the following procedure to create a multi-model endpoint through the SageMaker AI console. **To create a multi-model endpoint (console)** 1. Open the Amazon SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/). 1. Choose **Model**, and then from the **Inference** group, choose **Create model**. 1. For **Model name**, enter a name. 1. For **IAM role**, choose or create an IAM role that has the `AmazonSageMakerFullAccess` IAM policy attached. 1. In the **Container definition** section, for **Provide model artifacts and inference image options**, choose **Use multiple models**. ![\[The section of the Create model page where you can choose Use multiple models.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/mme-create-model-ux-2.PNG) 1. For the **Inference container image**, enter the Amazon ECR path for your desired container image. For GPU models, you must use a container backed by the NVIDIA Triton Inference Server. For a list of container images that work with GPU backed endpoints, see the [NVIDIA Triton Inference Containers (SM support only)](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#nvidia-triton-inference-containers-sm-support-only). For more information about the NVIDIA Triton Inference Server, see [Use Triton Inference Server with SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/triton.html). 1. Choose **Create model**. 1. Deploy your multi-model endpoint as you would a single model endpoint. For instructions, see [Deploy the Model to SageMaker AI Hosting Services](ex1-model-deployment.md#ex1-deploy-model). ## Create a multi-model endpoint using CPUs with the AWS SDK for Python (Boto3) Use the following section to create a multi-model endpoint backed by CPU instances. You create a multi-model endpoint using the Amazon SageMaker AI [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model), [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint_config](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint_config), and [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint) APIs just as you would create a single model endpoint, but with two changes. When defining the model container, you need to pass a new `Mode` parameter value, `MultiModel`. You also need to pass the `ModelDataUrl` field that specifies the prefix in Amazon S3 where the model artifacts are located, instead of the path to a single model artifact, as you would when deploying a single model. For a sample notebook that uses SageMaker AI to deploy multiple XGBoost models to an endpoint, see [Multi-Model Endpoint XGBoost Sample Notebook](https://sagemaker-examples.readthedocs.io/en/latest/advanced_functionality/multi_model_xgboost_home_value/xgboost_multi_model_endpoint_home_value.html). The following procedure outlines the key steps used in that sample to create a CPU backed multi-model endpoint. **To deploy the model (AWS SDK for Python (Boto 3))** 1. Get a container with an image that supports deploying multi-model endpoints. For a list of built-in algorithms and framework containers that support multi-model endpoints, see [Supported algorithms, frameworks, and instances for multi-model endpoints](multi-model-support.md). For this example, we use the [K-Nearest Neighbors (k-NN) Algorithm](k-nearest-neighbors.md) built-in algorithm. We call the [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/v2.html) utility function `image_uris.retrieve()` to get the address for the K-Nearest Neighbors built-in algorithm image. ``` import sagemaker region = sagemaker_session.boto_region_name image = sagemaker.image_uris.retrieve("knn",region=region) container = { 'Image': image, 'ModelDataUrl': 's3:///', 'Mode': 'MultiModel' } ``` 1. Get an AWS SDK for Python (Boto3) SageMaker AI client and create the model that uses this container. ``` import boto3 sagemaker_client = boto3.client('sagemaker') response = sagemaker_client.create_model( ModelName = '', ExecutionRoleArn = role, Containers = [container]) ``` 1. (Optional) If you are using a serial inference pipeline, get the additional container(s) to include in the pipeline, and include it in the `Containers` argument of `CreateModel`: ``` preprocessor_container = { 'Image': '.dkr.ecr..amazonaws.com/:' } multi_model_container = { 'Image': '.dkr.ecr..amazonaws.com/:', 'ModelDataUrl': 's3:///', 'Mode': 'MultiModel' } response = sagemaker_client.create_model( ModelName = '', ExecutionRoleArn = role, Containers = [preprocessor_container, multi_model_container] ) ``` **Note** You can use only one multi-model-enabled endpoint in a serial inference pipeline. 1. (Optional) If your use case does not benefit from model caching, set the value of the `ModelCacheSetting` field of the `MultiModelConfig` parameter to `Disabled`, and include it in the `Container` argument of the call to `create_model`. The value of the `ModelCacheSetting` field is `Enabled` by default. ``` container = { 'Image': image, 'ModelDataUrl': 's3:///', 'Mode': 'MultiModel' 'MultiModelConfig': { // Default value is 'Enabled' 'ModelCacheSetting': 'Disabled' } } response = sagemaker_client.create_model( ModelName = '', ExecutionRoleArn = role, Containers = [container] ) ``` 1. Configure the multi-model endpoint for the model. We recommend configuring your endpoints with at least two instances. This allows SageMaker AI to provide a highly available set of predictions across multiple Availability Zones for the models. ``` response = sagemaker_client.create_endpoint_config( EndpointConfigName = '', ProductionVariants=[ { 'InstanceType': 'ml.m4.xlarge', 'InitialInstanceCount': 2, 'InitialVariantWeight': 1, 'ModelName': '', 'VariantName': 'AllTraffic' } ] ) ``` **Note** You can use only one multi-model-enabled endpoint in a serial inference pipeline. 1. Create the multi-model endpoint using the `EndpointName` and `EndpointConfigName` parameters. ``` response = sagemaker_client.create_endpoint( EndpointName = '', EndpointConfigName = '') ``` ## Create a multi-model endpoint using GPUs with the AWS SDK for Python (Boto3) Use the following section to create a GPU backed multi-model endpoint. You create a multi-model endpoint using the Amazon SageMaker AI [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model), [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint_config](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint_config), and [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint) APIs similarly to creating single model endpoints, but there are several changes. When defining the model container, you need to pass a new `Mode` parameter value, `MultiModel`. You also need to pass the `ModelDataUrl` field that specifies the prefix in Amazon S3 where the model artifacts are located, instead of the path to a single model artifact, as you would when deploying a single model. For GPU backed multi-model endpoints, you also must use a container with the NVIDIA Triton Inference Server that is optimized for running on GPU instances. For a list of container images that work with GPU backed endpoints, see the [NVIDIA Triton Inference Containers (SM support only)](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#nvidia-triton-inference-containers-sm-support-only). For an example notebook that demonstrates how to create a multi-model endpoint backed by GPUs, see [Run mulitple deep learning models on GPUs with Amazon SageMaker AI Multi-model endpoints (MME)](https://github.com/aws/amazon-sagemaker-examples/blob/main/multi-model-endpoints/mme-on-gpu/cv/resnet50_mme_with_gpu.ipynb). The following procedure outlines the key steps to create a GPU backed multi-model endpoint. **To deploy the model (AWS SDK for Python (Boto 3))** 1. Define the container image. To create a multi-model endpoint with GPU support for ResNet models, define the container to use the [NVIDIA Triton Server image](https://docs.aws.amazon.com/sagemaker/latest/dg/triton.html). This container supports multi-model endpoints and is optimized for running on GPU instances. We call the [SageMaker AI Python SDK](https://sagemaker.readthedocs.io/en/stable/v2.html) utility function `image_uris.retrieve()` to get the address for the image. For example: ``` import sagemaker region = sagemaker_session.boto_region_name // Find the sagemaker-tritonserver image at // https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-triton/resnet50/triton_resnet50.ipynb // Find available tags at https://github.com/aws/deep-learning-containers/blob/master/available_images.md#nvidia-triton-inference-containers-sm-support-only image = ".dkr.ecr..amazonaws.com/sagemaker-tritonserver:".format( account_id=account_id_map[region], region=region ) container = { 'Image': image, 'ModelDataUrl': 's3:///', 'Mode': 'MultiModel', "Environment": {"SAGEMAKER_TRITON_DEFAULT_MODEL_NAME": "resnet"}, } ``` 1. Get an AWS SDK for Python (Boto3) SageMaker AI client and create the model that uses this container. ``` import boto3 sagemaker_client = boto3.client('sagemaker') response = sagemaker_client.create_model( ModelName = '', ExecutionRoleArn = role, Containers = [container]) ``` 1. (Optional) If you are using a serial inference pipeline, get the additional container(s) to include in the pipeline, and include it in the `Containers` argument of `CreateModel`: ``` preprocessor_container = { 'Image': '.dkr.ecr..amazonaws.com/:' } multi_model_container = { 'Image': '.dkr.ecr..amazonaws.com/:', 'ModelDataUrl': 's3:///', 'Mode': 'MultiModel' } response = sagemaker_client.create_model( ModelName = '', ExecutionRoleArn = role, Containers = [preprocessor_container, multi_model_container] ) ``` **Note** You can use only one multi-model-enabled endpoint in a serial inference pipeline. 1. (Optional) If your use case does not benefit from model caching, set the value of the `ModelCacheSetting` field of the `MultiModelConfig` parameter to `Disabled`, and include it in the `Container` argument of the call to `create_model`. The value of the `ModelCacheSetting` field is `Enabled` by default. ``` container = { 'Image': image, 'ModelDataUrl': 's3:///', 'Mode': 'MultiModel' 'MultiModelConfig': { // Default value is 'Enabled' 'ModelCacheSetting': 'Disabled' } } response = sagemaker_client.create_model( ModelName = '', ExecutionRoleArn = role, Containers = [container] ) ``` 1. Configure the multi-model endpoint with GPU backed instances for the model. We recommend configuring your endpoints with more than one instance to allow for high availability and higher cache hits. ``` response = sagemaker_client.create_endpoint_config( EndpointConfigName = '', ProductionVariants=[ { 'InstanceType': 'ml.g4dn.4xlarge', 'InitialInstanceCount': 2, 'InitialVariantWeight': 1, 'ModelName': '', 'VariantName': 'AllTraffic' } ] ) ``` 1. Create the multi-model endpoint using the `EndpointName` and `EndpointConfigName` parameters. ``` response = sagemaker_client.create_endpoint( EndpointName = '', EndpointConfigName = '') ``` # Invoke a Multi-Model Endpoint To invoke a multi-model endpoint, use the [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-runtime.html#SageMakerRuntime.Client.invoke_endpoint](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-runtime.html#SageMakerRuntime.Client.invoke_endpoint) from the SageMaker AI Runtime just as you would invoke a single model endpoint, with one change. Pass a new `TargetModel` parameter that specifies which of the models at the endpoint to target. The SageMaker AI Runtime `InvokeEndpoint` request supports `X-Amzn-SageMaker-Target-Model` as a new header that takes the relative path of the model specified for invocation. The SageMaker AI system constructs the absolute path of the model by combining the prefix that is provided as part of the `CreateModel` API call with the relative path of the model. The following procedures are the same for both CPU and GPU-backed multi-model endpoints. ------ #### [ AWS SDK for Python (Boto 3) ] The following example prediction request uses the [AWS SDK for Python (Boto 3)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-runtime.html) in the sample notebook. ``` response = runtime_sagemaker_client.invoke_endpoint( EndpointName = "", ContentType = "text/csv", TargetModel = ".tar.gz", Body = body) ``` ------ #### [ AWS CLI ] The following example shows how to make a CSV request with two rows using the AWS Command Line Interface (AWS CLI): ``` aws sagemaker-runtime invoke-endpoint \ --endpoint-name "" \ --body "1.0,2.0,5.0"$'\n'"2.0,3.0,4.0" \ --content-type "text/csv" \ --target-model ".tar.gz" output_file.txt ``` An `output_file.txt` with information about your inference requests is made if the inference was successful. For more examples on how to make predictions with the AWS CLI, see [Making predictions with the AWS CLI](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/deploying_tensorflow_serving.html#making-predictions-with-the-aws-cli) in the SageMaker Python SDK documentation. ------ The multi-model endpoint dynamically loads target models as needed. You can observe this when running the [MME Sample Notebook](https://sagemaker-examples.readthedocs.io/en/latest/advanced_functionality/multi_model_xgboost_home_value/xgboost_multi_model_endpoint_home_value.html) as it iterates through random invocations against multiple target models hosted behind a single endpoint. The first request against a given model takes longer because the model has to be downloaded from Amazon Simple Storage Service (Amazon S3) and loaded into memory. This is called a *cold start*, and it is expected on multi-model endpoints to optimize for better price performance for customers. Subsequent calls finish faster because there's no additional overhead after the model has loaded. **Note** For GPU backed instances, the HTTP response code with 507 from the GPU container indicates a lack of memory or other resources. This causes unused models to be unloaded from the container in order to load more frequently used models. ## Retry Requests on ModelNotReadyException Errors The first time you call `invoke_endpoint` for a model, the model is downloaded from Amazon Simple Storage Service and loaded into the inference container. This makes the first call take longer to return. Subsequent calls to the same model finish faster, because the model is already loaded. SageMaker AI returns a response for a call to `invoke_endpoint` within 60 seconds. Some models are too large to download within 60 seconds. If the model does not finish loading before the 60 second timeout limit, the request to `invoke_endpoint` returns with the error code `ModelNotReadyException`, and the model continues to download and load into the inference container for up to 360 seconds. If you get a `ModelNotReadyException` error code for an `invoke_endpoint` request, retry the request. By default, the AWS SDKs for Python (Boto 3) (using [Legacy retry mode](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#legacy-retry-mode)) and Java retry `invoke_endpoint` requests that result in `ModelNotReadyException` errors. You can configure the retry strategy to continue retrying the request for up to 360 seconds. If you expect your model to take longer than 60 seconds to download and load into the container, set the SDK socket timeout to 70 seconds. For more information about configuring the retry strategy for the AWS SDK for Python (Boto3), see [Configuring a retry mode](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#configuring-a-retry-mode). The following code shows an example that configures the retry strategy to retry calls to `invoke_endpoint` for up to 180 seconds. ``` import boto3 from botocore.config import Config # This example retry strategy sets the retry attempts to 2. # With this setting, the request can attempt to download and/or load the model # for upto 180 seconds: 1 orginal request (60 seconds) + 2 retries (120 seconds) config = Config( read_timeout=70, retries={ 'max_attempts': 2 # This value can be adjusted to 5 to go up to the 360s max timeout } ) runtime_sagemaker_client = boto3.client('sagemaker-runtime', config=config) ``` # Add or Remove Models You can deploy additional models to a multi-model endpoint and invoke them through that endpoint immediately. When adding a new model, you don't need to update or bring down the endpoint, so you avoid the cost of creating and running a separate endpoint for each new model. The process for adding and removing models is the same for CPU and GPU-backed multi-model endpoints. SageMaker AI unloads unused models from the container when the instance is reaching memory capacity and more models need to be downloaded into the container. SageMaker AI also deletes unused model artifacts from the instance storage volume when the volume is reaching capacity and new models need to be downloaded. The first invocation to a newly added model takes longer because the endpoint takes time to download the model from S3 to the container's memory in instance hosting the endpoint With the endpoint already running, copy a new set of model artifacts to the Amazon S3 location there you store your models. ``` # Add an AdditionalModel to the endpoint and exercise it aws s3 cp AdditionalModel.tar.gz s3://amzn-s3-demo-bucket/path/to/artifacts/ ``` **Important** To update a model, proceed as you would when adding a new model. Use a new and unique name. Don't overwrite model artifacts in Amazon S3 because the old version of the model might still be loaded in the containers or on the storage volume of the instances on the endpoint. Invocations to the new model could then invoke the old version of the model. Client applications can request predictions from the additional target model as soon as it is stored in S3. ``` response = runtime_sagemaker_client.invoke_endpoint( EndpointName='', ContentType='text/csv', TargetModel='AdditionalModel.tar.gz', Body=body) ``` To delete a model from a multi-model endpoint, stop invoking the model from the clients and remove it from the S3 location where model artifacts are stored. # Build Your Own Container for SageMaker AI Multi-Model Endpoints Refer to the following sections for bringing your own container and dependencies to multi-model endpoints. **Topics** + [Bring your own dependencies for multi-model endpoints on CPU backed instances](#build-multi-model-container-cpu) + [Bring your own dependencies for multi-model endpoints on GPU backed instances](#build-multi-model-container-gpu) + [Use the SageMaker AI Inference Toolkit](#multi-model-inference-toolkit) + [Custom Containers Contract for Multi-Model Endpoints](mms-container-apis.md) ## Bring your own dependencies for multi-model endpoints on CPU backed instances If none of the pre-built container images serve your needs, you can build your own container for use with CPU backed multi-model endpoints. Custom Amazon Elastic Container Registry (Amazon ECR) images deployed in Amazon SageMaker AI are expected to adhere to the basic contract described in [Custom Inference Code with Hosting Services](your-algorithms-inference-code.md) that govern how SageMaker AI interacts with a Docker container that runs your own inference code. For a container to be capable of loading and serving multiple models concurrently, there are additional APIs and behaviors that must be followed. This additional contract includes new APIs to load, list, get, and unload models, and a different API to invoke models. There are also different behaviors for error scenarios that the APIs need to abide by. To indicate that the container complies with the additional requirements, you can add the following command to your Docker file: ``` LABEL com.amazonaws.sagemaker.capabilities.multi-models=true ``` SageMaker AI also injects an environment variable into the container ``` SAGEMAKER_MULTI_MODEL=true ``` If you are creating a multi-model endpoint for a serial inference pipline, your Docker file must have the required labels for both multi-models and serial inference pipelines. For more information about serial information pipelines, see [Run Real-time Predictions with an Inference Pipeline](inference-pipeline-real-time.md). To help you implement these requirements for a custom container, two libraries are available: + [Multi Model Server](https://github.com/awslabs/multi-model-server) is an open source framework for serving machine learning models that can be installed in containers to provide the front end that fulfills the requirements for the new multi-model endpoint container APIs. It provides the HTTP front end and model management capabilities required by multi-model endpoints to host multiple models within a single container, load models into and unload models out of the container dynamically, and performs inference on a specified loaded model. It also provides a pluggable backend that supports a pluggable custom backend handler where you can implement your own algorithm. + [SageMaker AI Inference Toolkit](https://github.com/aws/sagemaker-inference-toolkit) is a library that bootstraps Multi Model Server with a configuration and settings that make it compatible with SageMaker AI multi-model endpoints. It also allows you to tweak important performance parameters, such as the number of workers per model, depending on the needs of your scenario. ## Bring your own dependencies for multi-model endpoints on GPU backed instances The bring your own container (BYOC) capability on multi-model endpoints with GPU backed instances is not currently supported by the Multi Model Server and SageMaker AI Inference Toolkit libraries. For creating multi-model endpoints with GPU backed instances, you can use the SageMaker AI supported [NVIDIA Triton Inference Server](https://docs.aws.amazon.com/sagemaker/latest/dg/triton.html). with the [NVIDIA Triton Inference Containers](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#nvidia-triton-inference-containers-sm-support-only). To bring your own dependencies, you can build your own container with the SageMaker AI supported [NVIDIA Triton Inference Server](https://docs.aws.amazon.com/sagemaker/latest/dg/triton.html) as the base image to your Docker file: ``` FROM 301217895009.dkr.ecr.us-west-2.amazonaws.com/sagemaker-tritonserver:22.07-py3 ``` **Important** Containers with the Triton Inference Server are the only supported containers you can use for GPU backed multi-model endpoints. ## Use the SageMaker AI Inference Toolkit **Note** The SageMaker AI Inference Toolkit is only supported for CPU backed multi-model endpoints. The SageMaker AI Inference Toolkit is not currently not supported for GPU backed multi-model endpoints. Pre-built containers that support multi-model endpoints are listed in [Supported algorithms, frameworks, and instances for multi-model endpoints](multi-model-support.md). If you want to use any other framework or algorithm, you need to build a container. The easiest way to do this is to use the [SageMaker AI Inference Toolkit](https://github.com/aws/sagemaker-inference-toolkit) to extend an existing pre-built container. The SageMaker AI inference toolkit is an implementation for the multi-model server (MMS) that creates endpoints that can be deployed in SageMaker AI. For a sample notebook that shows how to set up and deploy a custom container that supports multi-model endpoints in SageMaker AI, see the [Multi-Model Endpoint BYOC Sample Notebook](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality/multi_model_bring_your_own). **Note** The SageMaker AI inference toolkit supports only Python model handlers. If you want to implement your handler in any other language, you must build your own container that implements the additional multi-model endpoint APIs. For information, see [Custom Containers Contract for Multi-Model Endpoints](mms-container-apis.md). **To extend a container by using the SageMaker AI inference toolkit** 1. Create a model handler. MMS expects a model handler, which is a Python file that implements functions to pre-process, get preditions from the model, and process the output in a model handler. For an example of a model handler, see [model\$1handler.py](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/multi_model_bring_your_own/container/model_handler.py) from the sample notebook. 1. Import the inference toolkit and use its `model_server.start_model_server` function to start MMS. The following example is from the `dockerd-entrypoint.py` file from the sample notebook. Notice that the call to `model_server.start_model_server` passes the model handler described in the previous step: ``` import subprocess import sys import shlex import os from retrying import retry from subprocess import CalledProcessError from sagemaker_inference import model_server def _retry_if_error(exception): return isinstance(exception, CalledProcessError or OSError) @retry(stop_max_delay=1000 * 50, retry_on_exception=_retry_if_error) def _start_mms(): # by default the number of workers per model is 1, but we can configure it through the # environment variable below if desired. # os.environ['SAGEMAKER_MODEL_SERVER_WORKERS'] = '2' model_server.start_model_server(handler_service='/home/model-server/model_handler.py:handle') def main(): if sys.argv[1] == 'serve': _start_mms() else: subprocess.check_call(shlex.split(' '.join(sys.argv[1:]))) # prevent docker exit subprocess.call(['tail', '-f', '/dev/null']) main() ``` 1. In your `Dockerfile`, copy the model handler from the first step and specify the Python file from the previous step as the entrypoint in your `Dockerfile`. The following lines are from the [Dockerfile](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/multi_model_bring_your_own/container/Dockerfile) used in the sample notebook: ``` # Copy the default custom service file to handle incoming data and inference requests COPY model_handler.py /home/model-server/model_handler.py # Define an entrypoint script for the docker image ENTRYPOINT ["python", "/usr/local/bin/dockerd-entrypoint.py"] ``` 1. Build and register your container. The following shell script from the sample notebook builds the container and uploads it to an Amazon Elastic Container Registry repository in your AWS account: ``` %%sh # The name of our algorithm algorithm_name=demo-sagemaker-multimodel cd container account=$(aws sts get-caller-identity --query Account --output text) # Get the region defined in the current configuration (default to us-west-2 if none defined) region=$(aws configure get region) region=${region:-us-west-2} fullname="${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest" # If the repository doesn't exist in ECR, create it. aws ecr describe-repositories --repository-names "${algorithm_name}" > /dev/null 2>&1 if [ $? -ne 0 ] then aws ecr create-repository --repository-name "${algorithm_name}" > /dev/null fi # Get the login command from ECR and execute it directly $(aws ecr get-login --region ${region} --no-include-email) # Build the docker image locally with the image name and then push it to ECR # with the full name. docker build -q -t ${algorithm_name} . docker tag ${algorithm_name} ${fullname} docker push ${fullname} ``` You can now use this container to deploy multi-model endpoints in SageMaker AI. **Topics** + [Bring your own dependencies for multi-model endpoints on CPU backed instances](#build-multi-model-container-cpu) + [Bring your own dependencies for multi-model endpoints on GPU backed instances](#build-multi-model-container-gpu) + [Use the SageMaker AI Inference Toolkit](#multi-model-inference-toolkit) + [Custom Containers Contract for Multi-Model Endpoints](mms-container-apis.md) # Custom Containers Contract for Multi-Model Endpoints To handle multiple models, your container must support a set of APIs that enable Amazon SageMaker AI to communicate with the container for loading, listing, getting, and unloading models as required. The `model_name` is used in the new set of APIs as the key input parameter. The customer container is expected to keep track of the loaded models using `model_name` as the mapping key. Also, the `model_name` is an opaque identifier and is not necessarily the value of the `TargetModel` parameter passed into the `InvokeEndpoint` API. The original `TargetModel` value in the `InvokeEndpoint` request is passed to container in the APIs as a `X-Amzn-SageMaker-Target-Model` header that can be used for logging purposes. **Note** Multi-model endpoints for GPU backed instances are currently supported only with SageMaker AI's [NVIDIA Triton Inference Server container](https://docs.aws.amazon.com/sagemaker/latest/dg/triton.html). This container already implements the contract defined below. Customers can directly use this container with their multi-model GPU endpoints, without any additional work. You can configure the following APIs on your containers for CPU backed multi-model endpoints. **Topics** + [Load Model API](#multi-model-api-load-model) + [List Model API](#multi-model-api-list-model) + [Get Model API](#multi-model-api-get-model) + [Unload Model API](#multi-model-api-unload-model) + [Invoke Model API](#multi-model-api-invoke-model) ## Load Model API Instructs the container to load a particular model present in the `url` field of the body into the memory of the customer container and to keep track of it with the assigned `model_name`. After a model is loaded, the container should be ready to serve inference requests using this `model_name`. ``` POST /models HTTP/1.1 Content-Type: application/json Accept: application/json { "model_name" : "{model_name}", "url" : "/opt/ml/models/{model_name}/model", } ``` **Note** If `model_name` is already loaded, this API should return 409. Any time a model cannot be loaded due to lack of memory or to any other resource, this API should return a 507 HTTP status code to SageMaker AI, which then initiates unloading unused models to reclaim. ## List Model API Returns the list of models loaded into the memory of the customer container. ``` GET /models HTTP/1.1 Accept: application/json Response = { "models": [ { "modelName" : "{model_name}", "modelUrl" : "/opt/ml/models/{model_name}/model", }, { "modelName" : "{model_name}", "modelUrl" : "/opt/ml/models/{model_name}/model", }, .... ] } ``` This API also supports pagination. ``` GET /models HTTP/1.1 Accept: application/json Response = { "models": [ { "modelName" : "{model_name}", "modelUrl" : "/opt/ml/models/{model_name}/model", }, { "modelName" : "{model_name}", "modelUrl" : "/opt/ml/models/{model_name}/model", }, .... ] } ``` SageMaker AI can initially call the List Models API without providing a value for `next_page_token`. If a `nextPageToken` field is returned as part of the response, it will be provided as the value for `next_page_token` in a subsequent List Models call. If a `nextPageToken` is not returned, it means that there are no more models to return. ## Get Model API This is a simple read API on the `model_name` entity. ``` GET /models/{model_name} HTTP/1.1 Accept: application/json { "modelName" : "{model_name}", "modelUrl" : "/opt/ml/models/{model_name}/model", } ``` **Note** If `model_name` is not loaded, this API should return 404. ## Unload Model API Instructs the SageMaker AI platform to instruct the customer container to unload a model from memory. This initiates the eviction of a candidate model as determined by the platform when starting the process of loading a new model. The resources provisioned to `model_name` should be reclaimed by the container when this API returns a response. ``` DELETE /models/{model_name} ``` **Note** If `model_name` is not loaded, this API should return 404. ## Invoke Model API Makes a prediction request from the particular `model_name` supplied. The SageMaker AI Runtime `InvokeEndpoint` request supports `X-Amzn-SageMaker-Target-Model` as a new header that takes the relative path of the model specified for invocation. The SageMaker AI system constructs the absolute path of the model by combining the prefix that is provided as part of the `CreateModel` API call with the relative path of the model. ``` POST /models/{model_name}/invoke HTTP/1.1 Content-Type: ContentType Accept: Accept X-Amzn-SageMaker-Custom-Attributes: CustomAttributes X-Amzn-SageMaker-Target-Model: [relativePath]/{artifactName}.tar.gz ``` **Note** If `model_name` is not loaded, this API should return 404. Additionally, on GPU instances, if `InvokeEndpoint` fails due to a lack of memory or other resources, this API should return a 507 HTTP status code to SageMaker AI, which then initiates unloading unused models to reclaim. # Multi-Model Endpoint Security Models and data in a multi-model endpoint are co-located on instance storage volume and in container memory. All instances for Amazon SageMaker AI endpoints run on a single tenant container that you own. Only your models can run on your multi-model endpoint. It's your responsibility to manage the mapping of requests to models and to provide access for users to the correct target models. SageMaker AI uses [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) to provide IAM identity-based policies that you use to specify allowed or denied actions and resources and the conditions under which actions are allowed or denied. By default, an IAM principal with [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html) permissions on a multi-model endpoint can invoke any model at the address of the S3 prefix defined in the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) operation, provided that the IAM Execution Role defined in operation has permissions to download the model. If you need to restrict [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html) access to a limited set of models in S3, you can do one of the following: + Restrict `InvokeEndpont` calls to specific models hosted at the endpoint by using the `sagemaker:TargetModel` IAM condition key. For example, the following policy allows `InvokeEndpont` requests only when the value of the `TargetModel` field matches one of the specified regular expressions: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "sagemaker:InvokeEndpoint" ], "Effect": "Allow", "Resource": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/endpoint_name", "Condition": { "StringLike": { "sagemaker:TargetModel": ["company_a/*", "common/*"] } } } ] } ``` ------ For information about SageMaker AI condition keys, see [Condition Keys for Amazon SageMaker AI](https://docs.aws.amazon.com/IAM/latest/UserGuide/list_amazonsagemaker.html#amazonsagemaker-policy-keys) in the *AWS Identity and Access Management User Guide*. + Create multi-model endpoints with more restrictive S3 prefixes. For more information about how SageMaker AI uses roles to manage access to endpoints and perform operations on your behalf, see [How to use SageMaker AI execution roles](sagemaker-roles.md). Your customers might also have certain data isolation requirements dictated by their own compliance requirements that can be satisfied using IAM identities. # CloudWatch Metrics for Multi-Model Endpoint Deployments Amazon SageMaker AI provides metrics for endpoints so you can monitor the cache hit rate, the number of models loaded and the model wait times for loading, downloading, and uploading at a multi-model endpoint. Some of the metrics are different for CPU and GPU backed multi-model endpoints, so the following sections describe the Amazon CloudWatch metrics that you can use for each type of multi-model endpoint. For more information about the metrics, see **Multi-Model Endpoint Model Loading Metrics** and **Multi-Model Endpoint Model Instance Metrics** in [Amazon SageMaker AI metrics in Amazon CloudWatch](monitoring-cloudwatch.md). Per-model metrics aren't supported. ## CloudWatch metrics for CPU backed multi-model endpoints You can monitor the following metrics on CPU backed multi-model endpoints. The `AWS/SageMaker` namespace includes the following model loading metrics from calls to [ InvokeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html). Metrics are available at a 1-minute frequency. For information about how long CloudWatch metrics are retained for, see [GetMetricStatistics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html) in the *Amazon CloudWatch API Reference*. **Multi-Model Endpoint Model Loading Metrics** | Metric | Description | | --- | --- | | ModelLoadingWaitTime | The interval of time that an invocation request has waited for the target model to be downloaded, or loaded, or both in order to perform inference. Units: Microseconds Valid statistics: Average, Sum, Min, Max, Sample Count | | ModelUnloadingTime | The interval of time that it took to unload the model through the container's `UnloadModel` API call. Units: Microseconds Valid statistics: Average, Sum, Min, Max, Sample Count | | ModelDownloadingTime | The interval of time that it took to download the model from Amazon Simple Storage Service (Amazon S3). Units: Microseconds Valid statistics: Average, Sum, Min, Max, Sample Count | | ModelLoadingTime | The interval of time that it took to load the model through the container's `LoadModel` API call. Units: Microseconds Valid statistics: Average, Sum, Min, Max, Sample Count | | ModelCacheHit | The number of `InvokeEndpoint` requests sent to the multi-model endpoint for which the model was already loaded. The Average statistic shows the ratio of requests for which the model was already loaded. Units: None Valid statistics: Average, Sum, Sample Count | **Dimensions for Multi-Model Endpoint Model Loading Metrics** | Dimension | Description | | --- | --- | | EndpointName, VariantName | Filters endpoint invocation metrics for a `ProductionVariant` of the specified endpoint and variant. | The `/aws/sagemaker/Endpoints` namespaces include the following instance metrics from calls to [ InvokeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html). Metrics are available at a 1-minute frequency. For information about how long CloudWatch metrics are retained for, see [GetMetricStatistics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html) in the *Amazon CloudWatch API Reference*. **Multi-Model Endpoint Model Instance Metrics** | Metric | Description | | --- | --- | | LoadedModelCount | The number of models loaded in the containers of the multi-model endpoint. This metric is emitted per instance. The Average statistic with a period of 1 minute tells you the average number of models loaded per instance. The Sum statistic tells you the total number of models loaded across all instances in the endpoint. The models that this metric tracks are not necessarily unique because a model might be loaded in multiple containers at the endpoint. Units: None Valid statistics: Average, Sum, Min, Max, Sample Count | | CPUUtilization | The sum of each individual CPU core's utilization. The CPU utilization of each core range is 0–100. For example, if there are four CPUs, the `CPUUtilization` range is 0%–400%. For endpoint variants, the value is the sum of the CPU utilization of the primary and supplementary containers on the instance. Units: Percent | | MemoryUtilization | The percentage of memory that is used by the containers on an instance. This value range is 0%–100%. For endpoint variants, the value is the sum of the memory utilization of the primary and supplementary containers on the instance. Units: Percent | | DiskUtilization | The percentage of disk space used by the containers on an instance. This value range is 0%–100%. For endpoint variants, the value is the sum of the disk space utilization of the primary and supplementary containers on the instance. Units: Percent | ## CloudWatch metrics for GPU multi-model endpoint deployments You can monitor the following metrics on GPU backed multi-model endpoints. The `AWS/SageMaker` namespace includes the following model loading metrics from calls to [ InvokeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html). Metrics are available at a 1-minute frequency. For information about how long CloudWatch metrics are retained for, see [GetMetricStatistics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html) in the *Amazon CloudWatch API Reference*. **Multi-Model Endpoint Model Loading Metrics** | Metric | Description | | --- | --- | | ModelLoadingWaitTime | The interval of time that an invocation request has waited for the target model to be downloaded, or loaded, or both in order to perform inference. Units: Microseconds Valid statistics: Average, Sum, Min, Max, Sample Count | | ModelUnloadingTime | The interval of time that it took to unload the model through the container's `UnloadModel` API call. Units: Microseconds Valid statistics: Average, Sum, Min, Max, Sample Count | | ModelDownloadingTime | The interval of time that it took to download the model from Amazon Simple Storage Service (Amazon S3). Units: Microseconds Valid statistics: Average, Sum, Min, Max, Sample Count | | ModelLoadingTime | The interval of time that it took to load the model through the container's `LoadModel` API call. Units: Microseconds Valid statistics: Average, Sum, Min, Max, Sample Count | | ModelCacheHit | The number of `InvokeEndpoint` requests sent to the multi-model endpoint for which the model was already loaded. The Average statistic shows the ratio of requests for which the model was already loaded. Units: None Valid statistics: Average, Sum, Sample Count | **Dimensions for Multi-Model Endpoint Model Loading Metrics** | Dimension | Description | | --- | --- | | EndpointName, VariantName | Filters endpoint invocation metrics for a `ProductionVariant` of the specified endpoint and variant. | The `/aws/sagemaker/Endpoints` namespaces include the following instance metrics from calls to [ InvokeEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html). Metrics are available at a 1-minute frequency. For information about how long CloudWatch metrics are retained for, see [GetMetricStatistics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html) in the *Amazon CloudWatch API Reference*. **Multi-Model Endpoint Model Instance Metrics** | Metric | Description | | --- | --- | | LoadedModelCount | The number of models loaded in the containers of the multi-model endpoint. This metric is emitted per instance. The Average statistic with a period of 1 minute tells you the average number of models loaded per instance. The Sum statistic tells you the total number of models loaded across all instances in the endpoint. The models that this metric tracks are not necessarily unique because a model might be loaded in multiple containers at the endpoint. Units: None Valid statistics: Average, Sum, Min, Max, Sample Count | | CPUUtilization | The sum of each individual CPU core's utilization. The CPU utilization of each core range is 0‐100. For example, if there are four CPUs, the `CPUUtilization` range is 0%–400%. For endpoint variants, the value is the sum of the CPU utilization of the primary and supplementary containers on the instance. Units: Percent | | MemoryUtilization | The percentage of memory that is used by the containers on an instance. This value range is 0%‐100%. For endpoint variants, the value is the sum of the memory utilization of the primary and supplementary containers on the instance. Units: Percent | | GPUUtilization | The percentage of GPU units that are used by the containers on an instance. The value can range betweenrange is 0‐100 and is multiplied by the number of GPUs. For example, if there are four GPUs, the `GPUUtilization` range is 0%–400%. For endpoint variants, the value is the sum of the GPU utilization of the primary and supplementary containers on the instance. Units: Percent | | GPUMemoryUtilization | The percentage of GPU memory used by the containers on an instance. The value range is 0‐100 and is multiplied by the number of GPUs. For example, if there are four GPUs, the `GPUMemoryUtilization` range is 0%‐400%. For endpoint variants, the value is the sum of the GPU memory utilization of the primary and supplementary containers on the instance. Units: Percent | | DiskUtilization | The percentage of disk space used by the containers on an instance. This value range is 0%–100%. For endpoint variants, the value is the sum of the disk space utilization of the primary and supplementary containers on the instance. Units: Percent | # Set SageMaker AI multi-model endpoint model caching behavior By default, multi-model endpoints cache frequently used models in memory (CPU or GPU, depending on whether you have CPU or GPU backed instances) and on disk to provide low latency inference. The cached models are unloaded and/or deleted from disk only when a container runs out of memory or disk space to accommodate a newly targeted model. You can change the caching behavior of a multi-model endpoint and explicitly enable or disable model caching by setting the parameter `ModelCacheSetting` when you call [create\$1model](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model). We recommend setting the value of the `ModelCacheSetting` parameter to `Disabled` for use cases that do not benefit from model caching. For example, when a large number of models need to be served from the endpoint but each model is invoked only once (or very infrequently). For such use cases, setting the value of the `ModelCacheSetting` parameter to `Disabled` allows higher transactions per second (TPS) for `invoke_endpoint` requests compared to the default caching mode. Higher TPS in these use cases is because SageMaker AI does the following after the `invoke_endpoint` request: + Asynchronously unloads the model from memory and deletes it from disk immediately after it is invoked. + Provides higher concurrency for downloading and loading models in the inference container. For both CPU and GPU backed endpoints, the concurrency is a factor of the number of the vCPUs of the container instance. For guidelines on choosing a SageMaker AI ML instance type for a multi-model endpoint, see [Instance recommendations for multi-model endpoint deployments](multi-model-endpoint-instance.md). # Set Auto Scaling Policies for Multi-Model Endpoint Deployments SageMaker AI multi-model endpoints fully support automatic scaling, which manages replicas of models to ensure models scale based on traffic patterns. We recommend that you configure your multi-model endpoint and the size of your instances based on [Instance recommendations for multi-model endpoint deployments](multi-model-endpoint-instance.md) and also set up instance based auto scaling for your endpoint. The invocation rates used to trigger an auto-scale event are based on the aggregate set of predictions across the full set of models served by the endpoint. For additional details on setting up endpoint auto scaling, see [Automatically Scale Amazon SageMaker AI Models](https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling.html). You can set up auto scaling policies with predefined and custom metrics on both CPU and GPU backed multi-model endpoints. **Note** SageMaker AI multi-model endpoint metrics are available at one-minute granularity. ## Define a scaling policy To specify the metrics and target values for a scaling policy, you can configure a target-tracking scaling policy. You can use either a predefined metric or a custom metric. Scaling policy configuration is represented by a JSON block. You save your scaling policy configuration as a JSON block in a text file. You use that text file when invoking the AWS CLI or the Application Auto Scaling API. For more information about policy configuration syntax, see `[TargetTrackingScalingPolicyConfiguration](https://docs.aws.amazon.com/autoscaling/application/APIReference/API_TargetTrackingScalingPolicyConfiguration.html)` in the *Application Auto Scaling API Reference*. The following options are available for defining a target-tracking scaling policy configuration. ### Use a predefined metric To quickly define a target-tracking scaling policy for a variant, use the `SageMakerVariantInvocationsPerInstance` predefined metric. `SageMakerVariantInvocationsPerInstance` is the average number of times per minute that each instance for a variant is invoked. We strongly recommend using this metric. To use a predefined metric in a scaling policy, create a target tracking configuration for your policy. In the target tracking configuration, include a `PredefinedMetricSpecification` for the predefined metric and a `TargetValue` for the target value of that metric. The following example is a typical policy configuration for target-tracking scaling for a variant. In this configuration, we use the `SageMakerVariantInvocationsPerInstance` predefined metric to adjust the number of variant instances so that each instance has an `InvocationsPerInstance` metric of `70`. ``` {"TargetValue": 70.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "InvocationsPerInstance" } } ``` **Note** We recommend that you use `InvocationsPerInstance` while using multi-model endpoints. The `TargetValue` for this metric depends on your application’s latency requirements. We also recommend that you load test your endpoints to set up suitable scaling parameter values. To learn more about load testing and setting up autoscaling for your endpoints, see the blog [Configuring autoscaling inference endpoints in Amazon SageMaker AI](https://aws.amazon.com/blogs/machine-learning/configuring-autoscaling-inference-endpoints-in-amazon-sagemaker/). ### Use a custom metric If you need to define a target-tracking scaling policy that meets your custom requirements, define a custom metric. You can define a custom metric based on any production variant metric that changes in proportion to scaling. Not all SageMaker AI metrics work for target tracking. The metric must be a valid utilization metric, and it must describe how busy an instance is. The value of the metric must increase or decrease in inverse proportion to the number of variant instances. That is, the value of the metric should decrease when the number of instances increases. **Important** Before deploying automatic scaling in production, you must test automatic scaling with your custom metric. #### Example custom metric for a CPU backed multi-model endpoint The following example is a target-tracking configuration for a scaling policy. In this configuration, for a model named `my-model`, a custom metric of `CPUUtilization` adjusts the instance count on the endpoint based on an average CPU utilization of 50% across all instances. ``` {"TargetValue": 50, "CustomizedMetricSpecification": {"MetricName": "CPUUtilization", "Namespace": "/aws/sagemaker/Endpoints", "Dimensions": [ {"Name": "EndpointName", "Value": "my-endpoint" }, {"Name": "ModelName","Value": "my-model"} ], "Statistic": "Average", "Unit": "Percent" } } ``` #### Example custom metric for a GPU backed multi-model endpoint The following example is a target-tracking configuration for a scaling policy. In this configuration, for a model named `my-model`, a custom metric of `GPUUtilization` adjusts the instance count on the endpoint based on an average GPU utilization of 50% across all instances. ``` {"TargetValue": 50, "CustomizedMetricSpecification": {"MetricName": "GPUUtilization", "Namespace": "/aws/sagemaker/Endpoints", "Dimensions": [ {"Name": "EndpointName", "Value": "my-endpoint" }, {"Name": "ModelName","Value": "my-model"} ], "Statistic": "Average", "Unit": "Percent" } } ``` ## Add a cooldown period To add a cooldown period for scaling out your endpoint, specify a value, in seconds, for `ScaleOutCooldown`. Similarly, to add a cooldown period for scaling in your model, add a value, in seconds, for `ScaleInCooldown`. For more information about `ScaleInCooldown` and `ScaleOutCooldown`, see `[TargetTrackingScalingPolicyConfiguration](https://docs.aws.amazon.com/autoscaling/application/APIReference/API_TargetTrackingScalingPolicyConfiguration.html)` in the *Application Auto Scaling API Reference*. The following is an example target-tracking configuration for a scaling policy. In this configuration, the `SageMakerVariantInvocationsPerInstance` predefined metric is used to adjust scaling based on an average of `70` across all instances of that variant. The configuration provides a scale-in cooldown period of 10 minutes and a scale-out cooldown period of 5 minutes. ``` {"TargetValue": 70.0, "PredefinedMetricSpecification": {"PredefinedMetricType": "SageMakerVariantInvocationsPerInstance" }, "ScaleInCooldown": 600, "ScaleOutCooldown": 300 } ``` # Multi-container endpoints SageMaker AI multi-container endpoints enable customers to deploy multiple containers, that use different models or frameworks, on a single SageMaker AI endpoint. The containers can be run in a sequence as an inference pipeline, or each container can be accessed individually by using direct invocation to improve endpoint utilization and optimize costs. For information about invoking the containers in a multi-container endpoint in sequence, see [Inference pipelines in Amazon SageMaker AI](inference-pipelines.md). For information about invoking a specific container in a multi-container endpoint, see [Invoke a multi-container endpoint with direct invocation](multi-container-direct.md) **Topics** + [Create a multi-container endpoint (Boto 3)](multi-container-create.md) + [Update a multi-container endpoint](multi-container-update.md) + [Invoke a multi-container endpoint with direct invocation](multi-container-direct.md) + [Security with multi-container endpoints with direct invocation](multi-container-security.md) + [Metrics for multi-container endpoints with direct invocation](multi-container-metrics.md) + [Autoscale multi-container endpoints](multi-container-auto-scaling.md) + [Troubleshoot multi-container endpoints](multi-container-troubleshooting.md) # Create a multi-container endpoint (Boto 3) Create a Multi-container endpoint by calling [CreateModel](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html), [CreateEndpointConfig](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html), and [CreateEndpoint](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpoint.html) APIs as you would to create any other endpoints. You can run these containers sequentially as an inference pipeline, or run each individual container by using direct invocation. Multi-container endpoints have the following requirements when you call `create_model`: + Use the `Containers` parameter instead of `PrimaryContainer`, and include more than one container in the `Containers` parameter. + The `ContainerHostname` parameter is required for each container in a multi-container endpoint with direct invocation. + Set the `Mode` parameter of the `InferenceExecutionConfig` field to `Direct` for direct invocation of each container, or `Serial` to use containers as an inference pipeline. The default mode is `Serial`. **Note** Currently there is a limit of up to 15 containers supported on a multi-container endpoint. The following example creates a multi-container model for direct invocation. 1. Create container elements and `InferenceExecutionConfig` with direct invocation. ``` container1 = { 'Image': '123456789012.dkr.ecr.us-east-1.amazonaws.com/myimage1:mytag', 'ContainerHostname': 'firstContainer' } container2 = { 'Image': '123456789012.dkr.ecr.us-east-1.amazonaws.com/myimage2:mytag', 'ContainerHostname': 'secondContainer' } inferenceExecutionConfig = {'Mode': 'Direct'} ``` 1. Create the model with the container elements and set the `InferenceExecutionConfig` field. ``` import boto3 sm_client = boto3.Session().client('sagemaker') response = sm_client.create_model( ModelName = 'my-direct-mode-model-name', InferenceExecutionConfig = inferenceExecutionConfig, ExecutionRoleArn = role, Containers = [container1, container2] ) ``` To create an endoint, you would then call [create\$1endpoint\$1config](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint_config) and [create\$1endpoint](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint) as you would to create any other endpoint. # Update a multi-container endpoint To update an Amazon SageMaker AI multi-container endpoint, complete the following steps. 1. Call [create\$1model](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model) to create a new model with a new value for the `Mode` parameter in the `InferenceExecutionConfig` field. 1. Call [create\$1endpoint\$1config](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint_config) to create a new endpoint config with a different name by using the new model you created in the previous step. 1. Call [update\$1endpoint](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.update_endpoint) to update the endpoint with the new endpoint config you created in the previous step. # Invoke a multi-container endpoint with direct invocation SageMaker AI multi-container endpoints enable customers to deploy multiple containers to deploy different models on a SageMaker AI endpoint. You can host up to 15 different inference containers on a single endpoint. By using direct invocation, you can send a request to a specific inference container hosted on a multi-container endpoint. To invoke a multi-container endpoint with direct invocation, call [invoke\$1endpoint](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-runtime.html#SageMakerRuntime.Client.invoke_endpoint) as you would invoke any other endpoint, and specify which container you want to invoke by using the `TargetContainerHostname` parameter. The following example directly invokes the `secondContainer` of a multi-container endpoint to get a prediction. ``` import boto3 runtime_sm_client = boto3.Session().client('sagemaker-runtime') response = runtime_sm_client.invoke_endpoint( EndpointName ='my-endpoint', ContentType = 'text/csv', TargetContainerHostname='secondContainer', Body = body) ``` For each direct invocation request to a multi-container endpoint, only the container with the `TargetContainerHostname` processes the invocation request. You will get validation errors if you do any of the following: + Specify a `TargetContainerHostname` that does not exist in the endpoint + Do not specify a value for `TargetContainerHostname` in a request to an endpoint configured for direct invocation + Specify a value for `TargetContainerHostname` in a request to an endpoint that is not configured for direct invocation. # Security with multi-container endpoints with direct invocation For multi-container endpoints with direct invocation, there are multiple containers hosted in a single instance by sharing memory and a storage volume. It's your responsibility to use secure containers, maintain the correct mapping of requests to target containers, and provide users with the correct access to target containers. SageMaker AI uses IAM roles to provide IAM identity-based policies that you use to specify whether access to a resource is allowed or denied to that role, and under what conditions. For information about IAM roles, see [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) in the *AWS Identity and Access Management User Guide*. For information about identity-based policies, see [Identity-based policies and resource-based policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_identity-vs-resource.html). By default, an IAM principal with `InvokeEndpoint` permissions on a multi-container endpoint with direct invocation can invoke any container inside the endpoint with the endpoint name that you specify when you call `invoke_endpoint`. If you need to restrict `invoke_endpoint` access to a limited set of containers inside a multi-container endpoint, use the `sagemaker:TargetContainerHostname` IAM condition key. The following policies show how to limit calls to specific containers within an endpoint. The following policy allows `invoke_endpoint` requests only when the value of the `TargetContainerHostname` field matches one of the specified regular expressions. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "sagemaker:InvokeEndpoint" ], "Effect": "Allow", "Resource": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/endpoint_name", "Condition": { "StringLike": { "sagemaker:TargetModel": [ "customIps*", "common*" ] } } } ] } ``` ------ The following policy denies `invoke_endpoint` requests when the value of the `TargetContainerHostname` field matches one of the specified regular expressions in the `Deny` statement. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "sagemaker:InvokeEndpoint" ], "Effect": "Allow", "Resource": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/endpoint_name", "Condition": { "StringLike": { "sagemaker:TargetModel": [ "model_name*" ] } } }, { "Action": [ "sagemaker:InvokeEndpoint" ], "Effect": "Deny", "Resource": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/endpoint_name", "Condition": { "StringLike": { "sagemaker:TargetModel": [ "special-model_name*" ] } } } ] } ``` ------ For information about SageMaker AI condition keys, see [Condition Keys for SageMaker AI](https://docs.aws.amazon.com/IAM/latest/UserGuide/list_amazonsagemaker.html#amazonsagemaker-policy-keys) in the *AWS Identity and Access Management User Guide*. # Metrics for multi-container endpoints with direct invocation In addition to the endpoint metrics that are listed in [Amazon SageMaker AI metrics in Amazon CloudWatch](monitoring-cloudwatch.md), SageMaker AI also provides per-container metrics. Per-container metrics for multi-container endpoints with direct invocation are located in CloudWatch and categorized into two namespaces: `AWS/SageMaker` and `aws/sagemaker/Endpoints`. The `AWS/SageMaker` namespace includes invocation-related metrics, and the `aws/sagemaker/Endpoints` namespace includes memory and CPU utilization metrics. The following table lists the per-container metrics for multi-container endpoints with direct invocation. All the metrics use the [`EndpointName, VariantName, ContainerName`] dimension, which filters metrics at a specific endpoint, for a specific variant and corresponding to a specific container. These metrics share the same metric names as in those for inference pipelines, but at a per-container level [`EndpointName, VariantName, ContainerName`]. | | | | | | --- |--- |--- |--- | | Metric Name | Description | Dimension | NameSpace | | Invocations | The number of InvokeEndpoint requests sent to a container inside an endpoint. To get the total number of requests sent to that container, use the Sum statistic. Units: None Valid statistics: Sum, Sample Count | EndpointName, VariantName, ContainerName | AWS/SageMaker | | Invocation4XX Errors | The number of InvokeEndpoint requests that the model returned a 4xx HTTP response code for on a specific container. For each 4xx response, SageMaker AI sends a 1. Units: None Valid statistics: Average, Sum | EndpointName, VariantName, ContainerName | AWS/SageMaker | | Invocation5XX Errors | The number of InvokeEndpoint requests that the model returned a 5xx HTTP response code for on a specific container. For each 5xx response, SageMaker AI sends a 1. Units: None Valid statistics: Average, Sum | EndpointName, VariantName, ContainerName | AWS/SageMaker | | ContainerLatency | The time it took for the target container to respond as viewed from SageMaker AI. ContainerLatency includes the time it took to send the request, to fetch the response from the model's container, and to complete inference in the container. Units: Microseconds Valid statistics: Average, Sum, Min, Max, Sample Count | EndpointName, VariantName, ContainerName | AWS/SageMaker | | OverheadLatency | The time added to the time taken to respond to a client request by SageMaker AI for overhead. OverheadLatency is measured from the time that SageMaker AI receives the request until it returns a response to the client, minus theModelLatency. Overhead latency can vary depending on request and response payload sizes, request frequency, and authentication or authorization of the request, among other factors. Units: Microseconds Valid statistics: Average, Sum, Min, Max, `Sample Count ` | EndpointName, VariantName, ContainerName | AWS/SageMaker | | CPUUtilization | The percentage of CPU units that are used by each container running on an instance. The value ranges from 0% to 100%, and is multiplied by the number of CPUs. For example, if there are four CPUs, CPUUtilization can range from 0% to 400%. For endpoints with direct invocation, the number of CPUUtilization metrics equals the number of containers in that endpoint. Units: Percent | EndpointName, VariantName, ContainerName | aws/sagemaker/Endpoints | | MemoryUtilizaton | The percentage of memory that is used by each container running on an instance. This value ranges from 0% to 100%. Similar as CPUUtilization, in endpoints with direct invocation, the number of MemoryUtilization metrics equals the number of containers in that endpoint. Units: Percent | EndpointName, VariantName, ContainerName | aws/sagemaker/Endpoints | All the metrics in the previous table are specific to multi-container endpoints with direct invocation. Besides these special per-container metrics, there are also metrics at the variant level with dimension `[EndpointName, VariantName]` for all the metrics in the table expect `ContainerLatency`. # Autoscale multi-container endpoints If you want to configure automatic scaling for a multi-container endpoint using the `InvocationsPerInstance` metric, we recommend that the model in each container exhibits similar CPU utilization and latency on each inference request. This is recommended because if traffic to the multi-container endpoint shifts from a low CPU utilization model to a high CPU utilization model, but the overall call volume remains the same, the endpoint does not scale out and there may not be enough instances to handle all the requests to the high CPU utilization model. For information about automatically scaling endpoints, see [Automatic scaling of Amazon SageMaker AI models](endpoint-auto-scaling.md). # Troubleshoot multi-container endpoints The following sections can help you troubleshoot errors with multi-container endpoints. ## Ping Health Check Errors With multiple containers, endpoint memory and CPU are under higher pressure during endpoint creation. Specifically, the `MemoryUtilization` and `CPUUtilization` metrics are higher than for single-container endpoints, because utilization pressure is proportional to the number of containers. Because of this, we recommend that you choose instance types with enough memory and CPU to ensure that there is enough memory on the instance to have all the models loaded (the same guidance applies to deploying an inference pipeline). Otherwise, your endpoint creation might fail with an error such as `XXX did not pass the ping health check`. ## Missing accept-bind-to-port=true Docker label The containers in a multi-container endpoints listen on the port specified in the `SAGEMAKER_BIND_TO_PORT` environment variable instead of port 8080. When a container runs in a multi-container endpoint, SageMaker AI automatically provides this environment variable to the container. If this environment variable isn't present, containers default to using port 8080. To indicate that your container complies with this requirement, use the following command to add a label to your Dockerfile: ``` LABEL com.amazonaws.sagemaker.capabilities.accept-bind-to-port=true ``` Otherwise, You will see an error message such as `Your Ecr Image XXX does not contain required com.amazonaws.sagemaker.capabilities.accept-bind-to-port=true Docker label(s).` If your container needs to listen on a second port, choose a port in the range specified by the `SAGEMAKER_SAFE_PORT_RANGE` environment variable. Specify the value as an inclusive range in the format *XXXX*-*YYYY*, where XXXX and YYYY are multi-digit integers. SageMaker AI provides this value automatically when you run the container in a multi-container endpoint. # Inference pipelines in Amazon SageMaker AI An *inference pipeline* is a Amazon SageMaker AI model that is composed of a linear sequence of two to fifteen containers that process requests for inferences on data. You use an inference pipeline to define and deploy any combination of pretrained SageMaker AI built-in algorithms and your own custom algorithms packaged in Docker containers. You can use an inference pipeline to combine preprocessing, predictions, and post-processing data science tasks. Inference pipelines are fully managed. You can add SageMaker AI Spark ML Serving and scikit-learn containers that reuse the data transformers developed for training models. The entire assembled inference pipeline can be considered as a SageMaker AI model that you can use to make either real-time predictions or to process batch transforms directly without any external preprocessing. Within an inference pipeline model, SageMaker AI handles invocations as a sequence of HTTP requests. The first container in the pipeline handles the initial request, then the intermediate response is sent as a request to the second container, and so on, for each container in the pipeline. SageMaker AI returns the final response to the client. When you deploy the pipeline model, SageMaker AI installs and runs all of the containers on each Amazon Elastic Compute Cloud (Amazon EC2) instance in the endpoint or transform job. Feature processing and inferences run with low latency because the containers are co-located on the same EC2 instances. You define the containers for a pipeline model using the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) operation or from the console. Instead of setting one `PrimaryContainer`, you use the `Containers` parameter to set the containers that make up the pipeline. You also specify the order in which the containers are executed. A pipeline model is immutable, but you can update an inference pipeline by deploying a new one using the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UpdateEndpoint.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UpdateEndpoint.html) operation. This modularity supports greater flexibility during experimentation. For information on how to create an inference pipeline with the SageMaker Model Registry, see [Model Registration Deployment with Model Registry](model-registry.md). There are no additional costs for using this feature. You pay only for the instances running on an endpoint. **Topics** + [Sample Notebooks for Inference Pipelines](#inference-pipeline-sample-notebooks) + [Feature Processing with Spark ML and Scikit-learn](inference-pipeline-mleap-scikit-learn-containers.md) + [Create a Pipeline Model](inference-pipeline-create-console.md) + [Run Real-time Predictions with an Inference Pipeline](inference-pipeline-real-time.md) + [Batch transforms with inference pipelines](inference-pipeline-batch.md) + [Inference Pipeline Logs and Metrics](inference-pipeline-logs-metrics.md) + [Troubleshoot Inference Pipelines](inference-pipeline-troubleshoot.md) ## Sample Notebooks for Inference Pipelines For an example that shows how to create and deploy inference pipelines, see the [Inference Pipeline with Scikit-learn and Linear Learner](https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker-python-sdk/scikit_learn_inference_pipeline) sample notebook. For instructions on creating and accessing Jupyter notebook instances that you can use to run the example in SageMaker AI, see [Amazon SageMaker notebook instances](nbi.md). To see a list of all the SageMaker AI samples, after creating and opening a notebook instance, choose the **SageMaker AI Examples** tab. There are three inference pipeline notebooks. The first two inference pipeline notebooks just described are located in the `advanced_functionality` folder and the third notebook is in the `sagemaker-python-sdk` folder. To open a notebook, choose its **Use** tab, then choose **Create copy**. # Feature Processing with Spark ML and Scikit-learn Before training a model with either Amazon SageMaker AI built-in algorithms or custom algorithms, you can use Spark and scikit-learn preprocessors to transform your data and engineer features. ## Feature Processing with Spark ML You can run Spark ML jobs with [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html), a serverless ETL (extract, transform, load) service, from your SageMaker AI notebook. You can also connect to existing EMR clusters to run Spark ML jobs with [Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-what-is-emr.html). To do this, you need an AWS Identity and Access Management (IAM) role that grants permission for making calls from your SageMaker AI notebook to AWS Glue. **Note** To see which Python and Spark versions AWS Glue supports, refer to [AWS Glue Release Notes](/glue/latest/dg/release-notes.html). After engineering features, you package and serialize Spark ML jobs with MLeap into MLeap containers that you can add to an inference pipeline. You don't need to use externally managed Spark clusters. With this approach, you can seamlessly scale from a sample of rows to terabytes of data. The same transformers work for both training and inference, so you don't need to duplicate preprocessing and feature engineering logic or develop a one-time solution to make the models persist. With inference pipelines, you don't need to maintain outside infrastructure, and you can make predictions directly from data inputs. When you run a Spark ML job on AWS Glue, a Spark ML pipeline is serialized into [MLeap](https://github.com/combust/mleap) format. Then, you can use the job with the [SparkML Model Serving Container](https://github.com/aws/sagemaker-sparkml-serving-container) in a SageMaker AI Inference Pipeline. *MLeap* is a serialization format and execution engine for machine learning pipelines. It supports Spark, Scikit-learn, and TensorFlow for training pipelines and exporting them to a serialized pipeline called an MLeap Bundle. You can deserialize Bundles back into Spark for batch-mode scoring or into the MLeap runtime to power real-time API services. For an example that shows how to feature process with Spark ML, see the [Train an ML Model using Apache Spark in Amazon EMR and deploy in SageMaker AI](https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker-python-sdk/sparkml_serving_emr_mleap_abalone) sample notebook. ## Feature Processing with Scikit-Learn You can run and package scikit-learn jobs into containers directly in Amazon SageMaker AI. For an example of Python code for building a scikit-learn featurizer model that trains on [Fisher's Iris flower data set](http://archive.ics.uci.edu/ml/datasets/Iris) and predicts the species of Iris based on morphological measurements, see [IRIS Training and Prediction with Sagemaker Scikit-learn](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-python-sdk/scikit_learn_iris). # Create a Pipeline Model To create a pipeline model that can be deployed to an endpoint or used for a batch transform job, use the Amazon SageMaker AI console or the `CreateModel` operation. **To create an inference pipeline (console)** 1. Open the Amazon SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/). 1. Choose **Models**, and then choose **Create models** from the **Inference** group. 1. On the **Create model** page, provide a model name, choose an IAM role, and, if you want to use a private VPC, specify VPC values. ![\[The page for creating a model for an Inference Pipeline.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/create-pipeline-model.png) 1. To add information about the containers in the inference pipeline, choose **Add container**, then choose **Next**. 1. Complete the fields for each container in the order that you want to execute them, up to the maximum of fifteen. Complete the **Container input options**, , **Location of inference code image**, and, optionally, **Location of model artifacts**, **Container host name**, and **Environmental variables** fields. . ![\[Creating a pipeline model with containers.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/create-pipeline-model-containers.png) The **MyInferencePipelineModel** page summarizes the settings for the containers that provide input for the model. If you provided the environment variables in a corresponding container definition, SageMaker AI shows them in the **Environment variables** field. ![\[The summary of container settings for the pipeline model.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/pipeline-MyInferencePipelinesModel-recap.png) # Run Real-time Predictions with an Inference Pipeline You can use trained models in an inference pipeline to make real-time predictions directly without performing external preprocessing. When you configure the pipeline, you can choose to use the built-in feature transformers already available in Amazon SageMaker AI. Or, you can implement your own transformation logic using just a few lines of scikit-learn or Spark code. [MLeap](https://combust.github.io/mleap-docs/), a serialization format and execution engine for machine learning pipelines, supports Spark, scikit-learn, and TensorFlow for training pipelines and exporting them to a serialized pipeline called an MLeap Bundle. You can deserialize Bundles back into Spark for batch-mode scoring or into the MLeap runtime to power real-time API services. The containers in a pipeline listen on the port specified in the `SAGEMAKER_BIND_TO_PORT` environment variable (instead of 8080). When running in an inference pipeline, SageMaker AI automatically provides this environment variable to containers. If this environment variable isn't present, containers default to using port 8080. To indicate that your container complies with this requirement, use the following command to add a label to your Dockerfile: ``` LABEL com.amazonaws.sagemaker.capabilities.accept-bind-to-port=true ``` If your container needs to listen on a second port, choose a port in the range specified by the `SAGEMAKER_SAFE_PORT_RANGE` environment variable. Specify the value as an inclusive range in the format **"XXXX-YYYY"**, where `XXXX` and `YYYY` are multi-digit integers. SageMaker AI provides this value automatically when you run the container in a multicontainer pipeline. **Note** To use custom Docker images in a pipeline that includes [SageMaker AI built-in algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html), you need an [Amazon Elastic Container Registry (Amazon ECR) policy](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html). Your Amazon ECR repository must grant SageMaker AI permission to pull the image. For more information, see [Troubleshoot Amazon ECR Permissions for Inference Pipelines](inference-pipeline-troubleshoot.md#inference-pipeline-troubleshoot-permissions). ## Create and Deploy an Inference Pipeline Endpoint The following code creates and deploys a real-time inference pipeline model with SparkML and XGBoost models in series using the SageMaker AI SDK. ``` from sagemaker.model import Model from sagemaker.pipeline_model import PipelineModel from sagemaker.sparkml.model import SparkMLModel sparkml_data = 's3://{}/{}/{}'.format(s3_model_bucket, s3_model_key_prefix, 'model.tar.gz') sparkml_model = SparkMLModel(model_data=sparkml_data) xgb_model = Model(model_data=xgb_model.model_data, image=training_image) model_name = 'serial-inference-' + timestamp_prefix endpoint_name = 'serial-inference-ep-' + timestamp_prefix sm_model = PipelineModel(name=model_name, role=role, models=[sparkml_model, xgb_model]) sm_model.deploy(initial_instance_count=1, instance_type='ml.c4.xlarge', endpoint_name=endpoint_name) ``` ## Request Real-Time Inference from an Inference Pipeline Endpoint The following example shows how to make real-time predictions by calling an inference endpoint and passing a request payload in JSON format: ``` import sagemaker from sagemaker.predictor import json_serializer, json_deserializer, Predictor payload = { "input": [ { "name": "Pclass", "type": "float", "val": "1.0" }, { "name": "Embarked", "type": "string", "val": "Q" }, { "name": "Age", "type": "double", "val": "48.0" }, { "name": "Fare", "type": "double", "val": "100.67" }, { "name": "SibSp", "type": "double", "val": "1.0" }, { "name": "Sex", "type": "string", "val": "male" } ], "output": { "name": "features", "type": "double", "struct": "vector" } } predictor = Predictor(endpoint=endpoint_name, sagemaker_session=sagemaker.Session(), serializer=json_serializer, content_type='text/csv', accept='application/json') print(predictor.predict(payload)) ``` The response you get from `predictor.predict(payload)` is the model's inference result. ## Realtime inference pipeline example You can run this [example notebook using the SKLearn predictor](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/scikit_learn_randomforest/Sklearn_on_SageMaker_end2end.ipynb) that shows how to deploy an endpoint, run an inference request, then deserialize the response. Find this notebook and more examples in the [Amazon SageMaker example GitHub repository](https://github.com/awslabs/amazon-sagemaker-examples). # Batch transforms with inference pipelines To get inferences on an entire dataset you run a batch transform on a trained model. To run inferences on a full dataset, you can use the same inference pipeline model created and deployed to an endpoint for real-time processing in a batch transform job. To run a batch transform job in a pipeline, you download the input data from Amazon S3 and send it in one or more HTTP requests to the inference pipeline model. For an example that shows how to prepare data for a batch transform, see "Section 2 - Preprocess the raw housing data using Scikit Learn" of the [Amazon SageMaker Multi-Model Endpoints using Linear Learner sample notebook](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality/multi_model_linear_learner_home_value). For information about Amazon SageMaker AI batch transforms, see [Batch transform for inference with Amazon SageMaker AI](batch-transform.md). **Note** To use custom Docker images in a pipeline that includes [Amazon SageMaker AI built-in algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html), you need an [Amazon Elastic Container Registry (ECR) policy](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html). Your Amazon ECR repository must grant SageMaker AI permission to pull the image. For more information, see [Troubleshoot Amazon ECR Permissions for Inference Pipelines](inference-pipeline-troubleshoot.md#inference-pipeline-troubleshoot-permissions). The following example shows how to run a transform job using the [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable). In this example, `model_name` is the inference pipeline that combines SparkML and XGBoost models (created in previous examples). The Amazon S3 location specified by `input_data_path` contains the input data, in CSV format, to be downloaded and sent to the Spark ML model. After the transform job has finished, the Amazon S3 location specified by `output_data_path` contains the output data returned by the XGBoost model in CSV format. ``` import sagemaker input_data_path = 's3://{}/{}/{}'.format(default_bucket, 'key', 'file_name') output_data_path = 's3://{}/{}'.format(default_bucket, 'key') transform_job = sagemaker.transformer.Transformer( model_name = model_name, instance_count = 1, instance_type = 'ml.m4.xlarge', strategy = 'SingleRecord', assemble_with = 'Line', output_path = output_data_path, base_transform_job_name='inference-pipelines-batch', sagemaker_session=sagemaker.Session(), accept = CONTENT_TYPE_CSV) transform_job.transform(data = input_data_path, content_type = CONTENT_TYPE_CSV, split_type = 'Line') ``` # Inference Pipeline Logs and Metrics Monitoring is important for maintaining the reliability, availability, and performance of Amazon SageMaker AI resources. To monitor and troubleshoot inference pipeline performance, use Amazon CloudWatch logs and error messages. For information about the monitoring tools that SageMaker AI provides, see [Monitoring AWS resources in Amazon SageMaker AI](monitoring-overview.md). ## Use Metrics to Monitor Multi-container Models To monitor the multi-container models in Inference Pipelines, use Amazon CloudWatch. CloudWatch collects raw data and processes it into readable, near real-time metrics. SageMaker AI training jobs and endpoints write CloudWatch metrics and logs in the `AWS/SageMaker` namespace. The following tables list the metrics and dimensions for the following: + Endpoint invocations + Training jobs, batch transform jobs, and endpoint instances A *dimension* is a name/value pair that uniquely identifies a metric. You can assign up to 10 dimensions to a metric. For more information on monitoring with CloudWatch, see [Amazon SageMaker AI metrics in Amazon CloudWatch](monitoring-cloudwatch.md). **Endpoint Invocation Metrics** The `AWS/SageMaker` namespace includes the following request metrics from calls to [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InvokeEndpoint.html). Metrics are reported at a 1-minute intervals. | Metric | Description | | --- | --- | | Invocation4XXErrors | The number of `InvokeEndpoint` requests that the model returned a `4xx` HTTP response code for. For each `4xx` response, SageMaker AI sends a `1`. Units: None Valid statistics: `Average`, `Sum` | | Invocation5XXErrors | The number of `InvokeEndpoint` requests that the model returned a `5xx` HTTP response code for. For each `5xx` response, SageMaker AI sends a `1`. Units: None Valid statistics: `Average`, `Sum` | | Invocations | The `number of InvokeEndpoint` requests sent to a model endpoint. To get the total number of requests sent to a model endpoint, use the `Sum` statistic. Units: None Valid statistics: `Sum`, `Sample Count` | | InvocationsPerInstance | The number of endpoint invocations sent to a model, normalized by `InstanceCount` in each `ProductionVariant`. SageMaker AI sends 1/`numberOfInstances` as the value for each request, where `numberOfInstances` is the number of active instances for the ProductionVariant at the endpoint at the time of the request. Units: None Valid statistics: `Sum` | | ModelLatency | The time the model or models took to respond. This includes the time it took to send the request, to fetch the response from the model container, and to complete the inference in the container. ModelLatency is the total time taken by all containers in an inference pipeline.Units: MicrosecondsValid statistics: `Average`, `Sum`, `Min`, `Max`, Sample Count | | OverheadLatency | The time added to the time taken to respond to a client request by SageMaker AI for overhead. `OverheadLatency` is measured from the time that SageMaker AI receives the request until it returns a response to the client, minus the `ModelLatency`. Overhead latency can vary depending on request and response payload sizes, request frequency, and authentication or authorization of the request, among other factors. Units: Microseconds Valid statistics: `Average`, `Sum`, `Min`, `Max`, `Sample Count` | | ContainerLatency | The time it took for an Inference Pipelines container to respond as viewed from SageMaker AI. ContainerLatency includes the time it took to send the request, to fetch the response from the model's container, and to complete inference in the container.Units: MicrosecondsValid statistics: `Average`, `Sum`, `Min`, `Max`, `Sample Count` | **Dimensions for Endpoint Invocation Metrics** | Dimension | Description | | --- | --- | | EndpointName, VariantName, ContainerName | Filters endpoint invocation metrics for a `ProductionVariant` at the specified endpoint and for the specified variant. | For an inference pipeline endpoint, CloudWatch lists per-container latency metrics in your account as **Endpoint Container Metrics** and **Endpoint Variant Metrics** in the **SageMaker AI** namespace, as follows. The `ContainerLatency` metric appears only for inferences pipelines. ![\[The CloudWatch dashboard for an inference pipeline.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/pipeline-endpoint-metrics.png) For each endpoint and each container, latency metrics display names for the container, endpoint, variant, and metric. ![\[The latency metrics for an endpoint.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/pipeline-endpoint-metrics-details.png) **Training Job, Batch Transform Job, and Endpoint Instance Metrics** The namespaces `/aws/sagemaker/TrainingJobs`, `/aws/sagemaker/TransformJobs`, and `/aws/sagemaker/Endpoints` include the following metrics for training jobs and endpoint instances. Metrics are reported at a 1-minute intervals. | Metric | Description | | --- | --- | | CPUUtilization | The percentage of CPU units that are used by the containers running on an instance. The value ranges from 0% to 100%, and is multiplied by the number of CPUs. For example, if there are four CPUs, `CPUUtilization` can range from 0% to 400%. For training jobs, `CPUUtilization` is the CPU utilization of the algorithm container running on the instance. For batch transform jobs, `CPUUtilization` is the CPU utilization of the transform container running on the instance. For multi-container models, `CPUUtilization` is the sum of CPU utilization by all containers running on the instance. For endpoint variants, `CPUUtilization` is the sum of CPU utilization by all of the containers running on the instance. Units: Percent | | MemoryUtilization | The percentage of memory that is used by the containers running on an instance. This value ranges from 0% to 100%.For training jobs, `MemoryUtilization` is the memory used by the algorithm container running on the instance.For batch transform jobs, `MemoryUtilization` is the memory used by the transform container running on the instance.For multi-container models, MemoryUtilization is the sum of memory used by all containers running on the instance.For endpoint variants, `MemoryUtilization` is the sum of memory used by all of the containers running on the instance.Units: Percent | | GPUUtilization | The percentage of GPU units that are used by the containers running on an instance. `GPUUtilization` ranges from 0% to 100% and is multiplied by the number of GPUs. For example, if there are four GPUs, `GPUUtilization` can range from 0% to 400%. For training jobs, `GPUUtilization` is the GPU used by the algorithm container running on the instance. For batch transform jobs, `GPUUtilization` is the GPU used by the transform container running on the instance. For multi-container models, `GPUUtilization` is the sum of GPU used by all containers running on the instance. For endpoint variants, `GPUUtilization` is the sum of GPU used by all of the containers running on the instance. Units: Percent | | GPUMemoryUtilization | The percentage of GPU memory used by the containers running on an instance. GPUMemoryUtilization ranges from 0% to 100% and is multiplied by the number of GPUs. For example, if there are four GPUs, `GPUMemoryUtilization` can range from 0% to 400%. For training jobs, `GPUMemoryUtilization` is the GPU memory used by the algorithm container running on the instance. For batch transform jobs, `GPUMemoryUtilization` is the GPU memory used by the transform container running on the instance. For multi-container models, `GPUMemoryUtilization` is sum of GPU used by all containers running on the instance. For endpoint variants, `GPUMemoryUtilization` is the sum of the GPU memory used by all of the containers running on the instance. Units: Percent | | DiskUtilization | The percentage of disk space used by the containers running on an instance. DiskUtilization ranges from 0% to 100%. This metric is not supported for batch transform jobs. For training jobs, `DiskUtilization` is the disk space used by the algorithm container running on the instance. For endpoint variants, `DiskUtilization` is the sum of the disk space used by all of the provided containers running on the instance. Units: Percent | **Dimensions for Training Job, Batch Transform Job, and Endpoint Instance Metrics** | Dimension | Description | | --- | --- | | Host | For training jobs, `Host` has the format `[training-job-name]/algo-[instance-number-in-cluster]`. Use this dimension to filter instance metrics for the specified training job and instance. This dimension format is present only in the `/aws/sagemaker/TrainingJobs` namespace. For batch transform jobs, `Host` has the format `[transform-job-name]/[instance-id]`. Use this dimension to filter instance metrics for the specified batch transform job and instance. This dimension format is present only in the `/aws/sagemaker/TransformJobs` namespace. For endpoints, `Host` has the format `[endpoint-name]/[ production-variant-name ]/[instance-id]`. Use this dimension to filter instance metrics for the specified endpoint, variant, and instance. This dimension format is present only in the `/aws/sagemaker/Endpoints` namespace. | To help you debug your training jobs, endpoints, and notebook instance lifecycle configurations, SageMaker AI also sends anything an algorithm container, a model container, or a notebook instance lifecycle configuration sends to `stdout` or `stderr` to Amazon CloudWatch Logs. You can use this information for debugging and to analyze progress. ## Use Logs to Monitor an Inference Pipeline The following table lists the log groups and log streams SageMaker AI. sends to Amazon CloudWatch A *log stream* is a sequence of log events that share the same source. Each separate source of logs into CloudWatch makes up a separate log stream. A *log group* is a group of log streams that share the same retention, monitoring, and access control settings. **Logs** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipeline-logs-metrics.html) **Note** SageMaker AI creates the `/aws/sagemaker/NotebookInstances` log group when you create a notebook instance with a lifecycle configuration. For more information, see [Customization of a SageMaker notebook instance using an LCC script](notebook-lifecycle-config.md). For more information about SageMaker AI logging, see [CloudWatch Logs for Amazon SageMaker AI](logging-cloudwatch.md). # Troubleshoot Inference Pipelines To troubleshoot inference pipeline issues, use CloudWatch logs and error messages. If you are using custom Docker images in a pipeline that includes Amazon SageMaker AI built-in algorithms, you might also encounter permissions problems. To grant the required permissions, create an Amazon Elastic Container Registry (Amazon ECR) policy. **Topics** + [Troubleshoot Amazon ECR Permissions for Inference Pipelines](#inference-pipeline-troubleshoot-permissions) + [Use CloudWatch Logs to Troubleshoot SageMaker AI Inference Pipelines](#inference-pipeline-troubleshoot-logs) + [Use Error Messages to Troubleshoot Inference Pipelines](#inference-pipeline-troubleshoot-errors) ## Troubleshoot Amazon ECR Permissions for Inference Pipelines When you use custom Docker images in a pipeline that includes [SageMaker AI built-in algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html), you need an [Amazon ECR policy](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html). The policy allows your Amazon ECR repository to grant permission for SageMaker AI to pull the image. The policy must add the following permissions: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "allowSageMakerToPull", "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com" }, "Action": [ "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability" ], "Resource": "*" } ] } ``` ------ ## Use CloudWatch Logs to Troubleshoot SageMaker AI Inference Pipelines SageMaker AI publishes the container logs for endpoints that deploy an inference pipeline to Amazon CloudWatch at the following path for each container. ``` /aws/sagemaker/Endpoints/{EndpointName}/{Variant}/{InstanceId}/{ContainerHostname} ``` For example, logs for this endpoint are published to the following log groups and streams: ``` EndpointName: MyInferencePipelinesEndpoint Variant: MyInferencePipelinesVariant InstanceId: i-0179208609ff7e488 ContainerHostname: MyContainerName1 and MyContainerName2 ``` ``` logGroup: /aws/sagemaker/Endpoints/MyInferencePipelinesEndpoint logStream: MyInferencePipelinesVariant/i-0179208609ff7e488/MyContainerName1 logStream: MyInferencePipelinesVariant/i-0179208609ff7e488/MyContainerName2 ``` A *log stream* is a sequence of log events that share the same source. Each separate source of logs into CloudWatch makes up a separate log stream. A *log group* is a group of log streams that share the same retention, monitoring, and access control settings. **To see the log groups and streams** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation page, choose **Logs**. 1. In **Log Groups**. filter on **MyInferencePipelinesEndpoint**: ![\[The CloudWatch log groups filtered for the inference pipeline endpoint.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/pipeline-log-group-filter.png) 1. To see the log streams, on the CloudWatch **Log Groups** page, choose **MyInferencePipelinesEndpoint**, and then **Search Log Group**. ![\[The CloudWatch log stream for the inference pipeline.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/pipeline-log-streams-2.png) For a list of the logs that SageMaker AI publishes, see [Inference Pipeline Logs and Metrics](inference-pipeline-logs-metrics.md). ## Use Error Messages to Troubleshoot Inference Pipelines The inference pipeline error messages indicate which containers failed. If an error occurs while SageMaker AI is invoking an endpoint, the service returns a `ModelError` (error code 424), which indicates which container failed. If the request payload (the response from the previous container) exceeds the limit of 5 MB, SageMaker AI provides a detailed error message, such as: Received response from MyContainerName1 with status code 200. However, the request payload from MyContainerName1 to MyContainerName2 is 6000000 bytes, which has exceeded the maximum limit of 5 MB. `` If a container fails the ping health check while SageMaker AI is creating an endpoint, it returns a `ClientError` and indicates all of the containers that failed the ping check in the last health check. # Delete Endpoints and Resources Delete endpoints to stop incurring charges. ## Delete Endpoint Delete your endpoint programmatically using AWS SDK for Python (Boto3), with the AWS CLI, or interactively using the SageMaker AI console. SageMaker AI frees up all of the resources that were deployed when the endpoint was created. Deleting an endpoint will not delete the endpoint configuration or the SageMaker AI model. See [Delete Endpoint Configuration](#realtime-endpoints-delete-endpoint-config) and [Delete Model](#realtime-endpoints-delete-model) for information on how to delete your endpoint configuration and SageMaker AI model. ------ #### [ AWS SDK for Python (Boto3) ] Use the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DeleteEndpoint.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DeleteEndpoint.html) API to delete your endpoint. Specify the name of your endpoint for the `EndpointName` field. ``` import boto3 # Specify your AWS Region aws_region='' # Specify the name of your endpoint endpoint_name='' # Create a low-level SageMaker service client. sagemaker_client = boto3.client('sagemaker', region_name=aws_region) # Delete endpoint sagemaker_client.delete_endpoint(EndpointName=endpoint_name) ``` ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/sagemaker/delete-endpoint.html](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/delete-endpoint.html) command to delete your endpoint. Specify the name of your endpoint for the `endpoint-name` flag. ``` aws sagemaker delete-endpoint --endpoint-name ``` ------ #### [ SageMaker AI Console ] Delete your endpoint interactively with the SageMaker AI console. 1. In the SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/) navigation menu, choose **Inference**. 1. Choose **Endpoints** from the drop down menu. A list of endpoints created in you AWS account will appear by name, Amazon Resource Name (ARN), creation time, status, and a time stamp of when the endpoint was last updated. 1. Select the endpoint you want to delete. 1. Select the **Actions** dropdown button in the top right corner. 1. Choose **Delete**. ------ ## Delete Endpoint Configuration Delete your endpoint configuration programmaticially using AWS SDK for Python (Boto3), with the AWS CLI, or interactively using the SageMaker AI console. Deleting an endpoint configuration does not delete endpoints created using this configuration. See [Delete Endpoint](#realtime-endpoints-delete-endpoint) for information on how to delete your endpoint. Do not delete an endpoint configuration in use by an endpoint that is live or while the endpoint is being updated or created. You might lose visibility into the instance type the endpoint is using if you delete the endpoint configuration of an endpoint that is active or being created or updated. ------ #### [ AWS SDK for Python (Boto3) ] Use the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DeleteEndpointConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DeleteEndpointConfig.html) API to delete your endpoint. Specify the name of your endpoint configuration for the `EndpointConfigName` field. ``` import boto3 # Specify your AWS Region aws_region='' # Specify the name of your endpoint configuration endpoint_config_name='' # Create a low-level SageMaker service client. sagemaker_client = boto3.client('sagemaker', region_name=aws_region) # Delete endpoint configuration sagemaker_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name) ``` You can optionally use the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpointConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpointConfig.html) API to return information about the name of the your deployed models (production variants) such as the name of your model and the name of the endpoint configuration associated with that deployed model. Provide the name of your endpoint for the `EndpointConfigName` field. ``` # Specify the name of your endpoint endpoint_name='' # Create a low-level SageMaker service client. sagemaker_client = boto3.client('sagemaker', region_name=aws_region) # Store DescribeEndpointConfig response into a variable that we can index in the next step. response = sagemaker_client.describe_endpoint_config(EndpointConfigName=endpoint_name) # Delete endpoint endpoint_config_name = response['ProductionVariants'][0]['EndpointConfigName'] # Delete endpoint configuration sagemaker_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name) ``` For more information about other response elements returned by `DescribeEndpointConfig`, see [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpointConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpointConfig.html) in the [SageMaker API Reference guide](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_Operations_Amazon_SageMaker_Service.html). ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/sagemaker/delete-endpoint-config.html](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/delete-endpoint-config.html) command to delete your endpoint configuration. Specify the name of your endpoint configuration for the `endpoint-config-name` flag. ``` aws sagemaker delete-endpoint-config \ --endpoint-config-name ``` You can optionally use the [https://docs.aws.amazon.com/cli/latest/reference/sagemaker/describe-endpoint-config.html](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/describe-endpoint-config.html) command to return information about the name of the your deployed models (production variants) such as the name of your model and the name of the endpoint configuration associated with that deployed model. Provide the name of your endpoint for the `endpoint-config-name` flag. ``` aws sagemaker describe-endpoint-config --endpoint-config-name ``` This will return a JSON response. You can copy and paste, use a JSON parser, or use a tool built for JSON parsing to obtain the endpoint configuration name associated with that endpoint. ------ #### [ SageMaker AI Console ] Delete your endpoint configuration interactively with the SageMaker AI console. 1. In the SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/) navigation menu, choose **Inference**. 1. Choose **Endpoint configurations** from the dropdown menu. A list of endpoint configurations created in you AWS account will appear by name, Amazon Resource Name (ARN), and creation time. 1. Select the endpoint configuration you want to delete. 1. Select the **Actions** dropdown button in the top right corner. 1. Choose **Delete**. ------ ## Delete Model Delete your SageMaker AI model programmaticially using AWS SDK for Python (Boto3), with the AWS CLI, or interactively using the SageMaker AI console. Deleting a SageMaker AI model only deletes the model entry that was created in SageMaker AI. Deleting a model does not delete model artifacts, inference code, or the IAM role that you specified when creating the model. ------ #### [ AWS SDK for Python (Boto3) ] Use the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DeleteModel.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DeleteModel.html) API to delete your SageMaker AI model. Specify the name of your model for the `ModelName` field. ``` import boto3 # Specify your AWS Region aws_region='' # Specify the name of your endpoint configuration model_name='' # Create a low-level SageMaker service client. sagemaker_client = boto3.client('sagemaker', region_name=aws_region) # Delete model sagemaker_client.delete_model(ModelName=model_name) ``` You can optionally use the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpointConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpointConfig.html) API to return information about the name of the your deployed models (production variants) such as the name of your model and the name of the endpoint configuration associated with that deployed model. Provide the name of your endpoint for the `EndpointConfigName` field. ``` # Specify the name of your endpoint endpoint_name='' # Create a low-level SageMaker service client. sagemaker_client = boto3.client('sagemaker', region_name=aws_region) # Store DescribeEndpointConfig response into a variable that we can index in the next step. response = sagemaker_client.describe_endpoint_config(EndpointConfigName=endpoint_name) # Delete endpoint model_name = response['ProductionVariants'][0]['ModelName'] sagemaker_client.delete_model(ModelName=model_name) ``` For more information about other response elements returned by `DescribeEndpointConfig`, see [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpointConfig.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpointConfig.html) in the [SageMaker API Reference guide](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_Operations_Amazon_SageMaker_Service.html). ------ #### [ AWS CLI ] Use the [https://docs.aws.amazon.com/cli/latest/reference/sagemaker/delete-model.html](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/delete-model.html) command to delete your SageMaker AI model. Specify the name of your model for the `model-name` flag. ``` aws sagemaker delete-model \ --model-name ``` You can optionally use the [https://docs.aws.amazon.com/cli/latest/reference/sagemaker/describe-endpoint-config.html](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/describe-endpoint-config.html) command to return information about the name of the your deployed models (production variants) such as the name of your model and the name of the endpoint configuration associated with that deployed model. Provide the name of your endpoint for the `endpoint-config-name` flag. ``` aws sagemaker describe-endpoint-config --endpoint-config-name ``` This will return a JSON response. You can copy and paste, use a JSON parser, or use a tool built for JSON parsing to obtain the name of the model associated with that endpoint. ------ #### [ SageMaker AI Console ] Delete your SageMaker AI model interactively with the SageMaker AI console. 1. In the SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/) navigation menu, choose **Inference**. 1. Choose **Models** from the dropdown menu. A list of models created in you AWS account will appear by name, Amazon Resource Name (ARN), and creation time. 1. Select the model you want to delete. 1. Select the **Actions** dropdown button in the top right corner. 1. Choose **Delete**. ------