# Tutorial for building models with Notebook Instances
This Get Started tutorial walks you through how to create a SageMaker notebook instance, open a Jupyter notebook with a preconfigured kernel with the Conda environment for machine learning, and start a SageMaker AI session to run an end-to-end ML cycle. You'll learn how to save a dataset to a default Amazon S3 bucket automatically paired with the SageMaker AI session, submit a training job of an ML model to Amazon EC2, and deploy the trained model for prediction by hosting or batch inferencing through Amazon EC2.
This tutorial explicitly shows a complete ML flow of training the XGBoost model from the SageMaker AI built-in model pool. You use the [US Adult Census dataset](https://archive.ics.uci.edu/ml/datasets/adult), and you evaluate the performance of the trained SageMaker AI XGBoost model on predicting individuals' income.
+ [SageMaker AI XGBoost](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) – The [XGBoost](https://xgboost.readthedocs.io/en/latest/) model is adapted to the SageMaker AI environment and preconfigured as Docker containers. SageMaker AI provides a suite of [built-in algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html) that are prepared for using SageMaker AI features. To learn more about what ML algorithms are adapted to SageMaker AI, see [Choose an Algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/algorithms-choose.html) and [Use Amazon SageMaker Built-in Algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html). For the SageMaker AI built-in algorithm API operations, see [First-Party Algorithms](https://sagemaker.readthedocs.io/en/stable/algorithms/index.html) in the [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable).
+ [Adult Census dataset](https://archive.ics.uci.edu/ml/datasets/adult) – The dataset from the [1994 Census bureau database](http://www.census.gov/en.html) by Ronny Kohavi and Barry Becker (Data Mining and Visualization, Silicon Graphics). The SageMaker AI XGBoost model is trained using this dataset to predict if an individual makes over \$150,000 a year or less.
**Topics**
+ [Create an Amazon SageMaker Notebook Instance for the tutorial](gs-setup-working-env.md)
+ [Create a Jupyter notebook in the SageMaker notebook instance](ex1-prepare.md)
+ [Prepare a dataset](ex1-preprocess-data.md)
+ [Train a Model](ex1-train-model.md)
+ [Deploy the model to Amazon EC2](ex1-model-deployment.md)
+ [Evaluate the model](ex1-test-model.md)
+ [Clean up Amazon SageMaker notebook instance resources](ex1-cleanup.md)
# Create an Amazon SageMaker Notebook Instance for the tutorial
**Important**
Custom IAM policies that allow Amazon SageMaker Studio or Amazon SageMaker Studio Classic to create Amazon SageMaker resources must also grant permissions to add tags to those resources. The permission to add tags to resources is required because Studio and Studio Classic automatically tag any resources they create. If an IAM policy allows Studio and Studio Classic to create resources but does not allow tagging, "AccessDenied" errors can occur when trying to create resources. For more information, see [Provide permissions for tagging SageMaker AI resources](security_iam_id-based-policy-examples.md#grant-tagging-permissions).
[AWS managed policies for Amazon SageMaker AI](security-iam-awsmanpol.md) that give permissions to create SageMaker resources already include permissions to add tags while creating those resources.
An Amazon SageMaker notebook instance is a fully-managed machine learning (ML) Amazon Elastic Compute Cloud (Amazon EC2) compute instance. An Amazon SageMaker notebook instance runs the Jupyter Notebook application. Use the notebook instance to create and manage Jupyter notebooks for preprocessing data, train ML models, and deploy ML models.
**To create a SageMaker notebook instance**
![\[Animated screenshot that shows how to create a SageMaker notebook instance.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-create-instance.gif)
1. Open the Amazon SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/).
1. Choose **Notebook instances**, and then choose **Create notebook instance**.
1. On the **Create notebook instance** page, provide the following information (if a field is not mentioned, leave the default values):
1. For **Notebook instance name**, type a name for your notebook instance.
1. For **Notebook Instance type**, choose `ml.t2.medium`. This is the least expensive instance type that notebook instances support, and is enough for this exercise. If a `ml.t2.medium` instance type isn't available in your current AWS Region, choose `ml.t3.medium`.
1. For **Platform Identifier**, choose a platform type to create the notebook instance on. This platform type defines the Operating System and the JupyterLab version that your notebook instance is created with. The latest and recommended version is `notebook-al2023-v1`, for an Amazon Linux 2023 notebook instance. For information about platform identifier types, see [AL2023 notebook instances](nbi-al2023.md) and [Amazon Linux 2 notebook instances](nbi-al2.md). For information about JupyterLab versions, see [JupyterLab versioning](nbi-jl.md).
1. For **IAM role**, choose **Create a new role**, and then choose **Create role**. This IAM role automatically gets permissions to access any S3 bucket that has `sagemaker` in the name. It gets these permissions through the `AmazonSageMakerFullAccess` policy, which SageMaker AI attaches to the role.
**Note**
If you want to grant the IAM role permission to access S3 buckets without `sagemaker` in the name, you need to attach the `S3FullAccess` policy. You can also limit the permissions to specific S3 buckets to the IAM role. For more information and examples of adding bucket policies to the IAM role, see [Bucket Policy Examples](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html).
1. Choose **Create notebook instance**.
In a few minutes, SageMaker AI launches a notebook instance and attaches a 5 GB of Amazon EBS storage volume to it. The notebook instance has a preconfigured Jupyter notebook server, SageMaker AI and AWS SDK libraries, and a set of Anaconda libraries.
For more information about creating a SageMaker notebook instance, see [Create a Notebook Instance](https://docs.aws.amazon.com/sagemaker/latest/dg/howitworks-create-ws.html).
## (Optional) Change SageMaker Notebook Instance Settings
To change the ML compute instance type or the size of the Amazon EBS storage of a SageMaker AI notebook instance, edit the notebook instance settings.
**To change and update the SageMaker Notebook instance type and the EBS volume**
1. On the **Notebook instances** page in the SageMaker AI console, choose your notebook instance.
1. Choose **Actions**, choose **Stop**, and then wait until the notebook instance fully stops.
1. After the notebook instance status changes to **Stopped**, choose **Actions**, and then choose **Update settings**.
![\[Animated screenshot that shows how to update SageMaker notebook instance settings.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-update-instance.gif)
1. For **Notebook instance type**, choose a different ML instance type.
1. For **Volume size in GB**, type a different integer to specify a new EBS volume size.
**Note**
EBS storage volumes are encrypted, so SageMaker AI can't determine the amount of available free space on the volume. Because of this, you can increase the volume size when you update a notebook instance, but you can't decrease the volume size. If you want to decrease the size of the ML storage volume in use, create a new notebook instance with the desired size.
1. At the bottom of the page, choose **Update notebook instance**.
1. When the update is complete, **Start** the notebook instance with the new settings.
For more information about updating SageMaker notebook instance settings, see [Update a Notebook Instance](https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-update.html).
## (Optional) Advanced Settings for SageMaker Notebook Instances
The following tutorial video shows how to set up and use SageMaker notebook instances through the SageMaker AI console. It includes advanced options, such as SageMaker AI lifecycle configuration and importing GitHub repositories. (Length: 26:04)
[](http://www.youtube.com/watch?v=https://www.youtube.com/embed/X5CLunIzj3U)
For complete documentation about SageMaker notebook instance, see [Use Amazon SageMaker notebook Instances](https://docs.aws.amazon.com/sagemaker/latest/dg/nbi.html).
# Create a Jupyter notebook in the SageMaker notebook instance
**Important**
Custom IAM policies that allow Amazon SageMaker Studio or Amazon SageMaker Studio Classic to create Amazon SageMaker resources must also grant permissions to add tags to those resources. The permission to add tags to resources is required because Studio and Studio Classic automatically tag any resources they create. If an IAM policy allows Studio and Studio Classic to create resources but does not allow tagging, "AccessDenied" errors can occur when trying to create resources. For more information, see [Provide permissions for tagging SageMaker AI resources](security_iam_id-based-policy-examples.md#grant-tagging-permissions).
[AWS managed policies for Amazon SageMaker AI](security-iam-awsmanpol.md) that give permissions to create SageMaker resources already include permissions to add tags while creating those resources.
To start scripting for training and deploying your model, create a Jupyter notebook in the SageMaker notebook instance. Using the Jupyter notebook, you can run machine learning (ML) experiments for training and inference while using SageMaker AI features and the AWS infrastructure.
**To create a Jupyter notebook**
![\[Animated screenshot that shows how to create a Jupyter notebook in the SageMaker AI notebook instance.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-create-notebook.gif)
1. Open the notebook instance as follows:
1. Sign in to the SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/).
1. On the **Notebook instances** page, open your notebook instance by choosing either:
+ **Open JupyterLab** for the JupyterLab interface
+ **Open Jupyter** for the classic Jupyter view
**Note**
If the notebook instance status shows **Pending** in the **Status** column, your notebook instance is still being created. The status will change to **InService** when the notebook instance is ready to use.
1. Create a notebook as follows:
+ If you opened the notebook in the JupyterLab view, on the **File** menu, choose **New**, and then choose **Notebook**. For **Select Kernel**, choose **conda\$1python3**. This preinstalled environment includes the default Anaconda installation and Python 3.
+ If you opened the notebook in the classic Jupyter view, on the **Files** tab, choose **New**, and then choose **conda\$1python3**. This preinstalled environment includes the default Anaconda installation and Python 3.
1. Save the notebooks as follows:
+ In the JupyterLab view, choose **File**, choose **Save Notebook As...**, and then rename the notebook.
+ In the Jupyter classic view, choose **File**, choose **Save as...**, and then rename the notebook.
# Prepare a dataset
In this step, you load the [Adult Census dataset](https://archive.ics.uci.edu/ml/datasets/adult) to your notebook instance using the SHAP (SHapley Additive exPlanations) Library, review the dataset, transform it, and upload it to Amazon S3. SHAP is a game theoretic approach to explain the output of any machine learning model. For more information about SHAP, see [Welcome to the SHAP documentation](https://shap.readthedocs.io/en/latest/).
To run the following example, paste the sample code into a cell in your notebook instance.
## Load Adult Census Dataset Using SHAP
Using the SHAP library, import the Adult Census dataset as shown following:
```
import shap
X, y = shap.datasets.adult()
X_display, y_display = shap.datasets.adult(display=True)
feature_names = list(X.columns)
feature_names
```
**Note**
If the current Jupyter kernel does not have the SHAP library, install it by running the following `conda` command:
```
%conda install -c conda-forge shap
```
If you're using JupyterLab, you must manually refresh the kernel after the installation and updates have completed. Run the following IPython script to shut down the kernel (the kernel will restart automatically):
```
import IPython
IPython.Application.instance().kernel.do_shutdown(True)
```
The `feature_names` list object should return the following list of features:
```
['Age',
'Workclass',
'Education-Num',
'Marital Status',
'Occupation',
'Relationship',
'Race',
'Sex',
'Capital Gain',
'Capital Loss',
'Hours per week',
'Country']
```
**Tip**
If you're starting with unlabeled data, you can use Amazon SageMaker Ground Truth to create a data labeling workflow in minutes. To learn more, see [Label Data](https://docs.aws.amazon.com/sagemaker/latest/dg/data-label.html).
## Overview the Dataset
Run the following script to display the statistical overview of the dataset and histograms of the numeric features.
```
display(X.describe())
hist = X.hist(bins=30, sharey=True, figsize=(20, 10))
```
![\[Overview of the Adult Census dataset.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-prepare-data-1.png)
**Tip**
If you want to use a dataset that needs to be cleaned and transformed, you can simplify and streamline data preprocessing and feature engineering using Amazon SageMaker Data Wrangler. To learn more, see [Prepare ML Data with Amazon SageMaker Data Wrangler](https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler.html).
## Split the Dataset into Train, Validation, and Test Datasets
Using Sklearn, split the dataset into a training set and a test set. The training set is used to train the model, while the test set is used to evaluate the performance of the final trained model. The dataset is randomly sorted with the fixed random seed: 80 percent of the dataset for training set and 20 percent of it for a test set.
```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
X_train_display = X_display.loc[X_train.index]
```
Split the training set to separate out a validation set. The validation set is used to evaluate the performance of the trained model while tuning the model's hyperparameters. 75 percent of the training set becomes the final training set, and the rest is the validation set.
```
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=1)
X_train_display = X_display.loc[X_train.index]
X_val_display = X_display.loc[X_val.index]
```
Using the pandas package, explicitly align each dataset by concatenating the numeric features with the true labels.
```
import pandas as pd
train = pd.concat([pd.Series(y_train, index=X_train.index,
name='Income>50K', dtype=int), X_train], axis=1)
validation = pd.concat([pd.Series(y_val, index=X_val.index,
name='Income>50K', dtype=int), X_val], axis=1)
test = pd.concat([pd.Series(y_test, index=X_test.index,
name='Income>50K', dtype=int), X_test], axis=1)
```
Check if the dataset is split and structured as expected:
```
train
```
![\[The example training dataset.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-prepare-data-2-train.png)
```
validation
```
![\[The example validation dataset.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-prepare-data-2-validation.png)
```
test
```
![\[The example test dataset.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-prepare-data-2-test.png)
## Convert the Train and Validation Datasets to CSV Files
Convert the `train` and `validation` dataframe objects to CSV files to match the input file format for the XGBoost algorithm.
```
# Use 'csv' format to store the data
# The first column is expected to be the output column
train.to_csv('train.csv', index=False, header=False)
validation.to_csv('validation.csv', index=False, header=False)
```
## Upload the Datasets to Amazon S3
Using the SageMaker AI and Boto3, upload the training and validation datasets to the default Amazon S3 bucket. The datasets in the S3 bucket will be used by a compute-optimized SageMaker instance on Amazon EC2 for training.
The following code sets up the default S3 bucket URI for your current SageMaker AI session, creates a new `demo-sagemaker-xgboost-adult-income-prediction` folder, and uploads the training and validation datasets to the `data` subfolder.
```
import sagemaker, boto3, os
bucket = sagemaker.Session().default_bucket()
prefix = "demo-sagemaker-xgboost-adult-income-prediction"
boto3.Session().resource('s3').Bucket(bucket).Object(
os.path.join(prefix, 'data/train.csv')).upload_file('train.csv')
boto3.Session().resource('s3').Bucket(bucket).Object(
os.path.join(prefix, 'data/validation.csv')).upload_file('validation.csv')
```
Run the following AWS CLI to check if the CSV files are successfully uploaded to the S3 bucket.
```
! aws s3 ls {bucket}/{prefix}/data --recursive
```
This should return the following output:
![\[Output of the CLI command to check the datasets in the S3 bucket.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-prepare-data-3.png)
# Train a Model
In this step, you choose a training algorithm and run a training job for the model. The [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable) provides framework estimators and generic estimators to train your model while orchestrating the machine learning (ML) lifecycle accessing the SageMaker AI features for training and the AWS infrastructures, such as Amazon Elastic Container Registry (Amazon ECR), Amazon Elastic Compute Cloud (Amazon EC2), Amazon Simple Storage Service (Amazon S3). For more information about SageMaker AI built-in framework estimators, see [Frameworks](https://sagemaker.readthedocs.io/en/stable/frameworks/index.html)in the [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable) documentation. For more information about built-in algorithms, see [Built-in algorithms and pretrained models in Amazon SageMaker](algos.md).
**Topics**
+ [Choose the Training Algorithm](#ex1-train-model-select-algorithm)
+ [Create and Run a Training Job](#ex1-train-model-sdk)
## Choose the Training Algorithm
To choose the right algorithm for your dataset, you typically need to evaluate different models to find the most suitable models to your data. For simplicity, the SageMaker AI [XGBoost algorithm with Amazon SageMaker AI](xgboost.md) built-in algorithm is used throughout this tutorial without the pre-evaluation of models.
**Tip**
If you want SageMaker AI to find an appropriate model for your tabular dataset, use Amazon SageMaker Autopilot that automates a machine learning solution. For more information, see [SageMaker Autopilot](autopilot-automate-model-development.md).
## Create and Run a Training Job
After you figured out which model to use, start constructing a SageMaker AI estimator for training. This tutorial uses the XGBoost built-in algorithm for the SageMaker AI generic estimator.
**To run a model training job**
1. Import the [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable) and start by retrieving the basic information from your current SageMaker AI session.
```
import sagemaker
region = sagemaker.Session().boto_region_name
print("AWS Region: {}".format(region))
role = sagemaker.get_execution_role()
print("RoleArn: {}".format(role))
```
This returns the following information:
+ `region` – The current AWS Region where the SageMaker AI notebook instance is running.
+ `role` – The IAM role used by the notebook instance.
**Note**
Check the SageMaker Python SDK version by running `sagemaker.__version__`. This tutorial is based on `sagemaker>=2.20`. If the SDK is outdated, install the latest version by running the following command:
```
! pip install -qU sagemaker
```
If you run this installation in your exiting SageMaker Studio or notebook instances, you need to manually refresh the kernel to finish applying the version update.
1. Create an XGBoost estimator using the `sagemaker.estimator.Estimator` class. In the following example code, the XGBoost estimator is named `xgb_model`.
```
from sagemaker.debugger import Rule, ProfilerRule, rule_configs
from sagemaker.session import TrainingInput
s3_output_location='s3://{}/{}/{}'.format(bucket, prefix, 'xgboost_model')
container=sagemaker.image_uris.retrieve("xgboost", region, "1.2-1")
print(container)
xgb_model=sagemaker.estimator.Estimator(
image_uri=container,
role=role,
instance_count=1,
instance_type='ml.m4.xlarge',
volume_size=5,
output_path=s3_output_location,
sagemaker_session=sagemaker.Session(),
rules=[
Rule.sagemaker(rule_configs.create_xgboost_report()),
ProfilerRule.sagemaker(rule_configs.ProfilerReport())
]
)
```
To construct the SageMaker AI estimator, specify the following parameters:
+ `image_uri` – Specify the training container image URI. In this example, the SageMaker AI XGBoost training container URI is specified using `sagemaker.image_uris.retrieve`.
+ `role` – The AWS Identity and Access Management (IAM) role that SageMaker AI uses to perform tasks on your behalf (for example, reading training results, call model artifacts from Amazon S3, and writing training results to Amazon S3).
+ `instance_count` and `instance_type` – The type and number of Amazon EC2 ML compute instances to use for model training. For this training exercise, you use a single `ml.m4.xlarge` instance, which has 4 CPUs, 16 GB of memory, an Amazon Elastic Block Store (Amazon EBS) storage, and a high network performance. For more information about EC2 compute instance types, see [Amazon EC2 Instance Types](https://aws.amazon.com/ec2/instance-types/). For more information about billing, see [Amazon SageMaker pricing](https://aws.amazon.com/sagemaker/pricing/).
+ `volume_size` – The size, in GB, of the EBS storage volume to attach to the training instance. This must be large enough to store training data if you use `File` mode (`File` mode is on by default). If you don't specify this parameter, its value defaults to 30.
+ `output_path` – The path to the S3 bucket where SageMaker AI stores the model artifact and training results.
+ `sagemaker_session` – The session object that manages interactions with SageMaker API operations and other AWS service that the training job uses.
+ `rules` – Specify a list of SageMaker Debugger built-in rules. In this example, the `create_xgboost_report()` rule creates an XGBoost report that provides insights into the training progress and results, and the `ProfilerReport()` rule creates a report regarding the EC2 compute resource utilization. For more information, see [SageMaker Debugger interactive report for XGBoost](debugger-report-xgboost.md).
**Tip**
If you want to run distributed training of large sized deep learning models, such as convolutional neural networks (CNN) and natural language processing (NLP) models, use SageMaker AI Distributed for data parallelism or model parallelism. For more information, see [Distributed training in Amazon SageMaker AI](distributed-training.md).
1. Set the hyperparameters for the XGBoost algorithm by calling the `set_hyperparameters` method of the estimator. For a complete list of XGBoost hyperparameters, see [XGBoost hyperparameters](xgboost_hyperparameters.md).
```
xgb_model.set_hyperparameters(
max_depth = 5,
eta = 0.2,
gamma = 4,
min_child_weight = 6,
subsample = 0.7,
objective = "binary:logistic",
num_round = 1000
)
```
**Tip**
You can also tune the hyperparameters using the SageMaker AI hyperparameter optimization feature. For more information, see [Automatic model tuning with SageMaker AI](automatic-model-tuning.md).
1. Use the `TrainingInput` class to configure a data input flow for training. The following example code shows how to configure `TrainingInput` objects to use the training and validation datasets you uploaded to Amazon S3 in the [Split the Dataset into Train, Validation, and Test Datasets](ex1-preprocess-data.md#ex1-preprocess-data-transform) section.
```
from sagemaker.session import TrainingInput
train_input = TrainingInput(
"s3://{}/{}/{}".format(bucket, prefix, "data/train.csv"), content_type="csv"
)
validation_input = TrainingInput(
"s3://{}/{}/{}".format(bucket, prefix, "data/validation.csv"), content_type="csv"
)
```
1. To start model training, call the estimator's `fit` method with the training and validation datasets. By setting `wait=True`, the `fit` method displays progress logs and waits until training is complete.
```
xgb_model.fit({"train": train_input, "validation": validation_input}, wait=True)
```
For more information about model training, see [Train a Model with Amazon SageMaker](how-it-works-training.md). This tutorial training job might take up to 10 minutes.
After the training job has done, you can download an XGBoost training report and a profiling report generated by SageMaker Debugger. The XGBoost training report offers you insights into the training progress and results, such as the loss function with respect to iteration, feature importance, confusion matrix, accuracy curves, and other statistical results of training. For example, you can find the following loss curve from the XGBoost training report which clearly indicates that there is an overfitting problem.
![\[The chart in the XGBoost training report.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-train-loss-curve-validation-overfitting.png)
Run the following code to specify the S3 bucket URI where the Debugger training reports are generated and check if the reports exist.
```
rule_output_path = xgb_model.output_path + "/" + xgb_model.latest_training_job.job_name + "/rule-output"
! aws s3 ls {rule_output_path} --recursive
```
Download the Debugger XGBoost training and profiling reports to the current workspace:
```
! aws s3 cp {rule_output_path} ./ --recursive
```
Run the following IPython script to get the file link of the XGBoost training report:
```
from IPython.display import FileLink, FileLinks
display("Click link below to view the XGBoost Training report", FileLink("CreateXgboostReport/xgboost_report.html"))
```
The following IPython script returns the file link of the Debugger profiling report that shows summaries and details of the EC2 instance resource utilization, system bottleneck detection results, and python operation profiling results:
```
profiler_report_name = [rule["RuleConfigurationName"]
for rule in xgb_model.latest_training_job.rule_job_summary()
if "Profiler" in rule["RuleConfigurationName"]][0]
profiler_report_name
display("Click link below to view the profiler report", FileLink(profiler_report_name+"/profiler-output/profiler-report.html"))
```
**Tip**
If the HTML reports do not render plots in the JupyterLab view, you must choose **Trust HTML** at the top of the reports.
To identify training issues, such as overfitting, vanishing gradients, and other problems that prevents your model from converging, use SageMaker Debugger and take automated actions while prototyping and training your ML models. For more information, see [Amazon SageMaker Debugger](train-debugger.md). To find a complete analysis of model parameters, see the [Explainability with Amazon SageMaker Debugger](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-debugger/xgboost_census_explanations/xgboost-census-debugger-rules.html#Explainability-with-Amazon-SageMaker-Debugger) example notebook.
You now have a trained XGBoost model. SageMaker AI stores the model artifact in your S3 bucket. To find the location of the model artifact, run the following code to print the model\$1data attribute of the `xgb_model` estimator:
```
xgb_model.model_data
```
**Tip**
To measure biases that can occur during each stage of the ML lifecycle (data collection, model training and tuning, and monitoring of ML models deployed for prediction), use SageMaker Clarify. For more information, see [Model Explainability](clarify-model-explainability.md). For an end-to-end example, see the [Fairness and Explainability with SageMaker Clarify](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-clarify/fairness_and_explainability/fairness_and_explainability.html) example notebook.
# Deploy the model to Amazon EC2
To get predictions, deploy your model to Amazon EC2 using Amazon SageMaker AI.
**Topics**
+ [Deploy the Model to SageMaker AI Hosting Services](#ex1-deploy-model)
+ [(Optional) Use SageMaker AI Predictor to Reuse the Hosted Endpoint](#ex1-deploy-model-sdk-use-endpoint)
+ [(Optional) Make Prediction with Batch Transform](#ex1-batch-transform)
## Deploy the Model to SageMaker AI Hosting Services
To host a model through Amazon EC2 using Amazon SageMaker AI, deploy the model that you trained in [Create and Run a Training Job](ex1-train-model.md#ex1-train-model-sdk) by calling the `deploy` method of the `xgb_model` estimator. When you call the `deploy` method, you must specify the number and type of EC2 ML instances that you want to use for hosting an endpoint.
```
import sagemaker
from sagemaker.serializers import CSVSerializer
xgb_predictor=xgb_model.deploy(
initial_instance_count=1,
instance_type='ml.t2.medium',
serializer=CSVSerializer()
)
```
+ `initial_instance_count` (int) – The number of instances to deploy the model.
+ `instance_type` (str) – The type of instances that you want to operate your deployed model.
+ `serializer` (int) – Serialize input data of various formats (a NumPy array, list, file, or buffer) to a CSV-formatted string. We use this because the XGBoost algorithm accepts input files in CSV format.
The `deploy` method creates a deployable model, configures the SageMaker AI hosting services endpoint, and launches the endpoint to host the model. For more information, see the [SageMaker AI generic Estimator's deploy class method](https://sagemaker.readthedocs.io/en/stable/estimators.html#sagemaker.estimator.Estimator.deploy) in the [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable). To retrieve the name of endpoint that's generated by the `deploy` method, run the following code:
```
xgb_predictor.endpoint_name
```
This should return the endpoint name of the `xgb_predictor`. The format of the endpoint name is `"sagemaker-xgboost-YYYY-MM-DD-HH-MM-SS-SSS"`. This endpoint stays active in the ML instance, and you can make instantaneous predictions at any time unless you shut it down later. Copy this endpoint name and save it to reuse and make real-time predictions elsewhere in SageMaker Studio or SageMaker AI notebook instances.
**Tip**
To learn more about compiling and optimizing your model for deployment to Amazon EC2 instances or edge devices, see [Compile and Deploy Models with Neo](https://docs.aws.amazon.com/sagemaker/latest/dg/neo.html).
## (Optional) Use SageMaker AI Predictor to Reuse the Hosted Endpoint
After you deploy the model to an endpoint, you can set up a new SageMaker AI predictor by pairing the endpoint and continuously make real-time predictions in any other notebooks. The following example code shows how to use the SageMaker AI Predictor class to set up a new predictor object using the same endpoint. Re-use the endpoint name that you used for the `xgb_predictor`.
```
import sagemaker
xgb_predictor_reuse=sagemaker.predictor.Predictor(
endpoint_name="sagemaker-xgboost-YYYY-MM-DD-HH-MM-SS-SSS",
sagemaker_session=sagemaker.Session(),
serializer=sagemaker.serializers.CSVSerializer()
)
```
The `xgb_predictor_reuse` Predictor behaves exactly the same as the original `xgb_predictor`. For more information, see the [SageMaker AI Predictor](https://sagemaker.readthedocs.io/en/stable/predictors.html#sagemaker.predictor.RealTimePredictor) class in the [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable).
## (Optional) Make Prediction with Batch Transform
Instead of hosting an endpoint in production, you can run a one-time batch inference job to make predictions on a test dataset using the SageMaker AI batch transform. After your model training has completed, you can extend the estimator to a `transformer` object, which is based on the [SageMaker AI Transformer](https://sagemaker.readthedocs.io/en/stable/api/inference/transformer.html) class. The batch transformer reads in input data from a specified S3 bucket and makes predictions.
**To run a batch transform job**
1. Run the following code to convert the feature columns of the test dataset to a CSV file and uploads to the S3 bucket:
```
X_test.to_csv('test.csv', index=False, header=False)
boto3.Session().resource('s3').Bucket(bucket).Object(
os.path.join(prefix, 'test/test.csv')).upload_file('test.csv')
```
1. Specify S3 bucket URIs of input and output for the batch transform job as shown following:
```
# The location of the test dataset
batch_input = 's3://{}/{}/test'.format(bucket, prefix)
# The location to store the results of the batch transform job
batch_output = 's3://{}/{}/batch-prediction'.format(bucket, prefix)
```
1. Create a transformer object specifying the minimal number of parameters: the `instance_count` and `instance_type` parameters to run the batch transform job, and the `output_path` to save prediction data as shown following:
```
transformer = xgb_model.transformer(
instance_count=1,
instance_type='ml.m4.xlarge',
output_path=batch_output
)
```
1. Initiate the batch transform job by executing the `transform()` method of the `transformer` object as shown following:
```
transformer.transform(
data=batch_input,
data_type='S3Prefix',
content_type='text/csv',
split_type='Line'
)
transformer.wait()
```
1. When the batch transform job is complete, SageMaker AI creates the `test.csv.out` prediction data saved in the `batch_output` path, which should be in the following format: `s3://sagemaker--111122223333/demo-sagemaker-xgboost-adult-income-prediction/batch-prediction`. Run the following AWS CLI to download the output data of the batch transform job:
```
! aws s3 cp {batch_output} ./ --recursive
```
This should create the `test.csv.out` file under the current working directory. You'll be able to see the float values that are predicted based on the logistic regression of the XGBoost training job.
# Evaluate the model
Now that you have trained and deployed a model using Amazon SageMaker AI, evaluate the model to ensure that it generates accurate predictions on new data. For model evaluation, use the test dataset that you created in [Prepare a dataset](ex1-preprocess-data.md).
## Evaluate the Model Deployed to SageMaker AI Hosting Services
To evaluate the model and use it in production, invoke the endpoint with the test dataset and check whether the inferences you get returns a target accuracy you want to achieve.
**To evaluate the model**
1. Set up the following function to predict each line of the test set. In the following example code, the `rows` argument is to specify the number of lines to predict at a time. You can change the value of it to perform a batch inference that fully utilizes the instance's hardware resource.
```
import numpy as np
def predict(data, rows=1000):
split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))
predictions = ''
for array in split_array:
predictions = ','.join([predictions, xgb_predictor.predict(array).decode('utf-8')])
return np.fromstring(predictions[1:], sep=',')
```
1. Run the following code to make predictions of the test dataset and plot a histogram. You need to take only the feature columns of the test dataset, excluding the 0th column for the actual values.
```
import matplotlib.pyplot as plt
predictions=predict(test.to_numpy()[:,1:])
plt.hist(predictions)
plt.show()
```
![\[A histogram of predicted values.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-eval-predicted-values-histogram.png)
1. The predicted values are float type. To determine `True` or `False` based on the float values, you need to set a cutoff value. As shown in the following example code, use the Scikit-learn library to return the output confusion metrics and classification report with a cutoff of 0.5.
```
import sklearn
cutoff=0.5
print(sklearn.metrics.confusion_matrix(test.iloc[:, 0], np.where(predictions > cutoff, 1, 0)))
print(sklearn.metrics.classification_report(test.iloc[:, 0], np.where(predictions > cutoff, 1, 0)))
```
This should return the following confusion matrix:
![\[An example of confusion matrix and statistics after getting the inference of the deployed model.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-evaluate-confusion-matrix.png)
1. To find the best cutoff with the given test set, compute the log loss function of the logistic regression. The log loss function is defined as the negative log-likelihood of a logistic model that returns prediction probabilities for its ground truth labels. The following example code numerically and iteratively calculates the log loss values (`-(y*log(p)+(1-y)log(1-p)`), where `y` is the true label and `p` is a probability estimate of the corresponding test sample. It returns a log loss versus cutoff graph.
```
import matplotlib.pyplot as plt
cutoffs = np.arange(0.01, 1, 0.01)
log_loss = []
for c in cutoffs:
log_loss.append(
sklearn.metrics.log_loss(test.iloc[:, 0], np.where(predictions > c, 1, 0))
)
plt.figure(figsize=(15,10))
plt.plot(cutoffs, log_loss)
plt.xlabel("Cutoff")
plt.ylabel("Log loss")
plt.show()
```
This should return the following log loss curve.
![\[Example following log loss curve.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/get-started-ni/gs-ni-evaluate-logloss-vs-cutoff.png)
1. Find the minimum points of the error curve using the NumPy `argmin` and `min` functions:
```
print(
'Log loss is minimized at a cutoff of ', cutoffs[np.argmin(log_loss)],
', and the log loss value at the minimum is ', np.min(log_loss)
)
```
This should return: `Log loss is minimized at a cutoff of 0.53, and the log loss value at the minimum is 4.348539186773897`.
Instead of computing and minimizing the log loss function, you can estimate a cost function as an alternative. For example, if you want to train a model to perform a binary classification for a business problem such as a customer churn prediction problem, you can set weights to the elements of confusion matrix and calculate the cost function accordingly.
You have now trained, deployed, and evaluated your first model in SageMaker AI.
**Tip**
To monitor model quality, data quality, and bias drift, use Amazon SageMaker Model Monitor and SageMaker AI Clarify. To learn more, see [Amazon SageMaker Model Monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html), [Monitor Data Quality](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-data-quality.html), [Monitor Model Quality](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality.html), [Monitor Bias Drift](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-model-monitor-bias-drift.html), and [Monitor Feature Attribution Drift](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-model-monitor-feature-attribution-drift.html).
**Tip**
To get human review of low confidence ML predictions or a random sample of predictions, use Amazon Augmented AI human review workflows. For more information, see [Using Amazon Augmented AI for Human Review](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-use-augmented-ai-a2i-human-review-loops.html).
# Clean up Amazon SageMaker notebook instance resources
To avoid incurring unnecessary charges, use the AWS Management Console to delete the endpoints and resources that you created while running the exercises.
**Note**
Training jobs and logs cannot be deleted and are retained indefinitely.
**Note**
If you plan to explore other exercises in this guide, you might want to keep some of these resources, such as your notebook instance, S3 bucket, and IAM role.
1. Open the Amazon SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/) and delete the following resources:
+ The endpoint. Deleting the endpoint also deletes the ML compute instance or instances that support it.
1. Under **Inference**, choose **Endpoints**.
1. Choose the endpoint that you created in the example, choose **Actions**, and then choose **Delete**.
+ The endpoint configuration.
1. Under **Inference**, choose **Endpoint configurations**.
1. Choose the endpoint configuration that you created in the example, choose **Actions**, and then choose **Delete**.
+ The model.
1. Under **Inference**, choose **Models**.
1. Choose the model that you created in the example, choose **Actions**, and then choose **Delete**.
+ The notebook instance. Before deleting the notebook instance, stop it.
1. Under **Notebook**, choose **Notebook instances**.
1. Choose the notebook instance that you created in the example, choose **Actions**, and then choose **Stop**. The notebook instance takes several minutes to stop. When the **Status** changes to **Stopped**, move on to the next step.
1. Choose **Actions**, and then choose **Delete**.
1. Open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/), and then delete the bucket that you created for storing model artifacts and the training dataset.
1. Open the Amazon CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/), and then delete all of the log groups that have names starting with `/aws/sagemaker/`.