# AWS Deep Learning AMI GPU TensorFlow 2.16 (Amazon Linux 2)
<a name="aws-deep-learning-ami-gpu-tensorflow-2.16-amazon-linux-2"></a>

For help getting started, see [Getting started with DLAMI](getting-started.md).

#### AMI Name Format
<a name="name-gpu-tensorflow-2.16-amazon-linux-2"></a>
+ Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) \$1\$1YYYY-MM-DD\$1
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) \$1\$1YYYY-MM-DD\$1

#### Supported EC2 instances
<a name="instances-gpu-tensorflow-2.16-amazon-linux-2"></a>
+ Please refer to [Important changes to DLAMI](important-changes.md).
+ Deep Learning with OSS Nvidia Driver supports G4dn, G5, G6, Gr6, G6e, P4d, P4de, P5, P5e, P5en.
+ Deep Learning with Proprietary Nvidia Driver supports G3 (G3.16x not supported), P3, P3dn

#### The AMI includes the following:
<a name="contents-gpu-tensorflow-2.16-amazon-linux-2"></a>
+ **Supported AWS Service**: EC2
+ **Operating System**: Amazon Linux 2
+ **Compute Architecture**: x86
+ **Python**: /opt/tensorflow/bin/python3.10
+ **TensorFlow version**: 2.16
+ **NVIDIA Driver**:
  + OSS Nvidia driver: 550.144.03
  + Proprietary Nvidia driver: 550.144.03
+ **NVIDIA CUDA12 stack**:
  + CUDA, NCCL and cuDDN installation path: /usr/local/cuda-12.2/
+ **EFA Installer**: 1.34.0
+ **AWS CLI v2** as aws2 and **AWS CLI v1** as aws
+ **EBS volume type**: gp3
+ **Query AMI-ID with SSM Parameter (example region is us-east-1)**:
  + **OSS Nvidia Driver**:

    ```
    aws ssm get-parameter --name /aws/service/deeplearning/ami/x86_64/oss-nvidia-driver-gpu-tensorflow-2.16-amazon-linux-2/latest/ami-id --region us-east-1 --query "Parameter.Value" --output text
    ```
  + **Proprietary Nvidia Driver**:

    ```
    aws ssm get-parameter --name /aws/service/deeplearning/ami/x86_64/proprietary-nvidia-driver-gpu-tensorflow-2.16-amazon-linux-2/latest/ami-id --region us-east-1 --query "Parameter.Value" --output text
    ```
+ **Query AMI-ID with AWSCLI (example region is us-east-1)**:
  + **OSS Nvidia Driver**:

    ```
    aws ec2 describe-images --region us-east-1 --owners amazon --filters 'Name=name,Values=Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) ????????' 'Name=state,Values=available' --query 'reverse(sort_by(Images, &CreationDate))[:1].ImageId' --output text
    ```
  + **Proprietary Nvidia Driver**:

    ```
    aws ec2 describe-images --region us-east-1 --owners amazon --filters 'Name=name,Values=Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) ????????' 'Name=state,Values=available' --query 'reverse(sort_by(Images, &CreationDate))[:1].ImageId' --output text
    ```

#### Notice
<a name="notices-gpu-tensorflow-2.16-amazon-linux-2"></a>

##### NVIDIA Container Toolkit 1.17.4
<a name="nvidia-container-toolkit-gpu-tensorflow-2.16-amazon-linux-2"></a>

In Container Toolkit version 1.17.4 the mounting of CUDA compat libraries is now disabled. In order to ensure compatibility with multiple CUDA versions on container workflows, please ensure you update your LD\$1LIBRARY\$1PATH to include your CUDA compatibility libraries as shown in under the "If you use a CUDA compatibility layer" tutorial here - [https://docs.aws.amazon.com/sagemaker/latest/dg/inference-gpu-drivers.html\$1collapsible-cuda-compat](https://docs.aws.amazon.com/sagemaker/latest/dg/inference-gpu-drivers.html#collapsible-cuda-compat)

##### Future TensorFlow Operating System Updates
<a name="future-os-updates-gpu-tensorflow-2.16-amazon-linux-2"></a>

TensorFlow 2.16 will be the last DLAMI that utilizes the Ubuntu 20.04 Operating System. Starting with TensorFlow 2.17 and above, DLAMIs will begin to utilize Ubuntu 22.04 as the base Operating System. For customers looking to upgrade to these new versions, please ensure your workflows are ready for this upgrade.

##### Keras version pinned to 2.0 instead of 3.0
<a name="keras-version-gpu-tensorflow-2.16-amazon-linux-2"></a>

With the latest TF2.16 release, Keras has been upgraded from major version 2 to major version 3.0. This Keras version is a complete rewrite of the Keras package (please see the [Keras 3 documentation](https://keras.io/keras_3/) for more information). To ensure compatibility with customer workflows, we have pinned Keras versions to 2.0 using the environment variable TF\$1USE\$1LEGACY\$1KERAS=1. If your workflows require usage of Keras 3.0, please remove this environment variable from your TensorFlow virtual environment /opt/tensorflow using the following script:

```
source /opt/tensorflow/bin/activate
unset TF_USE_LEGACY_KERAS
```

#### Release Date: 2025-02-17
<a name="2025-02-17-gpu-tensorflow-2.16-amazon-linux-2"></a>

**AMI Names:**
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20250215
+ Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20250215

##### Updated
<a name="w2aac25c13b9c11c13b7"></a>
+ Updated NVIDIA Container Toolkit from version 1.17.3 to version 1.17.4
  + Please see the release notes page here for more information: [https://github.com/NVIDIA/nvidia-container-toolkit/releases/tag/v1.17.4](https://github.com/NVIDIA/nvidia-container-toolkit/releases/tag/v1.17.4)
  + In Container Toolkit version 1.17.4, the mounting of CUDA compat libraries is now disabled. In order to ensure compatibility with multiple CUDA versions on container workflows, please ensure you update your LD\$1LIBRARY\$1PATH to include your CUDA compatibility libraries as shown in under the "If you use a CUDA compatibility layer" tutorial here - [https://docs.aws.amazon.com/sagemaker/latest/dg/inference-gpu-drivers.html\$1collapsible-cuda-compat](https://docs.aws.amazon.com/sagemaker/latest/dg/inference-gpu-drivers.html#collapsible-cuda-compat)

##### Removed
<a name="w2aac25c13b9c11c13b9"></a>
+ Removed user space libraries cuobj and nvdisasm provided by [NVIDIA CUDA toolkit](https://docs.nvidia.com/cuda/) to address CVE's present in the [NVIDIA CUDA Toolkit Security Bulletin for February 18, 2025](https://nvidia.custhelp.com/app/answers/detail/a_id/5594)

#### Release Date: 2025-01-20
<a name="2025-01-20-gpu-tensorflow-2.16-amazon-linux-2"></a>

**AMI Names:**
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20250120
+ Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20250118

##### Updated
<a name="w2aac25c13b9c11c15b7"></a>
+ Upgraded Nvidia driver from version 550.127.05 to 550.144.03 to address CVE's present in the [NVIDIA GPU Display Driver Security Bulletin for January 2025](https://nvidia.custhelp.com/app/answers/detail/a_id/5614)

#### Release Date: 2024-10-23
<a name="2024-10-23-gpu-tensorflow-2.16-amazon-linux-2"></a>

**AMI Names:**
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20241022
+ Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20241023

##### Updated
<a name="w2aac25c13b9c11c17b7"></a>
+ Upgraded Nvidia driver from version 550.90.07 to 550.127.05 to address CVE's present in the [NVIDIA GPU Display Security Bulletin for October 2024](https://nvidia.custhelp.com/app/answers/detail/a_id/5586)

#### Release Date: 2024-09-28
<a name="2024-09-28-gpu-tensorflow-2.16-amazon-linux-2"></a>

**AMI Names:**
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240928
+ Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240928

##### Updated
<a name="w2aac25c13b9c11c19b7"></a>
+ Upgraded Nvidia Container Toolkit from version 1.16.1 to 1.16.2, addressing the security vulnerability [CVE-2024-0133](https://nvd.nist.gov/vuln/detail/CVE-2024-0133).

#### Release Date: 2024-09-21
<a name="2024-09-21-gpu-tensorflow-2.16-amazon-linux-2"></a>

**AMI Names:**
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240921
+ Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240921

##### Updated
<a name="w2aac25c13b9c11c21b7"></a>
+ Upgraded Nvidia driver and Fabric Manager from version 535.183.01 to 550.90.07
+ Upgraded EFA Version from 1.32.0 to 1.34.0
+ Updated PyTorch version from version 2.3.0 to 2.3.1

##### Added
<a name="w2aac25c13b9c11c21b9"></a>
+ Added support for P5e EC2 Instance on OSS Nvidia Driver Images.

#### Release Date: 2024-08-19
<a name="2024-08-19-gpu-tensorflow-2.16-amazon-linux-2"></a>

**AMI Names:**
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240817

##### Added
<a name="w2aac25c13b9c11c23b7"></a>
+ Added support for [G6e EC2 instance](https://aws.amazon.com/ec2/instance-types/g6e/).

#### Version 2.16.2 - Release Date: 2024-07-26
<a name="2024-07-26-gpu-tensorflow-2.16-amazon-linux-2"></a>

**AMI Names:**
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240725

##### Updated
<a name="w2aac25c13b9c11c25b7"></a>
+ Updated TensorFlow patch version from version 2.16.1 to 2.16.2
+ Resolved incorrect TensorFlow minor version in DLAMI released on 2024-07-17
  + Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240717 release inadvertently contained TensorFlow minor version 2.17 rather than 2.16. Please ensure workflows reliant on TensorFlow 2.16 are upgrading to the latest DLAMI.

#### Version 2.16.1 - Release Date: 2024-06-10
<a name="2024-06-10-gpu-tensorflow-2.16-amazon-linux-2"></a>

**AMI Names:**
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240607
+ Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240610

##### Updated
<a name="w2aac25c13b9c11c27b7"></a>
+ Updated Nvidia driver version to 535.183.01 from 535.161.08

#### Release Date: 2024-05-10
<a name="2024-05-10-gpu-tensorflow-2.16-amazon-linux-2"></a>

Please refer to [Important changes to DLAMI](important-changes.md)

**AMI Names:**
+ Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240510
+ Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) 20240510

##### Added
<a name="w2aac25c13b9c11c29b9"></a>
+ Initial release of:
  + Deep Learning Proprietary Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) series.
  + Deep Learning OSS Nvidia Driver AMI GPU TensorFlow 2.16 (Amazon Linux 2) series.
  + Software Includes the Following:
    + "nvidia-driver=535.161.08"
    + "fabric-manager=535.161.08"
    + "cuda=12.3"
    + "cudnn=8.9.7"
    + "efa=1.32.0"
    + "nccl=2.21.5"
    + "aws-nccl-ofi-plugin=v1.9.1-aws"
+ Added tensorflow virtual environment (activation command source /opt/tensorflow/bin/activate). This environment includes the following:
  + "tensorflow=2.16.1"
  + **NOTE**
    + Starting with TF2.16, the tf.estimator API is removed.
      + To continue using tf.estimator, you will need to use TF 2.15 or an earlier version. Please see the [TensorFlow 2.16.1 release notes](https://github.com/tensorflow/tensorflow/releases/tag/v2.16.1) for more information
    + To ensure compatibility with customer workflows, we have pinned Keras versions to 2.0 using the environment variable TF\$1USE\$1LEGACY\$1KERAS=1. If your workflows require usage of Keras 3.0, please remove this environment variable from your TensorFlow virtual environment /opt/tensorflow using the following script:

```
source /opt/tensorflow/bin/activate
unset TF_USE_LEGACY_KERAS
```