AWS Deep Learning Containers for SGLang with EFA Support on SageMaker

AWS Deep Learning Containers (DLCs) now support SGLang images that are optimized for large language model serving on Amazon SageMaker. The SGLang DLC provides a production-ready environment for deploying and serving LLMs with advanced features like RadixAttention for efficient KV cache reuse and optimized batch scheduling. With SGLang's advanced features and optimizations pre-configured, this specialized container offers an ideal starting point for high-performance, scalable and efficient LLM serving on SageMaker.

All software components in this container are scanned for security vulnerabilities and updated or patched in accordance with AWS Security best practices. A list of all available AWS DLCs can be found in our github repo. Get started quickly with AWS DLCs using the getting-started section in our developer guide. To ensure you're using the latest DLC releases, we invite you to subscribe to our DLC notification mechanism. For guide on how to use SGLang DLC on SageMaker, checkout tutorial. For guide on how to use SGLang, checkout SGLang documentation.

Changelog

To learn about latest changes in SGLang DLC, checkout the changelog.

A list of available containers can be found on GitHub.

SGLang v0.5.5 and above utilizes CUDA 12.9 which is only compatible with Nvidia Driver 535 and above (550 preferred). To deploy the container on SageMaker platform, please specify al2-ami-sagemaker-inference-gpu-3-1 as the ProductionVariant.

Security Advisory

All software components in these images are scanned for security vulnerabilities and updated or patched in accordance with AWS Security best practices.

AWS recommends that customers monitor critical security updates in the AWS Security Bulletin.

Python 3.12 Support

Python 3.12 is supported.

Instance Type Support

The containers support x86_64 instance types.

AWS Regions support

The containers are available in the following regions:

Build and Test

Known Issues

No known issues so far