# Edge Devices
Amazon SageMaker Neo provides compilation support for popular machine learning frameworks. You can deploy your Neo-compiled edge devices such as the Raspberry Pi 3, Texas Instruments' Sitara, Jetson TX1, and more. For a full list of supported frameworks and edge devices, see [Supported Frameworks, Devices, Systems, and Architectures](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-supported-devices-edge.html).
You must configure your edge device so that it can use AWS services. One way to do this is to install DLR and Boto3 to your device. To do this, you must set up the authentication credentials. See [Boto3 AWS Configuration](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration) for more information. Once your model is compiled and your edge device is configured, you can download the model from Amazon S3 to your edge device. From there, you can use the [Deep Learning Runtime (DLR)](https://neo-ai-dlr.readthedocs.io/en/latest/index.html) to read the compiled model and make inferences.
For first-time users, we recommend you check out the [Getting Started](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-getting-started-edge.html) guide. This guide walks you through how to set up your credentials, compile a model, deploy your model to a Raspberry Pi 3, and make inferences on images.
**Topics**
+ [Supported Frameworks, Devices, Systems, and Architectures](neo-supported-devices-edge.md)
+ [Deploy Models](neo-deployment-edge.md)
+ [Set up Neo on Edge Devices](neo-getting-started-edge.md)
# Supported Frameworks, Devices, Systems, and Architectures
Amazon SageMaker Neo supports common machine learning frameworks, edge devices, operating systems, and chip architectures. Find out if Neo supports your framework, edge device, OS, and chip architecture by selecting one of the topics below.
You can find a list of models that have been tested by the Amazon SageMaker Neo Team in the [Tested Models](neo-supported-edge-tested-models.md) section.
**Note**
Ambarella devices require additional files to be included within the compressed TAR file before it is sent for compilation. For more information, see [Troubleshoot Ambarella Errors](neo-troubleshooting-target-devices-ambarella.md).
TIM-VX (libtim-vx.so) is required for i.MX 8M Plus. For information on how to build TIM-VX, see the [TIM-VX GitHub repository](https://github.com/VeriSilicon/TIM-VX).
**Topics**
+ [Supported Frameworks](neo-supported-devices-edge-frameworks.md)
+ [Supported Devices, Chip Architectures, and Systems](neo-supported-devices-edge-devices.md)
+ [Tested Models](neo-supported-edge-tested-models.md)
# Supported Frameworks
Amazon SageMaker Neo supports the following frameworks.
| Framework | Framework Version | Model Version | Models | Model Formats (packaged in \$1.tar.gz) | Toolkits |
| --- | --- | --- | --- | --- | --- |
| MXNet | 1.8 | Supports 1.8 or earlier | Image Classification, Object Detection, Semantic Segmentation, Pose Estimation, Activity Recognition | One symbol file (.json) and one parameter file (.params) | GluonCV v0.8.0 |
| ONNX | 1.7 | Supports 1.7 or earlier | Image Classification, SVM | One model file (.onnx) | |
| Keras | 2.2 | Supports 2.2 or earlier | Image Classification | One model definition file (.h5) | |
| PyTorch | 1.7, 1.8 | Supports 1.7, 1.8 or earlier | Image Classification, Object Detection | One model definition file (.pth) | |
| TensorFlow | 1.15, 2.4, 2.5 (only for ml.inf1.\$1 instances) | Supports 1.15, 2.4, 2.5 (only for ml.inf1.\$1 instances) or earlier | Image Classification, Object Detection | \$1For saved models, one .pb or one .pbtxt file and a variables directory that contains variables \$1For frozen models, only one .pb or .pbtxt file | |
| TensorFlow-Lite | 1.15 | Supports 1.15 or earlier | Image Classification, Object Detection | One model definition flatbuffer file (.tflite) | |
| XGBoost | 1.3 | Supports 1.3 or earlier | Decision Trees | One XGBoost model file (.model) where the number of nodes in a tree is less than 2^31 | |
| DARKNET | | | Image Classification, Object Detection (Yolo model is not supported) | One config (.cfg) file and one weights (.weights) file | |
# Supported Devices, Chip Architectures, and Systems
Amazon SageMaker Neo supports the following devices, chip architectures, and operating systems.
## Devices
You can select a device using the dropdown list in the [Amazon SageMaker AI console](https://console.aws.amazon.com/sagemaker) or by specifying the `TargetDevice` in the output configuration of the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateCompilationJob.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateCompilationJob.html) API.
You can choose from one of the following edge devices:
| Device List | System on a Chip (SoC) | Operating System | Architecture | Accelerator | Compiler Options Example |
| --- | --- | --- | --- | --- | --- |
| aisage | None | Linux | ARM64 | Mali | None |
| amba\$1cv2 | CV2 | Arch Linux | ARM64 | cvflow | None |
| amba\$1cv22 | CV22 | Arch Linux | ARM64 | cvflow | None |
| amba\$1cv25 | CV25 | Arch Linux | ARM64 | cvflow | None |
| coreml | None | iOS, macOS | None | None | \$1"class\$1labels": "imagenet\$1labels\$11000.txt"\$1 |
| imx8qm | NXP imx8 | Linux | ARM64 | None | None |
| imx8mplus | i.MX 8M Plus | Linux | ARM64 | NPU | None |
| jacinto\$1tda4vm | TDA4VM | Linux | ARM | TDA4VM | None |
| jetson\$1nano | None | Linux | ARM64 | NVIDIA | \$1'gpu-code': 'sm\$153', 'trt-ver': '5.0.6', 'cuda-ver': '10.0'\$1For `TensorFlow2`, `{'JETPACK_VERSION': '4.6', 'gpu_code': 'sm_72'}` |
| jetson\$1tx1 | None | Linux | ARM64 | NVIDIA | \$1'gpu-code': 'sm\$153', 'trt-ver': '6.0.1', 'cuda-ver': '10.0'\$1 |
| jetson\$1tx2 | None | Linux | ARM64 | NVIDIA | \$1'gpu-code': 'sm\$162', 'trt-ver': '6.0.1', 'cuda-ver': '10.0'\$1 |
| jetson\$1xavier | None | Linux | ARM64 | NVIDIA | \$1'gpu-code': 'sm\$172', 'trt-ver': '5.1.6', 'cuda-ver': '10.0'\$1 |
| qcs605 | None | Android | ARM64 | Mali | \$1'ANDROID\$1PLATFORM': 27\$1 |
| qcs603 | None | Android | ARM64 | Mali | \$1'ANDROID\$1PLATFORM': 27\$1 |
| rasp3b | ARM A56 | Linux | ARM\$1EABIHF | None | \$1'mattr': ['\$1neon']\$1 |
| rasp4b | ARM A72 | None | None | None | None |
| rk3288 | None | Linux | ARM\$1EABIHF | Mali | None |
| rk3399 | None | Linux | ARM64 | Mali | None |
| sbe\$1c | None | Linux | x86\$164 | None | \$1'mcpu': 'core-avx2'\$1 |
| sitara\$1am57x | AM57X | Linux | ARM64 | EVE and/or C66x DSP | None |
| x86\$1win32 | None | Windows 10 | X86\$132 | None | None |
| x86\$1win64 | None | Windows 10 | X86\$132 | None | None |
For more information about JSON key-value compiler options for each target device, see the `CompilerOptions` field in the [`OutputConfig` API](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_OutputConfig.html) data type.
## Systems and Chip Architectures
The following look-up tables provide information regarding available operating systems and architectures for Neo model compilation jobs.
------
#### [ Linux ]
| Accelerator | X86\$164 | X86 | ARM64 | ARM\$1EABIHF | ARM\$1EABI |
| --- | --- | --- | --- | --- | --- |
| No accelerator (CPU) | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes |
| Nvidia GPU | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No |
| Intel\$1Graphics | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No |
| ARM Mali | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes |
------
#### [ Android ]
| Accelerator | X86\$164 | X86 | ARM64 | ARM\$1EABIHF | ARM\$1EABI |
| --- | --- | --- | --- | --- | --- |
| No accelerator (CPU) | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes |
| Nvidia GPU | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No |
| Intel\$1Graphics | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No |
| ARM Mali | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes |
------
#### [ Windows ]
| Accelerator | X86\$164 | X86 | ARM64 | ARM\$1EABIHF | ARM\$1EABI |
| --- | --- | --- | --- | --- | --- |
| No accelerator (CPU) | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/success_icon.png) Yes | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No | ![\[alt text not found\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/negative_icon.png) No |
------
# Tested Models
The following collapsible sections provide information about machine learning models that were tested by the Amazon SageMaker Neo team. Expand the collapsible section based on your framework to check if a model was tested.
**Note**
This is not a comprehensive list of models that can be compiled with Neo.
See [Supported Frameworks](neo-supported-devices-edge-frameworks.md) and [SageMaker AI Neo Supported Operators](https://aws.amazon.com/releasenotes/sagemaker-neo-supported-frameworks-and-operators/) to find out if you can compile your model with SageMaker Neo.
## DarkNet
| Models | ARM V8 | ARM Mali | Ambarella CV22 | Nvidia | Panorama | TI TDA4VM | Qualcomm QCS603 | X86\$1Linux | X86\$1Windows |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Alexnet | | | | | | | | | |
| Resnet50 | X | X | | X | X | X | | X | X |
| YOLOv2 | | | | X | X | X | | X | X |
| YOLOv2\$1tiny | X | X | | X | X | X | | X | X |
| YOLOv3\$1416 | | | | X | X | X | | X | X |
| YOLOv3\$1tiny | X | X | | X | X | X | | X | X |
## MXNet
| Models | ARM V8 | ARM Mali | Ambarella CV22 | Nvidia | Panorama | TI TDA4VM | Qualcomm QCS603 | X86\$1Linux | X86\$1Windows |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Alexnet | | | X | | | | | | |
| Densenet121 | | | X | | | | | | |
| DenseNet201 | X | X | X | X | X | X | | X | X |
| GoogLeNet | X | X | | X | X | X | | X | X |
| InceptionV3 | | | | X | X | X | | X | X |
| MobileNet0.75 | X | X | | X | X | X | | | X |
| MobileNet1.0 | X | X | X | X | X | X | | | X |
| MobileNetV2\$10.5 | X | X | | X | X | X | | | X |
| MobileNetV2\$11.0 | X | X | X | X | X | X | X | X | X |
| MobileNetV3\$1Large | X | X | X | X | X | X | X | X | X |
| MobileNetV3\$1Small | X | X | X | X | X | X | X | X | X |
| ResNeSt50 | | | | X | X | | | X | X |
| ResNet18\$1v1 | X | X | X | X | X | X | | | X |
| ResNet18\$1v2 | X | X | | X | X | X | | | X |
| ResNet50\$1v1 | X | X | X | X | X | X | | X | X |
| ResNet50\$1v2 | X | X | X | X | X | X | | X | X |
| ResNext101\$132x4d | | | | | | | | | |
| ResNext50\$132x4d | X | | X | X | X | | | X | X |
| SENet\$1154 | | | | X | X | X | | X | X |
| SE\$1ResNext50\$132x4d | X | X | | X | X | X | | X | X |
| SqueezeNet1.0 | X | X | X | X | X | X | | | X |
| SqueezeNet1.1 | X | X | X | X | X | X | | X | X |
| VGG11 | X | X | X | X | X | | | X | X |
| Xception | X | X | X | X | X | X | | X | X |
| darknet53 | X | X | | X | X | X | | X | X |
| resnet18\$1v1b\$10.89 | X | X | | X | X | X | | | X |
| resnet50\$1v1d\$10.11 | X | X | | X | X | X | | | X |
| resnet50\$1v1d\$10.86 | X | X | X | X | X | X | | X | X |
| ssd\$1512\$1mobilenet1.0\$1coco | X | | X | X | X | X | | X | X |
| ssd\$1512\$1mobilenet1.0\$1voc | X | | X | X | X | X | | X | X |
| ssd\$1resnet50\$1v1 | X | | X | X | X | | | X | X |
| yolo3\$1darknet53\$1coco | X | | | X | X | | | X | X |
| yolo3\$1mobilenet1.0\$1coco | X | X | | X | X | X | | X | X |
| deeplab\$1resnet50 | | | X | | | | | | |
## Keras
| Models | ARM V8 | ARM Mali | Ambarella CV22 | Nvidia | Panorama | TI TDA4VM | Qualcomm QCS603 | X86\$1Linux | X86\$1Windows |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| densenet121 | X | X | X | X | X | X | | X | X |
| densenet201 | X | X | X | X | X | X | | | X |
| inception\$1v3 | X | X | | X | X | X | | X | X |
| mobilenet\$1v1 | X | X | X | X | X | X | | X | X |
| mobilenet\$1v2 | X | X | X | X | X | X | | X | X |
| resnet152\$1v1 | | | | X | X | | | | X |
| resnet152\$1v2 | | | | X | X | | | | X |
| resnet50\$1v1 | X | X | X | X | X | | | X | X |
| resnet50\$1v2 | X | X | X | X | X | X | | X | X |
| vgg16 | | | X | X | X | | | X | X |
## ONNX
| Models | ARM V8 | ARM Mali | Ambarella CV22 | Nvidia | Panorama | TI TDA4VM | Qualcomm QCS603 | X86\$1Linux | X86\$1Windows |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| alexnet | | | X | | | | | | |
| mobilenetv2-1.0 | X | X | X | X | X | X | | X | X |
| resnet18v1 | X | | | X | X | | | | X |
| resnet18v2 | X | | | X | X | | | | X |
| resnet50v1 | X | | X | X | X | | | X | X |
| resnet50v2 | X | | X | X | X | | | X | X |
| resnet152v1 | | | | X | X | X | | | X |
| resnet152v2 | | | | X | X | X | | | X |
| squeezenet1.1 | X | | X | X | X | X | | X | X |
| vgg19 | | | X | | | | | | X |
## PyTorch (FP32)
| Models | ARM V8 | ARM Mali | Ambarella CV22 | Ambarella CV25 | Nvidia | Panorama | TI TDA4VM | Qualcomm QCS603 | X86\$1Linux | X86\$1Windows |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| densenet121 | X | X | X | X | X | X | X | | X | X |
| inception\$1v3 | | X | | | X | X | X | | X | X |
| resnet152 | | | | | X | X | X | | | X |
| resnet18 | X | X | | | X | X | X | | | X |
| resnet50 | X | X | X | X | X | X | | | X | X |
| squeezenet1.0 | X | X | | | X | X | X | | | X |
| squeezenet1.1 | X | X | X | X | X | X | X | | X | X |
| yolov4 | | | | | X | X | | | | |
| yolov5 | | | | X | X | X | | | | |
| fasterrcnn\$1resnet50\$1fpn | | | | | X | X | | | | |
| maskrcnn\$1resnet50\$1fpn | | | | | X | X | | | | |
## TensorFlow
------
#### [ TensorFlow ]
| Models | ARM V8 | ARM Mali | Ambarella CV22 | Ambarella CV25 | Nvidia | Panorama | TI TDA4VM | Qualcomm QCS603 | X86\$1Linux | X86\$1Windows |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| densenet201 | X | X | X | X | X | X | X | | X | X |
| inception\$1v3 | X | X | X | | X | X | X | | X | X |
| mobilenet100\$1v1 | X | X | X | | X | X | X | | | X |
| mobilenet100\$1v2.0 | X | X | X | | X | X | X | | X | X |
| mobilenet130\$1v2 | X | X | | | X | X | X | | | X |
| mobilenet140\$1v2 | X | X | X | | X | X | X | | X | X |
| resnet50\$1v1.5 | X | X | | | X | X | X | | X | X |
| resnet50\$1v2 | X | X | X | X | X | X | X | | X | X |
| squeezenet | X | X | X | X | X | X | X | | X | X |
| mask\$1rcnn\$1inception\$1resnet\$1v2 | | | | | X | | | | | |
| ssd\$1mobilenet\$1v2 | | | | | X | X | | | | |
| faster\$1rcnn\$1resnet50\$1lowproposals | | | | | X | | | | | |
| rfcn\$1resnet101 | | | | | X | | | | | |
------
#### [ TensorFlow.Keras ]
| Models | ARM V8 | ARM Mali | Ambarella CV22 | Nvidia | Panorama | TI TDA4VM | Qualcomm QCS603 | X86\$1Linux | X86\$1Windows |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DenseNet121 | X | X | | X | X | X | | X | X |
| DenseNet201 | X | X | | X | X | X | | | X |
| InceptionV3 | X | X | | X | X | X | | X | X |
| MobileNet | X | X | | X | X | X | | X | X |
| MobileNetv2 | X | X | | X | X | X | | X | X |
| NASNetLarge | | | | X | X | | | X | X |
| NASNetMobile | X | X | | X | X | X | | X | X |
| ResNet101 | | | | X | X | X | | | X |
| ResNet101V2 | | | | X | X | X | | | X |
| ResNet152 | | | | X | X | | | | X |
| ResNet152v2 | | | | X | X | | | | X |
| ResNet50 | X | X | | X | X | | | X | X |
| ResNet50V2 | X | X | | X | X | X | | X | X |
| VGG16 | | | | X | X | | | X | X |
| Xception | X | X | | X | X | X | | X | X |
------
## TensorFlow-Lite
------
#### [ TensorFlow-Lite (FP32) ]
| Models | ARM V8 | ARM Mali | Ambarella CV22 | Nvidia | Panorama | TI TDA4VM | Qualcomm QCS603 | X86\$1Linux | X86\$1Windows | i.MX 8M Plus |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| densenet\$12018\$104\$127 | X | | | X | X | X | | | X | |
| inception\$1resnet\$1v2\$12018\$104\$127 | | | | X | X | X | | | X | |
| inception\$1v3\$12018\$104\$127 | | | | X | X | X | | | X | X |
| inception\$1v4\$12018\$104\$127 | | | | X | X | X | | | X | X |
| mnasnet\$10.5\$1224\$109\$107\$12018 | X | | | X | X | X | | | X | |
| mnasnet\$11.0\$1224\$109\$107\$12018 | X | | | X | X | X | | | X | |
| mnasnet\$11.3\$1224\$109\$107\$12018 | X | | | X | X | X | | | X | |
| mobilenet\$1v1\$10.25\$1128 | X | | | X | X | X | | | X | X |
| mobilenet\$1v1\$10.25\$1224 | X | | | X | X | X | | | X | X |
| mobilenet\$1v1\$10.5\$1128 | X | | | X | X | X | | | X | X |
| mobilenet\$1v1\$10.5\$1224 | X | | | X | X | X | | | X | X |
| mobilenet\$1v1\$10.75\$1128 | X | | | X | X | X | | | X | X |
| mobilenet\$1v1\$10.75\$1224 | X | | | X | X | X | | | X | X |
| mobilenet\$1v1\$11.0\$1128 | X | | | X | X | X | | | X | X |
| mobilenet\$1v1\$11.0\$1192 | X | | | X | X | X | | | X | X |
| mobilenet\$1v2\$11.0\$1224 | X | | | X | X | X | | | X | X |
| resnet\$1v2\$1101 | | | | X | X | X | | | X | |
| squeezenet\$12018\$104\$127 | X | | | X | X | X | | | X | |
------
#### [ TensorFlow-Lite (INT8) ]
| Models | ARM V8 | ARM Mali | Ambarella CV22 | Nvidia | Panorama | TI TDA4VM | Qualcomm QCS603 | X86\$1Linux | X86\$1Windows | i.MX 8M Plus |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| inception\$1v1 | | | | | | | X | | | X |
| inception\$1v2 | | | | | | | X | | | X |
| inception\$1v3 | X | | | | | X | X | | X | X |
| inception\$1v4\$1299 | X | | | | | X | X | | X | X |
| mobilenet\$1v1\$10.25\$1128 | X | | | | | X | | | X | X |
| mobilenet\$1v1\$10.25\$1224 | X | | | | | X | | | X | X |
| mobilenet\$1v1\$10.5\$1128 | X | | | | | X | | | X | X |
| mobilenet\$1v1\$10.5\$1224 | X | | | | | X | | | X | X |
| mobilenet\$1v1\$10.75\$1128 | X | | | | | X | | | X | X |
| mobilenet\$1v1\$10.75\$1224 | X | | | | | X | X | | X | X |
| mobilenet\$1v1\$11.0\$1128 | X | | | | | X | | | X | X |
| mobilenet\$1v1\$11.0\$1224 | X | | | | | X | X | | X | X |
| mobilenet\$1v2\$11.0\$1224 | X | | | | | X | X | | X | X |
| deeplab-v3\$1513 | | | | | | | X | | | |
------
# Deploy Models
You can deploy the compute module to resource-constrained edge devices by: downloading the compiled model from Amazon S3 to your device and using [DLR](https://github.com/neo-ai/neo-ai-dlr), or you can use [AWS IoT Greengrass](https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html).
Before moving on, make sure your edge device must be supported by SageMaker Neo. See, [Supported Frameworks, Devices, Systems, and Architectures](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-supported-devices-edge.html) to find out what edge devices are supported. Make sure that you specified your target edge device when you submitted the compilation job, see [Use Neo to Compile a Model](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-job-compilation.html).
## Deploy a Compiled Model (DLR)
[DLR](https://github.com/neo-ai/neo-ai-dlr) is a compact, common runtime for deep learning models and decision tree models. DLR uses the [TVM](https://github.com/neo-ai/tvm) runtime, [Treelite](https://treelite.readthedocs.io/en/latest/install.html) runtime, NVIDIA TensorRT™, and can include other hardware-specific runtimes. DLR provides unified Python/C\$1\$1 APIs for loading and running compiled models on various devices.
You can install latest release of DLR package using the following pip command:
```
pip install dlr
```
For installation of DLR on GPU targets or non-x86 edge devices, please refer to [Releases](https://github.com/neo-ai/neo-ai-dlr/releases) for prebuilt binaries, or [Installing DLR](https://neo-ai-dlr.readthedocs.io/en/latest/install.html) for building DLR from source. For example, to install DLR for Raspberry Pi 3, you can use:
```
pip install https://neo-ai-dlr-release.s3-us-west-2.amazonaws.com/v1.3.0/pi-armv7l-raspbian4.14.71-glibc2_24-libstdcpp3_4/dlr-1.3.0-py3-none-any.whl
```
## Deploy a Model (AWS IoT Greengrass)
[AWS IoT Greengrass](https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html) extends cloud capabilities to local devices. It enables devices to collect and analyze data closer to the source of information, react autonomously to local events, and communicate securely with each other on local networks. With AWS IoT Greengrass, you can perform machine learning inference at the edge on locally generated data using cloud-trained models. Currently, you can deploy models on to all AWS IoT Greengrass devices based on ARM Cortex-A, Intel Atom, and Nvidia Jetson series processors. For more information on deploying a Lambda inference application to perform machine learning inferences with AWS IoT Greengrass, see [ How to configure optimized machine learning inference using the AWS Management Console](https://docs.aws.amazon.com/greengrass/latest/developerguide/ml-dlc-console.html).
# Set up Neo on Edge Devices
This guide to getting started with Amazon SageMaker Neo shows you how to compile a model, set up your device, and make inferences on your device. Most of the code examples use Boto3. We provide commands using AWS CLI where applicable, as well as instructions on how to satisfy prerequisites for Neo.
**Note**
You can run the following code snippets on your local machine, within a SageMaker notebook, within Amazon SageMaker Studio, or (depending on your edge device) on your edge device. The setup is similar; however, there are two main exceptions if you run this guide within a SageMaker notebook instance or SageMaker Studio session:
You do not need to install Boto3.
You do not need to add the `‘AmazonSageMakerFullAccess’` IAM policy
This guide assumes you are running the following instructions on your edge device.
# Prerequisites
SageMaker Neo is a capability that allows you to train machine learning models once and run them anywhere in the cloud and at the edge. Before you can compile and optimize your models with Neo, there are a few prerequisites you need to set up. You must install the necessary Python libraries, configure your AWS credentials, create an IAM role with the required permissions, and set up an S3 bucket for storing model artifacts. You must also have a trained machine learning model ready. The following steps guide you through the setup:
1. **Install Boto3**
If you are running these commands on your edge device, you must install the AWS SDK for Python (Boto3). Within a Python environment (preferably a virtual environment), run the following locally on your edge device's terminal or within a Jupyter notebook instance:
------
#### [ Terminal ]
```
pip install boto3
```
------
#### [ Jupyter Notebook ]
```
!pip install boto3
```
------
1. **Set Up AWS Credentials**
You need to set up Amazon Web Services credentials on your device in order to run SDK for Python (Boto3). By default, the AWS credentials should be stored in the file `~/.aws/credentials` on your edge device. Within the credentials file, you should see two environment variables: `aws_access_key_id` and `aws_secret_access_key`.
In your terminal, run:
```
$ more ~/.aws/credentials
[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
```
The [AWS General Reference Guide](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys) has instructions on how to get the necessary `aws_access_key_id` and `aws_secret_access_key`. For more information on how to set up credentials on your device, see the [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration) documentation.
1. **Set up an IAM Role and attach policies.**
Neo needs access to your S3 bucket URI. Create an IAM role that can run SageMaker AI and has permission to access the S3 URI. You can create an IAM role either by using SDK for Python (Boto3), the console, or the AWS CLI. The following example illustrates how to create an IAM role using SDK for Python (Boto3):
```
import boto3
AWS_REGION = 'aws-region'
# Create an IAM client to interact with IAM
iam_client = boto3.client('iam', region_name=AWS_REGION)
role_name = 'role-name'
```
For more information on how to create an IAM role with the console, AWS CLI, or through the AWS API, see [Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_api).
Create a dictionary describing the IAM policy you are attaching. This policy is used to create a new IAM role.
```
policy = {
'Statement': [
{
'Action': 'sts:AssumeRole',
'Effect': 'Allow',
'Principal': {'Service': 'sagemaker.amazonaws.com'},
}],
'Version': '2012-10-17 '
}
```
Create a new IAM role using the policy you defined above:
```
import json
new_role = iam_client.create_role(
AssumeRolePolicyDocument=json.dumps(policy),
Path='/',
RoleName=role_name
)
```
You need to know what your Amazon Resource Name (ARN) is when you create a compilation job in a later step, so store it in a variable as well.
```
role_arn = new_role['Role']['Arn']
```
Now that you have created a new role, attach the permissions it needs to interact with Amazon SageMaker AI and Amazon S3:
```
iam_client.attach_role_policy(
RoleName=role_name,
PolicyArn='arn:aws:iam::aws:policy/AmazonSageMakerFullAccess'
)
iam_client.attach_role_policy(
RoleName=role_name,
PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess'
);
```
1. **Create an Amazon S3 bucket to store your model artifacts**
SageMaker Neo will access your model artifacts from Amazon S3
------
#### [ Boto3 ]
```
# Create an S3 client
s3_client = boto3.client('s3', region_name=AWS_REGION)
# Name buckets
bucket='name-of-your-bucket'
# Check if bucket exists
if boto3.resource('s3').Bucket(bucket) not in boto3.resource('s3').buckets.all():
s3_client.create_bucket(
Bucket=bucket,
CreateBucketConfiguration={
'LocationConstraint': AWS_REGION
}
)
else:
print(f'Bucket {bucket} already exists. No action needed.')
```
------
#### [ CLI ]
```
aws s3 mb s3://'name-of-your-bucket' --region specify-your-region
# Check your bucket exists
aws s3 ls s3://'name-of-your-bucket'/
```
------
1. **Train a machine learning model**
See [Train a Model with Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-training.html) for more information on how to train a machine learning model using Amazon SageMaker AI. You can optionally upload your locally trained model directly into an Amazon S3 URI bucket.
**Note**
Make sure the model is correctly formatted depending on the framework you used. See [What input data shapes does SageMaker Neo expect?](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-job-compilation.html#neo-job-compilation-expected-inputs)
If you do not have a model yet, use the `curl` command to get a local copy of the `coco_ssd_mobilenet` model from TensorFlow’s website. The model you just copied is an object detection model trained from the [COCO dataset](https://cocodataset.org/#home). Type the following into your Jupyter notebook:
```
model_zip_filename = './coco_ssd_mobilenet_v1_1.0.zip'
!curl http://storage.googleapis.com/download.tensorflow.org/models/tflite/coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip \
--output {model_zip_filename}
```
Note that this particular example was packaged in a .zip file. Unzip this file and repackage it as a compressed tarfile (`.tar.gz`) before using it in later steps. Type the following into your Jupyter notebook:
```
# Extract model from zip file
!unzip -u {model_zip_filename}
model_filename = 'detect.tflite'
model_name = model_filename.split('.')[0]
# Compress model into .tar.gz so SageMaker Neo can use it
model_tar = model_name + '.tar.gz'
!tar -czf {model_tar} {model_filename}
```
1. **Upload trained model to an S3 bucket**
Once you have trained your machine learning mode, store it in an S3 bucket.
------
#### [ Boto3 ]
```
# Upload model
s3_client.upload_file(Filename=model_filename, Bucket=bucket, Key=model_filename)
```
------
#### [ CLI ]
Replace `your-model-filename` and `amzn-s3-demo-bucket` with the name of your S3 bucket.
```
aws s3 cp your-model-filename s3://amzn-s3-demo-bucket
```
------
# Compile the Model
Once you have satisfied the [Prerequisites](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-getting-started-edge.html#neo-getting-started-edge-step0), you can compile your model with Amazon SageMaker AI Neo. You can compile your model using the AWS CLI, the console or the [Amazon Web Services SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html), see [Use Neo to Compile a Model](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-job-compilation.html). In this example, you will compile your model with Boto3.
To compile a model, SageMaker Neo requires the following information:
1. **The Amazon S3 bucket URI where you stored the trained model.**
If you followed the prerequisites, the name of your bucket is stored in a variable named `bucket`. The following code snippet shows how to list all of your buckets using the AWS CLI:
```
aws s3 ls
```
For example:
```
$ aws s3 ls
2020-11-02 17:08:50 bucket
```
1. **The Amazon S3 bucket URI where you want to save the compiled model.**
The code snippet below concatenates your Amazon S3 bucket URI with the name of an output directory called `output`:
```
s3_output_location = f's3://{bucket}/output'
```
1. **The machine learning framework you used to train your model.**
Define the framework you used to train your model.
```
framework = 'framework-name'
```
For example, if you wanted to compile a model that was trained using TensorFlow, you could either use `tflite` or `tensorflow`. Use `tflite` if you want to use a lighter version of TensorFlow that uses less storage memory.
```
framework = 'tflite'
```
For a complete list of Neo-supported frameworks, see [Supported Frameworks, Devices, Systems, and Architectures](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-supported-devices-edge.html).
1. **The shape of your model's input.**
Neo requires the name and shape of your input tensor. The name and shape are passed in as key-value pairs. `value` is a list of the integer dimensions of an input tensor and `key` is the exact name of an input tensor in the model.
```
data_shape = '{"name": [tensor-shape]}'
```
For example:
```
data_shape = '{"normalized_input_image_tensor":[1, 300, 300, 3]}'
```
**Note**
Make sure the model is correctly formatted depending on the framework you used. See [What input data shapes does SageMaker Neo expect?](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-job-compilation.html#neo-job-compilation-expected-inputs) The key in this dictionary must be changed to the new input tensor's name.
1. **Either the name of the target device to compile for or the general details of the hardware platform**
```
target_device = 'target-device-name'
```
For example, if you want to deploy to a Raspberry Pi 3, use:
```
target_device = 'rasp3b'
```
You can find the entire list of supported edge devices in [Supported Frameworks, Devices, Systems, and Architectures](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-supported-devices-edge.html).
Now that you have completed the previous steps, you can submit a compilation job to Neo.
```
# Create a SageMaker client so you can submit a compilation job
sagemaker_client = boto3.client('sagemaker', region_name=AWS_REGION)
# Give your compilation job a name
compilation_job_name = 'getting-started-demo'
print(f'Compilation job for {compilation_job_name} started')
response = sagemaker_client.create_compilation_job(
CompilationJobName=compilation_job_name,
RoleArn=role_arn,
InputConfig={
'S3Uri': s3_input_location,
'DataInputConfig': data_shape,
'Framework': framework.upper()
},
OutputConfig={
'S3OutputLocation': s3_output_location,
'TargetDevice': target_device
},
StoppingCondition={
'MaxRuntimeInSeconds': 900
}
)
# Optional - Poll every 30 sec to check completion status
import time
while True:
response = sagemaker_client.describe_compilation_job(CompilationJobName=compilation_job_name)
if response['CompilationJobStatus'] == 'COMPLETED':
break
elif response['CompilationJobStatus'] == 'FAILED':
raise RuntimeError('Compilation failed')
print('Compiling ...')
time.sleep(30)
print('Done!')
```
If you want additional information for debugging, include the following print statement:
```
print(response)
```
If the compilation job is successful, your compiled model isstored in the output Amazon S3 bucket you specified earlier (`s3_output_location`). Download your compiled model locally:
```
object_path = f'output/{model}-{target_device}.tar.gz'
neo_compiled_model = f'compiled-{model}.tar.gz'
s3_client.download_file(bucket, object_path, neo_compiled_model)
```
# Set Up Your Device
You will need to install packages on your edge device so that your device can make inferences. You will also need to either install [AWS IoT Greengrass](https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html) core or [Deep Learning Runtime (DLR)](https://github.com/neo-ai/neo-ai-dlr). In this example, you will install packages required to make inferences for the `coco_ssd_mobilenet` object detection algorithm and you will use DLR.
1. **Install additional packages**
In addition to Boto3, you must install certain libraries on your edge device. What libraries you install depends on your use case.
For example, for the `coco_ssd_mobilenet` object detection algorithm you downloaded earlier, you need to install [NumPy](https://numpy.org/) for data manipulation and statistics, [PIL](https://pillow.readthedocs.io/en/stable/) to load images, and [Matplotlib](https://matplotlib.org/) to generate plots. You also need a copy of TensorFlow if you want to gauge the impact of compiling with Neo versus a baseline.
```
!pip3 install numpy pillow tensorflow matplotlib
```
1. **Install inference engine on your device**
To run your Neo-compiled model, install the [Deep Learning Runtime (DLR)](https://github.com/neo-ai/neo-ai-dlr) on your device. DLR is a compact, common runtime for deep learning models and decision tree models. On x86\$164 CPU targets running Linux, you can install the latest release of the DLR package using the following `pip` command:
```
!pip install dlr
```
For installation of DLR on GPU targets or non-x86 edge devices, refer to [Releases](https://github.com/neo-ai/neo-ai-dlr/releases) for prebuilt binaries, or [Installing DLR](https://neo-ai-dlr.readthedocs.io/en/latest/install.html) for building DLR from source. For example, to install DLR for Raspberry Pi 3, you can use:
```
!pip install https://neo-ai-dlr-release.s3-us-west-2.amazonaws.com/v1.3.0/pi-armv7l-raspbian4.14.71-glibc2_24-libstdcpp3_4/dlr-1.3.0-py3-none-any.whl
```
# Make Inferences on Your Device
In this example, you will use Boto3 to download the output of your compilation job onto your edge device. You will then import DLR, download an example images from the dataset, resize this image to match the model’s original input, and then you will make a prediction.
1. **Download your compiled model from Amazon S3 to your device and extract it from the compressed tarfile.**
```
# Download compiled model locally to edge device
object_path = f'output/{model_name}-{target_device}.tar.gz'
neo_compiled_model = f'compiled-{model_name}.tar.gz'
s3_client.download_file(bucket_name, object_path, neo_compiled_model)
# Extract model from .tar.gz so DLR can use it
!mkdir ./dlr_model # make a directory to store your model (optional)
!tar -xzvf ./compiled-detect.tar.gz --directory ./dlr_model
```
1. **Import DLR and an initialized `DLRModel` object.**
```
import dlr
device = 'cpu'
model = dlr.DLRModel('./dlr_model', device)
```
1. **Download an image for inferencing and format it based on how your model was trained**.
For the `coco_ssd_mobilenet` example, you can download an image from the [COCO dataset](https://cocodataset.org/#home) and then reform the image to `300x300`:
```
from PIL import Image
# Download an image for model to make a prediction
input_image_filename = './input_image.jpg'
!curl https://farm9.staticflickr.com/8325/8077197378_79efb4805e_z.jpg --output {input_image_filename}
# Format image so model can make predictions
resized_image = image.resize((300, 300))
# Model is quantized, so convert the image to uint8
x = np.array(resized_image).astype('uint8')
```
1. **Use DLR to make inferences**.
Finally, you can use DLR to make a prediction on the image you just downloaded:
```
out = model.run(x)
```
For more examples using DLR to make inferences from a Neo-compiled model on an edge device, see the [neo-ai-dlr Github repository](https://github.com/neo-ai/neo-ai-dlr).