End of support notice: On May 31, 2026, AWS will end support for AWS Panorama. After May 31, 2026, you will no longer be able to access the AWS Panorama console or AWS Panorama resources. For more information, see [AWS Panorama end of support](https://docs.aws.amazon.com/panorama/latest/dev/panorama-end-of-support.html). # Getting started with AWS Panorama Getting started To get started with AWS Panorama, first learn about [the service's concepts](gettingstarted-concepts.md) and the terminology used in this guide. Then you can use the AWS Panorama console to [register your AWS Panorama Appliance](gettingstarted-setup.md) and [create an application](gettingstarted-deploy.md). In about an hour, you can configure the device, update its software, and deploy a sample application. To complete the tutorials in this section, you use the AWS Panorama Appliance and a camera that streams video over a local network. **Note** To purchase an AWS Panorama Appliance, visit [the AWS Panorama console](https://console.aws.amazon.com/panorama/home#get-device-quote). The [AWS Panorama sample application](gettingstarted-sample.md) demonstrates use of AWS Panorama features. It includes a model that has been trained with SageMaker AI and sample code that uses the AWS Panorama Application SDK to run inference and output video. The sample application include a CloudFormation template and scripts that show how to automate development and deployment workflows from the command line. The final two topics in this chapter detail [requirements for models and cameras](gettingstarted-compatibility.md), and the [hardware specifications of the AWS Panorama Appliance](gettingstarted-hardware.md). If you haven't obtained an appliance and cameras yet, or plan on developing your own computer vision models, see these topics first for more information. **Topics** + [ # AWS Panorama concepts ](gettingstarted-concepts.md) + [ # Setting up the AWS Panorama Appliance ](gettingstarted-setup.md) + [ # Deploying the AWS Panorama sample application ](gettingstarted-deploy.md) + [ # Developing AWS Panorama applications ](gettingstarted-sample.md) + [ # Supported computer vision models and cameras ](gettingstarted-compatibility.md) + [ # AWS Panorama Appliance specifications ](gettingstarted-hardware.md) + [ # Service quotas ](gettingstarted-quotas.md) # AWS Panorama concepts Concepts In AWS Panorama, you create computer vision applications and deploy them to the AWS Panorama Appliance or a compatible device to analyze video streams from network cameras. You write application code in Python and build application containers with Docker. You use the AWS Panorama Application CLI to import machine learning models locally or from Amazon Simple Storage Service (Amazon S3). Applications use the AWS Panorama Application SDK to receive video input from a camera and interact with a model. **Topics** + [ ## The AWS Panorama Appliance ](#gettingstarted-concepts-appliance) + [ ## Compatible devices ](#gettingstarted-concepts-devices) + [ ## Applications ](#gettingstarted-concepts-application) + [ ## Nodes ](#gettingstarted-concepts-node) + [ ## Models ](#gettingstarted-concepts-model) ## The AWS Panorama Appliance The AWS Panorama Appliance is the hardware that runs your applications. You use the AWS Panorama console to register an appliance, update its software, and deploy applications to it. The software on the AWS Panorama Appliance connects to camera streams, sends frames of video to your application, and displays video output on an attached display. The AWS Panorama Appliance is an *edge device* [powered by Nvidia Jetson AGX Xavier](gettingstarted-hardware.md). Instead of sending images to the AWS Cloud for processing, it runs applications locally on optimized hardware. This enables you to analyze video in real time and process the results locally. The appliance requires an internet connection to report its status, to upload logs, and to perform software updates and deployments. For more information, see [Managing the AWS Panorama Appliance](panorama-appliance.md). ## Compatible devices In addition to the AWS Panorama Appliance, AWS Panorama supports compatible devices from AWS Partners. Compatible devices support the same features as the AWS Panorama Appliance. You register and manage compatible devices with the AWS Panorama console and API, and build and deploy applications in the same way. **** + [Lenovo ThinkEdge® SE70](https://techtoday.lenovo.com/us/en/solutions/smb/thinkedge) – Powered by Nvidia Jetson Xavier NX The content and sample applications in this guide are developed with the AWS Panorama Appliance. For more information about specific hardware and software features for your device, refer to the manufacturer's documentation. ## Applications Applications run on the AWS Panorama Appliance to perform computer vision tasks on video streams. You can build computer vision applications by combining Python code and machine learning models, and deploy them to the AWS Panorama Appliance over the internet. Applications can send video to a display, or use the AWS SDK to send results to AWS services. To build and deploy applications, you use the AWS Panorama Application CLI. The AWS Panorama Application CLI is a command-line tool that generates default application folders and configuration files, builds containers with Docker, and uploads assets. You can run multiple applications on one device. For more information, see [Managing AWS Panorama applications](panorama-applications.md). ## Nodes An application comprises multiple components called *nodes*, which represent inputs, outputs, models, and code. A node can be configuration only (inputs and outputs), or include artifacts (models and code). An application's code node are bundled in *node packages* that you upload to an Amazon S3 access point, where the AWS Panorama Appliance can access them. An *application manifest* is a configuration file that defines connections between the nodes. For more information, see [Application nodes](applications-nodes.md). ## Models A computer vision model is a machine learning network that is trained to process images. Computer vision models can perform various tasks such as classification, detection, segmentation, and tracking. A computer vision model takes an image as input and outputs information about the image or objects in the image. AWS Panorama supports models built with PyTorch, Apache MXNet, and TensorFlow. You can build models with Amazon SageMaker AI or in your development environment. For more information, see [Computer vision models](applications-models.md). # Setting up the AWS Panorama Appliance Setting up To get started using your AWS Panorama Appliance or [compatible device](gettingstarted-concepts.md#gettingstarted-concepts-devices), register it in the AWS Panorama console and update its software. During the setup process, you create an appliance *resource* in AWS Panorama that represents the physical appliance, and copy files to the appliance with a USB drive. The appliance uses these certificates and configuration files to connect to the AWS Panorama service. Then you use the AWS Panorama console to update the appliance's software and register cameras. **Topics** + [ ## Prerequisites ](#gettingstarted-prerequisites) + [ ## Register and configure the AWS Panorama Appliance ](#gettingstarted-device) + [ ## Upgrade the appliance software ](#gettingstarted-upgrade) + [ ## Add a camera stream ](#gettingstarted-setup-camera) + [ ## Next steps ](#gettingstarted-setup-nextsteps) ## Prerequisites To follow this tutorial, you need an AWS Panorama Appliance or compatible device and the following hardware: **** + **Display** – A display with HDMI input for viewing the sample application output. + **USB drive** (included with AWS Panorama Appliance) – A FAT32-formatted USB 3.0 flash memory drive with at least 1 GB of storage, for transferring an archive with configuration files and a certificate to the AWS Panorama Appliance. + **Camera** – An IP camera that outputs an RTSP video stream. Use the tools and instructions provided by your camera's manufacturer to identify the camera's IP address and stream path. You can use a video player such as [VLC](https://www.videolan.org/) to verify the stream URL, by opening it as a network media source: ![\[\]](http://docs.aws.amazon.com/panorama/latest/dev/images/vlc-stream.png) The AWS Panorama console uses other AWS services to assemble application components, manage permissions, and verify settings. To register an appliance and deploy the sample application, you need the following permissions: **** + [AWSPanoramaFullAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AWSPanoramaFullAccess) – Provides full access to AWS Panorama, AWS Panorama access points in Amazon S3, appliance credentials in AWS Secrets Manager, and appliance logs in Amazon CloudWatch. Includes permission to create a [service-linked role](permissions-services.md) for AWS Panorama. + **AWS Identity and Access Management (IAM)** – On first run, to create roles used by the AWS Panorama service and the AWS Panorama Appliance. If you don't have permission to create roles in IAM, have an administrator open [the AWS Panorama console](https://console.aws.amazon.com/panorama/home) and accept the prompt to create service roles. ## Register and configure the AWS Panorama Appliance The AWS Panorama Appliance is a hardware device that connects to network-enabled cameras over a local network connection. It uses a Linux-based operating system that includes the AWS Panorama Application SDK and supporting software for running computer vision applications. To connect to AWS for appliance management and application deployment, the appliance uses a device certificate. You use the AWS Panorama console to generate a provisioning certificate. The appliance uses this temporary certificate to complete initial setup and download a permanent device certificate. **Important** The provisioning certificate that you generate in this procedure is only valid for 5 minutes. If you do not complete the registration process within this time frame, you must start over. **To register a appliance** 1. Connect the USB drive to your computer. Prepare the appliance by connecting the network and power cables. The appliance powers on and waits for a USB drive to be connected. 1. Open the AWS Panorama console [Getting started page](https://console.aws.amazon.com/panorama/home#getting-started). 1. Choose **Add device**. 1. Choose **Begin setup**. 1. Enter a name and description for the device resource that represents the appliance in AWS Panorama. Choose **Next** ![\[\]](http://docs.aws.amazon.com/panorama/latest/dev/images/setup-name.png) 1. If you need to manually assign an IP address, NTP server, or DNS settings, choose **Advanced network settings**. Otherwise, choose **Next**. 1. Choose **Download archive**. Choose **Next**. 1. Copy the configuration archive to the root directory of the USB drive. 1. Connect the USB drive to the USB 3.0 port on the front of the appliance, next to the HDMI port. When you connect the USB drive, the appliance copies the configuration archive and network configuration file to itself and connects to the AWS Cloud. The appliance's status light turns from green to blue while it completes the connection, and then back to green. 1. To continue, choose **Next**. ![\[\]](http://docs.aws.amazon.com/panorama/latest/dev/images/setup-poweron.png) 1. Choose **Done**. ## Upgrade the appliance software The AWS Panorama Appliance has several software components, including a Linux operating system, the [AWS Panorama application SDK](applications-panoramasdk.md), and supporting computer vision libraries and frameworks. To ensure that you can use the latest features and applications with your appliance, upgrade its software after setup and whenever an update is available. **To update the appliance software** 1. Open the AWS Panorama console [Devices page](https://console.aws.amazon.com/panorama/home#devices). 1. Choose an appliance. 1. Choose **Settings** 1. Under **System software**, choose **Install software update**. ![\[\]](http://docs.aws.amazon.com/panorama/latest/dev/images/setup-upgrade.png) 1. Choose a new version and then choose **Install**. **Important** Before you continue, remove the USB drive from the appliance and format it to delete its contents. The configuration archive contains sensitive data and is not deleted automatically. The upgrade process can take 30 minutes or more. You can monitor its progress in the AWS Panorama console or on a connected monitor. When the process completes, the appliance reboots. ## Add a camera stream Next, register a camera stream with the AWS Panorama console. **To register a camera stream** 1. Open the AWS Panorama console [Data sources page](https://console.aws.amazon.com/panorama/home#data-sources). 1. Choose **Add data source**. ![\[\]](http://docs.aws.amazon.com/panorama/latest/dev/images/setup-addstream.png) 1. Configure the following settings. **** + **Name** – A name for the camera stream. + **Description** – A short description of the camera, its location, or other details. + **RTSP URL** – A URL that specifies the camera's IP address and the path to the stream. For example, `rtsp://192.168.0.77/live/mpeg4/` + **Credentials** – If the camera stream is password protected, specify the username and password. 1. Choose **Save**. AWS Panorama stores your camera's credentials securely in AWS Secrets Manager. Multiple applications can process the same camera stream simultaneously. ## Next steps If you encountered errors during setup, see [Troubleshooting](panorama-troubleshooting.md). To deploy a sample application, continue to [the next topic](gettingstarted-deploy.md). # Deploying the AWS Panorama sample application Deploying an application After you've [set up your AWS Panorama Appliance or compatible device](gettingstarted-setup.md) and upgraded its software, deploy a sample application. In the following sections, you import a sample application with the AWS Panorama Application CLI and deploy it with the AWS Panorama console. The sample application uses a machine learning model to classify objects in frames of video from a network camera. It uses the AWS Panorama Application SDK to load a model, get images, and run the model. The application then overlays the results on top of the original video and outputs it to a connected display. In a retail setting, analyzing foot traffic patterns enables you to predict traffic levels. By combining the analysis with other data, you can plan for increased staffing needs around holidays and other events, measure the effectiveness of advertisements and sales promotions, or optimize display placement and inventory management. **Topics** + [ ## Prerequisites ](#gettingstarted-deploy-prerequisites) + [ ## Import the sample application ](#gettingstarted-deploy-import) + [ ## Deploy the application ](#gettingstarted-deploy-deploy) + [ ## View the output ](#gettingstarted-deploy-view) + [ ## Enable the SDK for Python ](#gettingstarted-deploy-redeploy) + [ ## Clean up ](#gettingstarted-deploy-cleanup) + [ ## Next steps ](#gettingstarted-deploy-next) ## Prerequisites To follow the procedures in this tutorial, you need a command line terminal or shell to run commands. In the code listings, commands are preceded by a prompt symbol (\$1) and the name of the current directory, when appropriate. ``` ~/panorama-project$ this is a command this is output ``` For long commands, we use an escape character (`\`) to split a command over multiple lines. On Linux and macOS, use your preferred shell and package manager. On Windows 10, you can [install the Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-win10) to get a Windows-integrated version of Ubuntu and Bash. For help setting up a development environment in Windows, see [Setting up a development environment in Windows](applications-devenvwindows.md). You use Python to develop AWS Panorama applications and install tools with pip, Python's package manager. If you don't already have Python, [install the latest version](https://www.python.org/downloads/). If you have Python 3 but not pip, install pip with your operating system's package manager, or install a new version of Python, which comes with pip. In this tutorial, you use Docker to build the container that runs your application code. Install Docker from the Docker website: [Get Docker](https://docs.docker.com/get-docker/) This tutorial uses the AWS Panorama Application CLI to import the sample application, build packages, and upload artifacts. The AWS Panorama Application CLI uses the AWS Command Line Interface (AWS CLI) to call service API operations. If you already have the AWS CLI, upgrade it to the latest version. To install the AWS Panorama Application CLI and AWS CLI, use `pip`. ``` $ pip3 install --upgrade awscli panoramacli ``` Download the sample application, and extract it into your workspace. **** + **Sample application** – [aws-panorama-sample.zip](https://github.com/awsdocs/aws-panorama-developer-guide/releases/download/v1.0-ga/aws-panorama-sample.zip) ## Import the sample application To import the sample application for use in your account, use the AWS Panorama Application CLI. The application's folders and manifest contain references to a placeholder account number. To update these with your account number, run the `panorama-cli import-application` command. ``` aws-panorama-sample$ panorama-cli import-application ``` The `SAMPLE_CODE` package, in the `packages` directory, contains the application's code and configuration, including a Dockerfile that uses the application base image, `panorama-application`. To build the application container that runs on the appliance, use the `panorama-cli build-container` command. ``` aws-panorama-sample$ ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account') aws-panorama-sample$ panorama-cli build-container --container-asset-name code_asset --package-path packages/${ACCOUNT_ID}-SAMPLE_CODE-1.0 ``` The final step with the AWS Panorama Application CLI is to register the application's code and model nodes, and upload assets to an Amazon S3 access point provided by the service. The assets include the code's container image, the model, and a descriptor file for each. To register the nodes and upload assets, run the `panorama-cli package-application` command. ``` aws-panorama-sample$ panorama-cli package-application Uploading package model Registered model with patch version bc9c58bd6f83743f26aa347dc86bfc3dd2451b18f964a6de2cc4570cb6f891f9 Uploading package code Registered code with patch version 11fd7001cb31ea63df6aaed297d600a5ecf641a987044a0c273c78ceb3d5d806 ``` ## Deploy the application Use the AWS Panorama console to deploy the application to your appliance. **To deploy the application** 1. Open the AWS Panorama console [Deployed applications page](https://console.aws.amazon.com/panorama/home#deployed-applications). 1. Choose **Deploy application**. 1. Paste the contents of the application manifest, `graphs/aws-panorama-sample/graph.json`, into the text editor. Choose **Next**. 1. For **Application name**, enter `aws-panorama-sample`. 1. Choose **Proceed to deploy**. 1. Choose **Begin deployment**. 1. Choose **Next** without selecting a role. 1. Choose **Select device**, and then choose your appliance. Choose **Next**. 1. On the **Select data sources** step, choose **View input(s)**, and add your camera stream as a data source. Choose **Next**. 1. On the **Configure** step, choose **Next**. 1. Choose **Deploy**, and then choose **Done**. 1. In the list of deployed applications, choose **aws-panorama-sample**. Refresh this page for updates, or use the following script to monitor the deployment from the command line. **Example monitor-deployment.sh** ``` while true; do aws panorama list-application-instances --query 'ApplicationInstances[?Name==`aws-panorama-sample`]' sleep 10 done ``` ``` [ { "Name": "aws-panorama-sample", "ApplicationInstanceId": "applicationInstance-x264exmpl33gq5pchc2ekoi6uu", "DefaultRuntimeContextDeviceName": "my-appliance", "Status": "DEPLOYMENT_PENDING", "HealthStatus": "NOT_AVAILABLE", "StatusDescription": "Deployment Workflow has been scheduled.", "CreatedTime": 1630010747.443, "Arn": "arn:aws:panorama:us-west-2:123456789012:applicationInstance/applicationInstance-x264exmpl33gq5pchc2ekoi6uu", "Tags": {} } ] [ { "Name": "aws-panorama-sample", "ApplicationInstanceId": "applicationInstance-x264exmpl33gq5pchc2ekoi6uu", "DefaultRuntimeContextDeviceName": "my-appliance", "Status": "DEPLOYMENT_PENDING", "HealthStatus": "NOT_AVAILABLE", "StatusDescription": "Deployment Workflow has completed data validation.", "CreatedTime": 1630010747.443, "Arn": "arn:aws:panorama:us-west-2:123456789012:applicationInstance/applicationInstance-x264exmpl33gq5pchc2ekoi6uu", "Tags": {} } ] ... ``` If the application doesn't start running, check the [application and device logs](monitoring-logging.md) in Amazon CloudWatch Logs. ## View the output When the deployment is complete, the application starts processing the video stream and sends logs to CloudWatch. **To view logs in CloudWatch Logs** 1. Open the [Log groups page of the CloudWatch Logs console](https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups). 1. Find AWS Panorama application and appliance logs in the following groups: **** + **Device logs** – `/aws/panorama/devices/device-id` + **Application logs** – `/aws/panorama/devices/device-id/applications/instance-id` ``` 2022-08-26 17:43:39 INFO INITIALIZING APPLICATION 2022-08-26 17:43:39 INFO ## ENVIRONMENT VARIABLES {'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'TERM': 'xterm', 'container': 'podman'...} 2022-08-26 17:43:39 INFO Configuring parameters. 2022-08-26 17:43:39 INFO Configuring AWS SDK for Python. 2022-08-26 17:43:39 INFO Initialization complete. 2022-08-26 17:43:39 INFO PROCESSING STREAMS 2022-08-26 17:46:19 INFO epoch length: 160.183 s (0.936 FPS) 2022-08-26 17:46:19 INFO avg inference time: 805.597 ms 2022-08-26 17:46:19 INFO max inference time: 120023.984 ms 2022-08-26 17:46:19 INFO avg frame processing time: 1065.129 ms 2022-08-26 17:46:19 INFO max frame processing time: 149813.972 ms 2022-08-26 17:46:29 INFO epoch length: 10.562 s (14.202 FPS) 2022-08-26 17:46:29 INFO avg inference time: 7.185 ms 2022-08-26 17:46:29 INFO max inference time: 15.693 ms 2022-08-26 17:46:29 INFO avg frame processing time: 66.561 ms 2022-08-26 17:46:29 INFO max frame processing time: 123.774 ms ``` To view the application's video output, connect the appliance to a monitor with an HDMI cable. By default, the application shows any classification result that has more than 20% confidence. **Example [squeezenet\$1classes.json](https://github.com/awsdocs/aws-panorama-developer-guide/blob/main/sample-apps/aws-panorama-sample/packages/123456789012-SAMPLE_CODE-1.0/squeezenet_classes.json)** ``` ["tench", "goldfish", "great white shark", "tiger shark", "hammerhead", "electric ray", "stingray", "cock", "hen", "ostrich", "brambling", "goldfinch", "house finch", "junco", "indigo bunting", "robin", "bulbul", "jay", "magpie", "chickadee", "water ouzel", "kite", "bald eagle", "vulture", "great grey owl", "European fire salamander", "common newt", "eft", "spotted salamander", "axolotl", "bullfrog", "tree frog", ... ``` The sample model has 1000 classes including many animals, food, and common objects. Try pointing your camera at a keyboard or coffee mug. ![\[\]](http://docs.aws.amazon.com/panorama/latest/dev/images/mug.jpg) For simplicity, the sample application uses a lightweight classification model. The model outputs a single array with a probability for each of its classes. Real-world applications more frequently use object detection models that have multidimensional output. For sample applications with more complex models, see [Sample applications, scripts, and templates](panorama-samples.md). ## Enable the SDK for Python The sample application uses the AWS SDK for Python (Boto) to send metrics to Amazon CloudWatch. To enable this functionality, create a role that grants the application permission to send metrics, and redeploy the application with the role attached. The sample application includes a CloudFormation template that creates a role with the permissions that it needs. To create the role, use the `aws cloudformation deploy` command. ``` $ aws cloudformation deploy --template-file aws-panorama-sample.yml --stack-name aws-panorama-sample-runtime --capabilities CAPABILITY_NAMED_IAM ``` **To redeploy the application** 1. Open the AWS Panorama console [Deployed applications page](https://console.aws.amazon.com/panorama/home#deployed-applications). 1. Choose an application. 1. Choose **Replace**. 1. Complete the steps to deploy the application. In the **Specify IAM role**, choose the role that you created. Its name starts with `aws-panorama-sample-runtime`. 1. When the deployment completes, open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/home#metricsV2:graph=~();namespace=~'AWSPanoramaApplication) and view the metrics in the `AWSPanoramaApplication` namespace. Every 150 frames, the application logs and uploads metrics for frame processing and inference time. ## Clean up If you are done working with the sample application, you can use the AWS Panorama console to remove it from the appliance. **To remove the application from the appliance** 1. Open the AWS Panorama console [Deployed applications page](https://console.aws.amazon.com/panorama/home#deployed-applications). 1. Choose an application. 1. Choose **Delete from device**. ## Next steps If you encountered errors while deploying or running the sample application, see [Troubleshooting](panorama-troubleshooting.md). To learn more about the sample application's features and implementation, continue to [the next topic](gettingstarted-sample.md). # Developing AWS Panorama applications Developing applications You can use the sample application to learn about AWS Panorama application structure, and as a starting point for your own application. The following diagram shows the major components of the application running on an AWS Panorama Appliance. The application code uses the AWS Panorama Application SDK to get images and interact with the model, which it doesn't have direct access to. The application outputs video to a connected display but does not send image data outside of your local network. ![\[\]](http://docs.aws.amazon.com/panorama/latest/dev/images/sample-app.png) In this example, the application uses the AWS Panorama Application SDK to get frames of video from a camera, preprocess the video data, and send the data to a computer vision model that detects objects. The application displays the result on an HDMI display connected to the appliance. **Topics** + [ ## The application manifest ](#gettingstarted-sample-manifest) + [ ## Building with the sample application ](#gettingstarted-sample-adapting) + [ ## Changing the computer vision model ](#gettingstarted-sample-model) + [ ## Preprocessing images ](#gettingstarted-sample-preprocessing) + [ ## Uploading metrics with the SDK for Python ](#gettingstarted-sample-metrics) + [ ## Next steps ](#gettingstarted-sample-nextsteps) ## The application manifest The application manifest is a file named `graph.json` in the `graphs` folder. The manifest defines the application's components, which are packages, nodes, and edges. Packages are code, configuration, and binary files for application code, models, cameras, and displays. The sample application uses 4 packages: **Example `graphs/aws-panorama-sample/graph.json` – Packages** ``` "packages": [ { "name": "123456789012::SAMPLE_CODE", "version": "1.0" }, { "name": "123456789012::SQUEEZENET_PYTORCH_V1", "version": "1.0" }, { "name": "panorama::abstract_rtsp_media_source", "version": "1.0" }, { "name": "panorama::hdmi_data_sink", "version": "1.0" } ], ``` The first two packages are defined within the application, in the `packages` directory. They contain the code and model specific to this application. The second two packages are generic camera and display packages provided by the AWS Panorama service. The `abstract_rtsp_media_source` package is a placeholder for a camera that you override during deployment. The `hdmi_data_sink` package represents the HDMI output connector on the device. Nodes are interfaces to packages, as well as non-package parameters that can have default values that you override at deploy time. The code and model packages define interfaces in `package.json` files that specify inputs and outputs, which can be video streams or a basic data type such as a float, boolean, or string. For example, the `code_node` node refers to an interface from the `SAMPLE_CODE` package. ``` "nodes": [ { "name": "code_node", "interface": "123456789012::SAMPLE_CODE.interface", "overridable": false, "launch": "onAppStart" }, ``` This interface is defined in the package configuration file, `package.json`. The interface specifies that the package is business logic and that it takes a video stream named `video_in` and a floating point number named `threshold` as inputs. The interface also specifies that the code requires a video stream buffer named `video_out` to output video to a display **Example `packages/123456789012-SAMPLE_CODE-1.0/package.json`** ``` { "nodePackage": { "envelopeVersion": "2021-01-01", "name": "SAMPLE_CODE", "version": "1.0", "description": "Computer vision application code.", "assets": [], "interfaces": [ { "name": "interface", "category": "business_logic", "asset": "code_asset", "inputs": [ { "name": "video_in", "type": "media" }, { "name": "threshold", "type": "float32" } ], "outputs": [ { "description": "Video stream output", "name": "video_out", "type": "media" } ] } ] } } ``` Back in the application manifest, the `camera_node` node represents a video stream from a camera. It includes a decorator that appears in the console when you deploy the application, prompting you to choose a camera stream. **Example `graphs/aws-panorama-sample/graph.json` – Camera node** ``` { "name": "camera_node", "interface": "panorama::abstract_rtsp_media_source.rtsp_v1_interface", "overridable": true, "launch": "onAppStart", "decorator": { "title": "Camera", "description": "Choose a camera stream." } }, ``` A parameter node, `threshold_param`, defines the confidence threshold parameter used by the application code. It has a default value of 60, and can be overriden during deployment. **Example `graphs/aws-panorama-sample/graph.json` – Parameter node** ``` { "name": "threshold_param", "interface": "float32", "value": 60.0, "overridable": true, "decorator": { "title": "Confidence threshold", "description": "The minimum confidence for a classification to be recorded." } } ``` The final section of the application manifest, `edges`, makes connections between nodes. The camera's video stream and the threshold parameter connect to the input of the code node, and the video output from the code node connects to the display. **Example `graphs/aws-panorama-sample/graph.json` – Edges** ``` "edges": [ { "producer": "camera_node.video_out", "consumer": "code_node.video_in" }, { "producer": "code_node.video_out", "consumer": "output_node.video_in" }, { "producer": "threshold_param", "consumer": "code_node.threshold" } ] ``` ## Building with the sample application You can use the sample application as a starting point for your own application. The name of each package must be unique in your account. If you and another user in your account both use a generic package name such as `code` or `model`, you might get the wrong version of the package when you deploy. Change the name of the code package to one that represents your application. **To rename the code package** 1. Rename the package folder: `packages/123456789012-SAMPLE_CODE-1.0/`. 1. Update the package name in the following locations. **** + **Application manifest** – `graphs/aws-panorama-sample/graph.json` + **Package configuration** – `packages/123456789012-SAMPLE_CODE-1.0/package.json` + **Build script** – `3-build-container.sh` **To update the application's code** 1. Modify the application code in `packages/123456789012-SAMPLE_CODE-1.0/src/application.py`. 1. To build the container, run `3-build-container.sh`. ``` aws-panorama-sample$ ./3-build-container.sh TMPDIR=$(pwd) docker build -t code_asset packages/123456789012-SAMPLE_CODE-1.0 Sending build context to Docker daemon 61.44kB Step 1/2 : FROM public.ecr.aws/panorama/panorama-application ---> 9b197f256b48 Step 2/2 : COPY src /panorama ---> 55c35755e9d2 Successfully built 55c35755e9d2 Successfully tagged code_asset:latest docker export --output=code_asset.tar $(docker create code_asset:latest) gzip -9 code_asset.tar Updating an existing asset with the same name { "name": "code_asset", "implementations": [ { "type": "container", "assetUri": "98aaxmpl1c1ef64cde5ac13bd3be5394e5d17064beccee963b4095d83083c343.tar.gz", "descriptorUri": "1872xmpl129481ed053c52e66d6af8b030f9eb69b1168a29012f01c7034d7a8f.json" } ] } Container asset for the package has been succesfully built at ~/aws-panorama-sample-dev/assets/98aaxmpl1c1ef64cde5ac13bd3be5394e5d17064beccee963b4095d83083c343.tar.gz ``` The CLI automatically deletes the old container asset from the `assets` folder and updates the package configuration. 1. To upload the packages, run `4-package-application.py`. 1. Open the AWS Panorama console [Deployed applications page](https://console.aws.amazon.com/panorama/home#deployed-applications). 1. Choose an application. 1. Choose **Replace**. 1. Complete the steps to deploy the application. If needed, you can make changes to the application manifest, camera streams, or parameters. ## Changing the computer vision model The sample application includes a computer vision model. To use your own model, modify the model node's configuration, and use the AWS Panorama Application CLI to import it as an asset. The following example uses an MXNet SSD ResNet50 model that you can download from this guide's GitHub repo: [ssd\$1512\$1resnet50\$1v1\$1voc.tar.gz](https://github.com/awsdocs/aws-panorama-developer-guide/releases/download/v0.1-preview/ssd_512_resnet50_v1_voc.tar.gz) **To change the sample application's model** 1. Rename the package folder to match your model. For example, to `packages/123456789012-SSD_512_RESNET50_V1_VOC-1.0/`. 1. Update the package name in the following locations. **** + **Application manifest** – `graphs/aws-panorama-sample/graph.json` + **Package configuration** – `packages/123456789012-SSD_512_RESNET50_V1_VOC-1.0/package.json` 1. In the package configuration file (`package.json`). Change the `assets` value to a blank array. ``` { "nodePackage": { "envelopeVersion": "2021-01-01", "name": "SSD_512_RESNET50_V1_VOC", "version": "1.0", "description": "Compact classification model", "assets": [], ``` 1. Open the package descriptor file (`descriptor.json`). Update the `framework` and `shape` values to match your model. ``` { "mlModelDescriptor": { "envelopeVersion": "2021-01-01", "framework": "MXNET", "inputs": [ { "name": "data", "shape": [ 1, 3, 512, 512 ] } ] } } ``` The value for **shape**, `1,3,512,512`, indicates the number of images that the model takes as input (1), the number of channels in each image (3--red, green, and blue), and the dimensions of the image (512 x 512). The values and order of the array varies among models. 1. Import the model with the AWS Panorama Application CLI. The AWS Panorama Application CLI copies the model and descriptor files into the `assets` folder with unique names, and updates the package configuration. ``` aws-panorama-sample$ panorama-cli add-raw-model --model-asset-name model-asset \ --model-local-path ssd_512_resnet50_v1_voc.tar.gz \ --descriptor-path packages/123456789012-SSD_512_RESNET50_V1_VOC-1.0/descriptor.json \ --packages-path packages/123456789012-SSD_512_RESNET50_V1_VOC-1.0 { "name": "model-asset", "implementations": [ { "type": "model", "assetUri": "b1a1589afe449b346ff47375c284a1998c3e1522b418a7be8910414911784ce1.tar.gz", "descriptorUri": "a6a9508953f393f182f05f8beaa86b83325f4a535a5928580273e7fe26f79e78.json" } ] } ``` 1. To upload the model, run `panorama-cli package-application`. ``` $ panorama-cli package-application Uploading package SAMPLE_CODE Patch Version 1844d5a59150d33f6054b04bac527a1771fd2365e05f990ccd8444a5ab775809 already registered, ignoring upload Uploading package SSD_512_RESNET50_V1_VOC Patch version for the package 244a63c74d01e082ad012ebf21e67eef5d81ce0de4d6ad1ae2b69d0bc498c8fd upload: assets/b1a1589afe449b346ff47375c284a1998c3e1522b418a7be8910414911784ce1.tar.gz to s3://arn:aws:s3:us-west-2:454554846382:accesspoint/panorama-123456789012-wc66m5eishf4si4sz5jefhx 63a/123456789012/nodePackages/SSD_512_RESNET50_V1_VOC/binaries/b1a1589afe449b346ff47375c284a1998c3e1522b418a7be8910414911784ce1.tar.gz upload: assets/a6a9508953f393f182f05f8beaa86b83325f4a535a5928580273e7fe26f79e78.json to s3://arn:aws:s3:us-west-2:454554846382:accesspoint/panorama-123456789012-wc66m5eishf4si4sz5jefhx63 a/123456789012/nodePackages/SSD_512_RESNET50_V1_VOC/binaries/a6a9508953f393f182f05f8beaa86b83325f4a535a5928580273e7fe26f79e78.json { "ETag": "\"2381dabba34f4bc0100c478e67e9ab5e\"", "ServerSideEncryption": "AES256", "VersionId": "KbY5fpESdpYamjWZ0YyGqHo3.LQQWUC2" } Registered SSD_512_RESNET50_V1_VOC with patch version 244a63c74d01e082ad012ebf21e67eef5d81ce0de4d6ad1ae2b69d0bc498c8fd Uploading package SQUEEZENET_PYTORCH_V1 Patch Version 568138c430e0345061bb36f05a04a1458ac834cd6f93bf18fdacdffb62685530 already registered, ignoring upload ``` 1. Update the application code. Most of the code can be reused. The code specific to the model's response is in the `process_results` method. ``` def process_results(self, inference_results, stream): """Processes output tensors from a computer vision model and annotates a video frame.""" for class_tuple in inference_results: indexes = self.topk(class_tuple[0]) for j in range(2): label = 'Class [%s], with probability %.3f.'% (self.classes[indexes[j]], class_tuple[0][indexes[j]]) stream.add_label(label, 0.1, 0.25 + 0.1*j) ``` Depending on your model, you might also need to update the `preprocess` method. ## Preprocessing images Before the application sends an image to the model, it prepares it for inference by resizing it and normalizing color data. The model that the application uses requires a 224 x 224 pixel image with three color channels, to match the number of inputs in its first layer. The application adjusts each color value by converting it to a number between 0 and 1, subtracting the average value for that color, and dividing by the standard deviation. Finally, it combines the color channels and converts it to a NumPy array that the model can process. **Example [application.py](https://github.com/awsdocs/aws-panorama-developer-guide/blob/main/sample-apps/aws-panorama-sample/packages/123456789012-SAMPLE_CODE-1.0/application.py) – Preprocessing** ``` def preprocess(self, img, width): resized = cv2.resize(img, (width, width)) mean = [0.485, 0.456, 0.406] std = [0.229, 0.224, 0.225] img = resized.astype(np.float32) / 255. img_a = img[:, :, 0] img_b = img[:, :, 1] img_c = img[:, :, 2] # Normalize data in each channel img_a = (img_a - mean[0]) / std[0] img_b = (img_b - mean[1]) / std[1] img_c = (img_c - mean[2]) / std[2] # Put the channels back together x1 = [[[], [], []]] x1[0][0] = img_a x1[0][1] = img_b x1[0][2] = img_c return np.asarray(x1) ``` This process gives the model values in a predictable range centered around 0. It matches the preprocessing applied to images in the training dataset, which is a standard approach but can vary per model. ## Uploading metrics with the SDK for Python The sample application uses the SDK for Python to upload metrics to Amazon CloudWatch. **Example [application.py](https://github.com/awsdocs/aws-panorama-developer-guide/blob/main/sample-apps/aws-panorama-sample/packages/123456789012-SAMPLE_CODE-1.0/application.py) – SDK for Python** ``` def process_streams(self): """Processes one frame of video from one or more video streams.""" ... logger.info('epoch length: {:.3f} s ({:.3f} FPS)'.format(epoch_time, epoch_fps)) logger.info('avg inference time: {:.3f} ms'.format(avg_inference_time)) logger.info('max inference time: {:.3f} ms'.format(max_inference_time)) logger.info('avg frame processing time: {:.3f} ms'.format(avg_frame_processing_time)) logger.info('max frame processing time: {:.3f} ms'.format(max_frame_processing_time)) self.inference_time_ms = 0 self.inference_time_max = 0 self.frame_time_ms = 0 self.frame_time_max = 0 self.epoch_start = time.time() self.put_metric_data('AverageInferenceTime', avg_inference_time) self.put_metric_data('AverageFrameProcessingTime', avg_frame_processing_time) def put_metric_data(self, metric_name, metric_value): """Sends a performance metric to CloudWatch.""" namespace = 'AWSPanoramaApplication' dimension_name = 'Application Name' dimension_value = 'aws-panorama-sample' try: metric = self.cloudwatch.Metric(namespace, metric_name) metric.put_data( Namespace=namespace, MetricData=[{ 'MetricName': metric_name, 'Value': metric_value, 'Unit': 'Milliseconds', 'Dimensions': [ { 'Name': dimension_name, 'Value': dimension_value }, { 'Name': 'Device ID', 'Value': self.device_id } ] }] ) logger.info("Put data for metric %s.%s", namespace, metric_name) except ClientError: logger.warning("Couldn't put data for metric %s.%s", namespace, metric_name) except AttributeError: logger.warning("CloudWatch client is not available.") ``` It gets permission from a runtime role that you assign during deployment. The role is defined in the `aws-panorama-sample.yml` CloudFormation template. **Example [aws-panorama-sample.yml](https://github.com/awsdocs/aws-panorama-developer-guide/blob/main/sample-apps/aws-panorama-sample/aws-panorama-sample.yml)** ``` Resources: runtimeRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: - panorama.amazonaws.com Action: - sts:AssumeRole Policies: - PolicyName: cloudwatch-putmetrics PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: 'cloudwatch:PutMetricData' Resource: '*' Path: /service-role/ ``` The sample application installs the SDK for Python and other dependencies with pip. When you build the application container, the `Dockerfile` runs commands to install libraries on top of what comes with the base image. **Example [Dockerfile](https://github.com/awsdocs/aws-panorama-developer-guide/blob/main/sample-apps/aws-panorama-sample/packages/123456789012-SAMPLE_CODE-1.0/Dockerfile)** ``` FROM public.ecr.aws/panorama/panorama-application WORKDIR /panorama COPY . . RUN pip install --no-cache-dir --upgrade pip && \ pip install --no-cache-dir -r requirements.txt ``` To use the AWS SDK in your application code, first modify the template to add permissions for all API actions that the application uses. Update the CloudFormation stack by running the `1-create-role.sh` each time you make a change. Then, deploy changes to your application code. For actions that modify or use existing resources, it is a best practice to minimize the scope of this policy by specifying a name or pattern for the target `Resource` in a separate statement. For details on the actions and resources supported by each service, see [Action, resources, and condition keys](https://docs.aws.amazon.com/service-authorization/latest/reference/reference_policies_actions-resources-contextkeys.html) in the Service Authorization Reference ## Next steps For instructions on using the AWS Panorama Application CLI to build applications and create packages from scratch, see the CLI's README. **** + [github.com/aws/aws-panorama-cli](https://github.com/aws/aws-panorama-cli) For more sample code and a test utility that you can use to validate your application code prior to deploying, visit the AWS Panorama samples repository. **** + [github.com/aws-samples/aws-panorama-samples](https://github.com/aws-samples/aws-panorama-samples) # Supported computer vision models and cameras Supported models and cameras AWS Panorama supports models built with PyTorch, Apache MXNet, and TensorFlow. When you deploy an application, AWS Panorama compiles your model in SageMaker AI Neo. You can build models in Amazon SageMaker AI or in your development environment, as long as you use layers that are compatible with SageMaker AI Neo. To process video and get images to send to a model, the AWS Panorama Appliance connects to an H.264 encoded video stream with the RTSP protocol. AWS Panorama tests a variety of common cameras for compatibility. **Topics** + [ ## Supported models ](#gettingstarted-compatibility-models) + [ ## Supported cameras ](#gettingstarted-compatibility-cameras) ## Supported models When you build an application for AWS Panorama, you provide a machine learning model that the application uses for computer vision. You can use pre-built and pre-trained models provided by model frameworks, [a sample model](gettingstarted-sample.md#gettingstarted-sample-model), or a model that you build and train yourself. **Note** For a list of pre-built models that have been tested with AWS Panorama, see [Model compatibility](https://github.com/awsdocs/aws-panorama-developer-guide/blob/main/resources/model-compatibility.md). When you deploy an application, AWS Panorama uses the SageMaker AI Neo compiler to compile your computer vision model. SageMaker AI Neo is a compiler that optimizes models to run efficiently on a target platform, which can be an instance in Amazon Elastic Compute Cloud (Amazon EC2), or an edge device such as the AWS Panorama Appliance. AWS Panorama supports the versions of PyTorch, Apache MXNet, and TensorFlow that are supported for edge devices by SageMaker AI Neo. When you build your own model, you can use the framework versions listed in the [SageMaker AI Neo release notes](https://aws.amazon.com/releasenotes/sagemaker-neo-supported-frameworks-and-operators/). In SageMaker AI, you can use the built-in [image classification algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/image-classification.html). For more information about using models in AWS Panorama, see [Computer vision models](applications-models.md). ## Supported cameras The AWS Panorama Appliance supports H.264 video streams from cameras that output RTSP over a local network. For camera streams greater than 2 megapixels, the appliance scales down the image to 1920x1080 pixels or an equivalent size that preserves the stream's aspect ratio. The following camera models have been tested for compatibility with the AWS Panorama Appliance: + [Axis](https://www.axis.com/) – M3057-PLVE, M3058-PLVE, P1448-LE, P3225-LV Mk II + [LaView](https://www.laviewsecurity.com/) – LV-PB3040W + [Vivotek](https://www.vivotek.com/) – IB9360-H + [Amcrest](https://amcrest.com/) – IP2M-841B + **Anpviz** – IPC-B850W-S-3X, IPC-D250W-S + **WGCC** – Dome PoE 4MP ONVIF For the appliance's hardware specifications, see [AWS Panorama Appliance specifications](gettingstarted-hardware.md). # AWS Panorama Appliance specifications Appliance specifications The AWS Panorama Appliance has the following hardware specifications. For other [compatible devices](gettingstarted-concepts.md#gettingstarted-concepts-devices), refer to the manufacturer's documentation. | Component | Specification | | --- | --- | | Processor and GPU | [Nvidia Jetson AGX Xavier](https://developer.nvidia.com/embedded/jetson-agx-xavier-developer-kit) with 32GB RAM | | Ethernet | 2x 1000 Base-T (Gigabyte) | | USB | 1x USB 2.0 and 1x USB 3.0 type-A female | | HDMI output | 2.0a | | Dimensions | 7.75” x 9.6” x 1.6” (197mm x 243mm x 40mm) | | Weight | 3.7lbs (1.7kg) | | Power supply | 100V-240V 50-60Hz AC 65W | | Power input | IEC 60320 C6 (3-pin) receptacle | | Dust and liquid protection | IP-62 | | EMI/EMC regulatory compliance | FCC Part-15 (US) | | Thermal touch limits | IEC-62368 | | Operating temperature | -20°C to 60°C | | Operating humidity | 0% to 95% RH | | Storage temperature | -20°C to 85°C | | Storage humidity | Uncontrolled for low temperature. 90% RH at high temperature | | Cooling | Forced-air heat extraction (fan) | | Mounting options | Rackmount or free standing | | Power cord | 6-foot (1.8 meter) | | Power control | Push-button | | Reset | Momentary switch | | Status and network LEDs | Programmable 3-color RGB LED | Wi-Fi, Bluetooth and SD card storage are present on the appliance but are not usable. The AWS Panorama Appliance includes two screws for mounting on a server rack. You can mount two appliances side-by-side on a 19-inch rack. # Service quotas Quotas AWS Panorama applies quotas to the resources that you create in your account and the applications that you deploy. If you use AWS Panorama in multiple AWS Regions, quotas apply separately to each Region. AWS Panorama quotas are not adjustable. Resources in AWS Panorama include devices, application node packages, and application instances. **** + **Devices** – Up to 50 registered appliances per Region. + **Node packages** – 50 packages per Region, with up to 20 versions per package. + **Application instances** – Up to 10 applications per device. Each application can monitor up to 8 camera streams. Deployments are limited to 200 per day for each device. When you use the AWS Panorama Application CLI, AWS Command Line Interface, or AWS SDK with the AWS Panorama service, quotas apply to the number of API calls that you make. You can make up to 5 requests total per second. A subset of API operations that create or modify resources apply an additional limit of 1 request per second. For a complete list of quotas, visit the [Service Quotas console](https://console.aws.amazon.com/servicequotas/home/services/panorama/quotas), or see [AWS Panorama endpoints and quotas ](https://docs.aws.amazon.com/general/latest/gr/panorama.html) in the Amazon Web Services General Reference.