# Paravirtual drivers for Windows instances Windows PV drivers Windows AMIs contain a set of drivers to permit access to virtualized hardware. These drivers are used by Amazon EC2 to map instance store and Amazon EBS volumes to their devices. The following table shows key differences between the different drivers. | | Red Hat PV | Citrix PV | AWS PV | | --- | --- | --- | --- | | Instance type | Not supported for all instance types. If you specify an unsupported instance type, the instance is impaired. | Supported for Xen instance types. | Supported for Xen instance types. | | Attached volumes | Supports up to 16 attached volumes. | Supports more than 16 attached volumes. | Supports more than 16 attached volumes. | | Network | The driver has known issues where the network connection resets under high loads; for example, fast FTP file transfers. | | The driver automatically configures jumbo frames on the network adapter when on a compatible instance type. When the instance is in a cluster placement group, this offers better network performance between instances that are in the cluster placement group. For more information, see [Placement groups for your Amazon EC2 instances](placement-groups.md). | The following table shows which PV drivers you should run on each version of Windows Server on Amazon EC2. | Windows Server version | PV driver version | | --- | --- | | Windows Server 2025 | Not supported | | Windows Server 2022 | AWS PV latest version | | Windows Server 2019 | AWS PV latest version | | Windows Server 2016 | AWS PV latest version | | Windows Server 2012 R2 | AWS PV version 8.4.3 | | Windows Server 2012 | AWS PV version 8.4.3 | | Windows Server 2008 R2 | AWS PV version 8.3.5 | | Windows Server 2008 | Citrix PV 5.9 | | Windows Server 2003 | Citrix PV 5.9 | **Topics** + [ ## AWS PV drivers ](#xen-driver-awspv) + [ ## Citrix PV drivers ](#xen-driver-citrix) + [ ## Red Hat PV drivers ](#xen-driver-redhat) + [ ## Subscribe to notifications ](#drivers-subscribe-notifications) + [ # Upgrade PV drivers on EC2 Windows instances ](Upgrading_PV_drivers.md) + [ # Troubleshoot PV drivers on Windows instances ](pvdrivers-troubleshooting.md) ## AWS PV drivers The AWS PV drivers are stored in the `%ProgramFiles%\Amazon\Xentools` directory. This directory also contains public symbols and a command line tool, `xenstore_client.exe`, that enables you to access entries in XenStore. For example, the following PowerShell command returns the current time from the Hypervisor: ``` PS C:\> [DateTime]::FromFileTimeUTC((gwmi -n root\wmi -cl AWSXenStoreBase).XenTime).ToString("hh:mm:ss") 11:17:00 ``` The AWS PV driver components are listed in the Windows registry under `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services`. These driver components are as follows: xenbus, xeniface, xennet, xenvbd, and xenvif. AWS PV drivers also have a Windows service named LiteAgent, which runs in user-mode. It handles tasks such as shutdown and restart events from AWS APIs on Xen generation instances. You can access and manage services by running `Services.msc` from the command line. When running on Nitro generation instances, the AWS PV drivers are not used and the LiteAgent service will self-stop starting with driver version 8.2.4. Updating to the latest AWS PV driver also updates the LiteAgent and improves reliability on all instance generations. ### Install the latest AWS PV drivers Amazon Windows AMIs contain a set of drivers to permit access to virtualized hardware. These drivers are used by Amazon EC2 to map instance store and Amazon EBS volumes to their devices. We recommend that you install the latest drivers to improve stability and performance of your EC2 Windows instances. **Installation options** + Use AWS Systems Manager to automatically update the PV drivers. For more information, see [Walkthrough: Automatically Update PV Drivers on EC2 Windows Instances](https://docs.aws.amazon.com/systems-manager/latest/userguide/state-manager-update-pv-drivers.html) in the *AWS Systems Manager User Guide*. + [Download](https://s3.amazonaws.com/ec2-windows-drivers-downloads/AWSPV/Latest/AWSPVDriver.zip) the driver package and run the install program manually. Be sure to check the `readme.txt` file for system requirements. For information about downloading and installing the AWS PV drivers, or upgrading a domain controller, see [Upgrade Windows Server instances (AWS PV upgrade) manually](Upgrading_PV_drivers.md#aws-pv-upgrade). ### AWS PV driver package history The following table shows the changes to AWS PV drivers for each driver release. | Package version | Details | Release date | | --- | --- | --- | | [8.6.0](https://s3.amazonaws.com/ec2-windows-drivers-downloads/AWSPV/8.6.0/AWSPVDriver.zip) | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/xen-drivers-overview.html) | 27 May 2025 | | 8.5.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/xen-drivers-overview.html) | 31 October 2024 | | [8.4.3](https://s3.amazonaws.com/ec2-windows-drivers-downloads/AWSPV/8.4.3/AWSPVDriver.zip) | Fixed bugs in the package installer to improve the upgrade experience. This is the last version that can run on Windows Server 2012 and 2012 R2. This version is available for download, however it is no longer supported since Windows Server 2012 and 2012 R2 have reached end of support. | 24 January 2023 | | 8.4.2 | Stability fixes to address race condition. | 13 April 2022 | | 8.4.1 | Improved package installer. | 7 January 2022 | | 8.4.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/xen-drivers-overview.html) | 2 March 2021 | | [8.3.5](https://s3.amazonaws.com/ec2-windows-drivers-downloads/AWSPV/8.3.5/AWSPVDriver.zip) | Improved package installer. This is the last version that can run on Windows Server 2008 R2. This version is available for download but no longer supported. Windows Server 2008 R2 has reached end-of-life, and is no longer supported by Microsoft. | 7 January 2022 | | 8.3.4 | Improved reliability of network device attachment. | 4 August 2020 | | 8.3.3 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/xen-drivers-overview.html) To update this driver on Windows Server 2008 R2 instances, you must first verify that the appropriate patches are installed to address the following Microsoft Security Advisory: [Microsoft Security Advisory 3033929](https://learn.microsoft.com/en-us/security-updates/SecurityAdvisories/2015/3033929). | 4 February 2020 | | 8.3.2 | Enhanced reliability of networking components. | 30 July 2019 | | 8.3.1 | Improved performance and robustness of storage component. | 12 June 2019 | | 8.2.7 | Improved efficiency to support migrating to latest generation instance types. | 20 May 2019 | | 8.2.6 | Improved efficiency of crash dump path. | 15 January 2019 | | 8.2.5 | Additional security enhancements. PowerShell installer now available in package. | 12 December 2018 | | 8.2.4 | Reliability improvements. | 2 October 2018 | | 8.2.3 | Bug fixes and performance improvements. Report EBS volume ID as disk serial number for EBS volumes. This enables cluster scenarios such as S2D. | 29 May 2018 | | 8.2.1 | Network and storage performance improvements plus multiple robustness fixes. To verify that this version has been installed, refer to the following Windows registry value: `HKLM\Software\Amazon\PVDriver\Version 8.2.1`. | 8 March 2018 | | 7.4.3 | Added support for Windows Server 2016. Stability fixes for all supported Windows OS versions. \$1AWS PV driver version 7.4.3's signature expires on March 29, 2019. We recommend updating to the latest AWS PV driver. | 18 Nov 2016 | | 7.4.2 | Stability fixes for support of X1 instance type. | 2 Aug 2016 | | 7.4.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/xen-drivers-overview.html) | 12 July 2016 | | 7.3.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/xen-drivers-overview.html) | 24 June 2015 | | 7.3.1 | TRIM update: Fix related to TRIM requests. This fix stabilizes instances and improves instance performance when managing large numbers of TRIM requests. | | | 7.3.0 | TRIM support: The AWS PV driver now sends TRIM requests to the hypervisor. Ephemeral disks will properly process TRIM requests given the underlying storage supports TRIM (SSD). Note that EBS-based storage does not support TRIM as of March 2015. | | | 7.2.5 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/xen-drivers-overview.html) | | | 7.2.4 | Device ID persistence: This driver fix masks the platform PCI device ID and forces the system to always surface the same device ID, even if the instance is moved. More generally, the fix affects how the hypervisor surfaces virtual devices. The fix also includes modifications to the co-installer for the AWS PV drivers so the system persists mapped virtual devices. | | | 7.2.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/xen-drivers-overview.html) | | | 7.2.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/xen-drivers-overview.html) | | | 7.2.0 | Base: The AWS PV base version. | | ## Citrix PV drivers The Citrix PV drivers are stored in the `%ProgramFiles%\Citrix\XenTools` (32-bit instances) or `%ProgramFiles(x86)%\Citrix\XenTools` (64-bit instances) directory. The Citrix PV driver components are listed in the Windows registry under `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services`. These driver components are as follows: xenevtchn, xeniface, xennet, Xennet6, xensvc, xenvbd, and xenvif. Citrix also has a driver component named XenGuestAgent, which runs as a Windows service. It handles tasks such as shutdown and restart events from the API. You can access and manage services by running `Services.msc` from the command line. If you are encountering networking errors while performing certain workloads, you may need to disable the TCP offloading feature for the Citrix PV driver. For more information, see [TCP offloading](pvdrivers-troubleshooting.md#citrix-tcp-offloading). ## Red Hat PV drivers Red Hat drivers are supported for legacy instances, but are not recommended on newer instances with more than 12GB of RAM due to driver limitations. Instances with more than 12GB of RAM running Red Hat drivers can fail to boot and become inaccessible. We recommend upgrading Red Hat drivers to Citrix PV drivers, and then upgrade Citrix PV drivers to AWS PV drivers. The source files for the Red Hat drivers are in the `%ProgramFiles%\RedHat` (32-bit instances) or `%ProgramFiles(x86)%\RedHat` (64-bit instances) directory. The two drivers are `rhelnet`, the Red Hat Paravirtualized network driver, and `rhelscsi`, the Red Hat SCSI miniport driver. ## Subscribe to notifications Amazon SNS can notify you when new versions of EC2 Windows Drivers are released. You can subscribe to these notifications. **Note** You must specify the Region in which the SNS topic was created. Whenever new EC2 Windows drivers are released, we send notifications to subscribers. If you no longer want to receive these notifications, you can unsubscribe. For more information, see [Delete an SNS topic and subscription](https://docs.aws.amazon.com/sns/latest/dg/sns-delete-subscription-topic.html). ------ #### [ Console ] **To subscribe to notifications** 1. Open the Amazon SNS console at [https://console.aws.amazon.com/sns/v3/home](https://console.aws.amazon.com/sns/v3/home). 1. In the navigation bar, change the Region to **US East (N. Virginia)**, if necessary. You must select this Region because the SNS notifications that you are subscribing to are in this Region. 1. In the navigation pane, choose **Subscriptions**. 1. Choose **Create subscription**. 1. In the **Create subscription** dialog box, do the following: 1. For **TopicARN**, copy the following Amazon Resource Name (ARN): arn:aws:sns:us-east-1:801119661308:ec2-windows-drivers 1. For **Protocol**, choose `Email`. 1. For **Endpoint**, type an email address that you can use to receive the notifications. 1. Choose **Create subscription**. 1. You'll receive a confirmation email. Open the email and follow the directions to complete your subscription. ------ #### [ AWS CLI ] **To subscribe to notifications** Use the following command. ``` aws sns subscribe \ --topic-arn arn:aws:sns:us-east-1:801119661308:ec2-windows-drivers \ --region us-east-1 \ --protocol email \ --notification-endpoint YourUserName@YourDomainName.ext ``` ------ #### [ PowerShell ] **To subscribe to notifications** Use the following command. ``` Connect-SNSNotification ` -TopicArn 'arn:aws:sns:us-east-1:801119661308:ec2-windows-drivers' ` -Region us-east-1 ` -Protocol email ` -Endpoint "YourUserName@YourDomainName.ext" ``` ------ # Upgrade PV drivers on EC2 Windows instances Upgrade PV drivers We recommend that you install the latest PV drivers to improve the stability and performance of your EC2 Windows instances. The directions on this page help you download the driver package and run the install program. **To verify which driver your Windows instance uses** Open **Device Manager** and view **Network Adapters**. Check whether the PV driver is one of the following: + AWS PV Network Device + Citrix PV Ethernet Adapter + Red Hat PV NIC Driver **System requirements** Be sure to check the `readme.txt` file in the download for system requirements. **Topics** + [ ## Upgrade Windows Server instances (AWS PV upgrade) with Distributor ](#aws-pv-upgrade-distributor) + [ ## Upgrade Windows Server instances (AWS PV upgrade) manually ](#aws-pv-upgrade) + [ ## Upgrade a domain controller (AWS PV upgrade) ](#aws-pv-upgrade-dc) + [ ## Upgrade Windows Server 2008 and 2008 R2 instances (Red Hat to Citrix PV upgrade) ](#win2008-citrix-upgrade) + [ ## Upgrade your Citrix Xen guest agent service ](#citrix-pv-guest-agent-upgrade) ## Upgrade Windows Server instances (AWS PV upgrade) with Distributor You can use Distributor, a capability of AWS Systems Manager, to install or upgrade the AWS PV driver package. The installation or upgrade can be performed one time, or you can install or update it on a schedule. The `In-place update` option for **Installation Type** isn't supported for this Distributor package. **Important** If your instance is a domain controller, see [Upgrade a domain controller (AWS PV upgrade)](#aws-pv-upgrade-dc). The upgrade process for domain controller instances is different than standard editions of Windows. 1. We recommend that you create a backup in case you need to roll back your changes. **Tip** Instead of creating the AMI from the Amazon EC2 console, you can use Systems Manager Automation to create the AMI using the `AWS-CreateImage` runbook. For more information, see [https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-aws-createimage.html](https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-aws-createimage.html) in the *AWS Systems Manager Automation runbook reference User Guide*. 1. When you stop an instance, the data on any instance store volumes is erased. Before you stop an instance, verify that you've copied any data that you need from your instance store volumes to persistent storage, such as Amazon EBS or Amazon S3. 1. In the navigation pane, choose **Instances**. 1. Select the instance that requires the driver upgrade, and choose **Instance state**, **Stop instance**. 1. After the instance is stopped, select the instance, choose **Actions**, then **Image and templates**, and then choose **Create image**. 1. Choose **Instance state**, **Start instance**. 1. Connect to the instance using Remote Desktop. For more information, see [Connect to your Windows instance using an RDP client](connect-rdp.md). 1. We recommend that you take all non-system disks offline and note any drive letter mappings to the secondary disks in Disk Management before you perform this upgrade. This step is not required if you are performing an in-place update of AWS PV drivers. We also recommend setting non-essential services to **Manual** start-up in the Services console. 1. For the instructions for how to install or upgrade the AWS PV driver package using Distributor, see the procedures in [Install or update packages](https://docs.aws.amazon.com/systems-manager/latest/userguide/distributor-working-with-packages-deploy.html) in the *AWS Systems Manager User Guide*. 1. For **Name**, choose **AWSPVDriver**. 1. For **Installation type**, select **Uninstall and reinstall**. 1. Configure the other parameters for the package as necessary and run installation or upgrade using the referenced procedure in [Step 4](#distributor-procedure-awspv). After running the Distributor package, the instance automatically reboots and then upgrades the driver. The instance will not be available for up to 15 minutes. 1. After the upgrade is complete, and the instance passes both health checks in the Amazon EC2 console, verify that the new driver was installed by connecting to the instance using Remote Desktop. 1. After you are connected, run the following PowerShell command: ``` Get-ItemProperty HKLM:\SOFTWARE\Amazon\PVDriver ``` 1. Verify that the driver version is the same as the latest version listed in the Driver Version History table. For more information, see [AWS PV driver package history](xen-drivers-overview.md#pv-driver-history) Open Disk Management to review any offline secondary volumes and bring them online corresponding to the drive letters noted in [Step 3](#secondary-disks-step-distributor). If you previously disabled [TCP offloading](pvdrivers-troubleshooting.md#citrix-tcp-offloading) using Netsh for Citrix PV drivers we recommend that you re-enable this feature after upgrading to AWS PV drivers. TCP Offloading issues with Citrix drivers are not present in the AWS PV drivers. As a result, TCP Offloading provides better performance with AWS PV drivers. If you previously applied a static IP address or DNS configuration to the network interface, you might need to reapply the static IP address or DNS configuration after upgrading AWS PV drivers. ## Upgrade Windows Server instances (AWS PV upgrade) manually Use the following procedure to perform an in-place upgrade of AWS PV drivers, or to upgrade from Citrix PV drivers to AWS PV drivers on Windows Server 2008 R2, Windows Server 2012, Windows Server 2012 R2, Windows Server 2016, Windows Server 2019, or Windows Server 2022. This upgrade is not available for Red Hat drivers, or for other versions of Windows Server. Some older versions of Windows Server can't use the latest drivers. To verify which driver version to use for your operating system, see the driver version table in the [Paravirtual drivers for Windows instances](xen-drivers-overview.md) page. **Important** If your instance is a domain controller, see [Upgrade a domain controller (AWS PV upgrade)](#aws-pv-upgrade-dc). The upgrade process for domain controller instances is different than standard editions of Windows. **To upgrade AWS PV drivers manually** 1. We recommend that you create a backup in case you need to roll back your changes. **Tip** Instead of creating the AMI from the Amazon EC2 console, you can use Systems Manager Automation to create the AMI using the `AWS-CreateImage` runbook. For more information, see [https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-aws-createimage.html](https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-aws-createimage.html) in the *AWS Systems Manager Automation runbook reference User Guide*. 1. When you stop an instance, the data on any instance store volumes is erased. Before you stop an instance, verify that you've copied any data that you need from your instance store volumes to persistent storage, such as Amazon EBS or Amazon S3. 1. In the navigation pane, choose **Instances**. 1. Select the instance that requires the driver upgrade, and choose **Instance state**, **Stop instance**. 1. After the instance is stopped, select the instance, choose **Actions**, then **Image and templates**, and then choose **Create image**. 1. Choose **Instance state**, **Start instance**. 1. Connect to the instance using Remote Desktop. 1. We recommend that you take all non-system disks offline and note any drive letter mappings to the secondary disks in Disk Management before you perform this upgrade. This step is not required if you are performing an in-place update of AWS PV drivers. We also recommend setting non-essential services to **Manual** start-up in the Services console. 1. Download the drivers to your instance using one of the following options: + **Browser** – [Download](https://s3.amazonaws.com/ec2-windows-drivers-downloads/AWSPV/Latest/AWSPVDriver.zip) the latest driver package to the instance and extract the zip archive. + **PowerShell** – Run the following commands: ``` Invoke-WebRequest https://s3.amazonaws.com/ec2-windows-drivers-downloads/AWSPV/Latest/AWSPVDriver.zip -outfile $env:USERPROFILE\pv_driver.zip Expand-Archive $env:userprofile\pv_driver.zip -DestinationPath $env:userprofile\pv_drivers ``` If you receive an error when downloading the file, and you are using Windows Server 2016 or earlier, TLS 1.2 might need to be enabled for your PowerShell terminal. You can enable TLS 1.2 for the current PowerShell session with the following command and then try again: ``` [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ``` 1. Run `AWSPVDriverSetup.msi`. After running the MSI, the instance automatically reboots and then upgrades the driver. The instance will not be available for up to 15 minutes. After the upgrade is complete and the instance passes both health checks in the Amazon EC2 console, you can verify that the new driver was installed by connecting to the instance using Remote Desktop and then running the following PowerShell command: ``` Get-ItemProperty HKLM:\SOFTWARE\Amazon\PVDriver ``` Verify that the driver version is the same as the latest version listed in the Driver Version History table. For more information, see [AWS PV driver package history](xen-drivers-overview.md#pv-driver-history) Open Disk Management to review any offline secondary volumes and bring them online corresponding to the drive letters noted in [Step 3](#secondary-disks-step-manual). If you previously disabled [TCP offloading](pvdrivers-troubleshooting.md#citrix-tcp-offloading) using Netsh for Citrix PV drivers we recommend that you re-enable this feature after upgrading to AWS PV drivers. TCP Offloading issues with Citrix drivers are not present in the AWS PV drivers. As a result, TCP Offloading provides better performance with AWS PV drivers. If you previously applied a static IP address or DNS configuration to the network interface, you might need to reapply the static IP address or DNS configuration after upgrading AWS PV drivers. ## Upgrade a domain controller (AWS PV upgrade) Use the following procedure on a domain controller to perform either an in-place upgrade of AWS PV drivers, or to upgrade from Citrix PV drivers to AWS PV drivers. To ensure that your FSMO roles remain operational during the upgrade, we recommend that you transfer those roles to other domain controllers before you start the upgrade. For more information, see [How to view and transfer FSMO roles](https://learn.microsoft.com/en-us/troubleshoot/windows-server/active-directory/view-transfer-fsmo-roles) on the *Microsoft Learn* website. **To upgrade a domain controller** 1. We recommend that you create a backup of your domain controller in case you need to roll back your changes. Using an AMI as a backup is not supported. For more information, see [Backup and restore considerations](https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controllers-hyper-v#backup-and-restore-considerations) in the Microsoft documentation. 1. Run the following command to configure Windows to boot into Directory Services Restore Mode (DSRM). **Warning** Before running this command, confirm that you know the DSRM password. You'll need this information so that you can log in to your instance after the upgrade is complete and the instance automatically reboots. ``` bcdedit /set {default} safeboot dsrepair ``` PowerShell: ``` PS C:\> bcdedit /set "{default}" safeboot dsrepair ``` The system must boot into DSRM because the upgrade utility removes Citrix PV storage drivers so it can install AWS PV drivers. Therefore we recommend noting any drive letter and folder mappings to the secondary disks in Disk Management. When Citrix PV storage drivers are not present, secondary drives are not detected. Domain controllers that use an NTDS folder on secondary drives will not boot because the secondary disk is not detected. **Warning** After you run this command do not manually reboot the system. The system will be unreachable because Citrix PV drivers do not support DSRM. 1. Run the following command to add **DisableDCCheck** to the registry: ``` reg add HKLM\SOFTWARE\Wow6432Node\Amazon\AWSPVDriverSetup /v DisableDCCheck /t REG_SZ /d true ``` 1. [Download](https://s3.amazonaws.com/ec2-windows-drivers-downloads/AWSPV/Latest/AWSPVDriver.zip) the latest driver package to the instance and extract the zip archive. 1. Run `AWSPVDriverSetup.msi`. After running the MSI, the instance automatically reboots and then upgrades the driver. The instance will not be available for up to 15 minutes. 1. After the upgrade is complete and the instance passes both health checks in the Amazon EC2 console, connect to the instance using Remote Desktop. Open Disk Management to review any offline secondary volumes and bring them online corresponding to the drive letters and folder mappings noted earlier. You must connect to the instance by specifying the username in the following format *hostname*\$1administrator. For example, Win2k12TestBox\$1administrator. 1. Run the following command to remove the DSRM boot configuration: ``` bcdedit /deletevalue safeboot ``` 1. Reboot the instance. 1. To complete the upgrade process, verify that the new driver was installed. In Device Manager, under **Storage Controllers**, locate **AWS PV Storage Host Adapter**. Verify that the driver version is the same as the latest version listed in the Driver Version History table. For more information, see [AWS PV driver package history](xen-drivers-overview.md#pv-driver-history). 1. Run the following command to delete **DisableDCCheck** from the registry: ``` reg delete HKLM\SOFTWARE\Wow6432Node\Amazon\AWSPVDriverSetup /v DisableDCCheck ``` **Note** If you previously disabled [TCP offloading](pvdrivers-troubleshooting.md#citrix-tcp-offloading) using Netsh for Citrix PV drivers we recommend that you re-enable this feature after upgrading to AWS PV Drivers. TCP Offloading issues with Citrix drivers are not present in the AWS PV drivers. As a result, TCP Offloading provides better performance with AWS PV drivers. ## Upgrade Windows Server 2008 and 2008 R2 instances (Red Hat to Citrix PV upgrade) Upgrade Server 2008, 2008 R2 (Red Hat to Citrix PV upgrade) Before you start upgrading your Red Hat drivers to Citrix PV drivers, make sure you do the following: + Install the latest version of the EC2Config service. For more information, see [Install the latest version of EC2Config](UsingConfig_Install.md). + Verify that you have Windows PowerShell 3.0 installed. To verify the version that you have installed, run the following command in a PowerShell window: ``` PS C:\> $PSVersionTable.PSVersion ``` Windows PowerShell 3.0 is bundled in the Windows Management Framework (WMF) version 3.0 install package. If you need to install Windows PowerShell 3.0, see [Windows Management Framework 3.0](https://www.microsoft.com/en-us/download/details.aspx?id=34595) in the Microsoft Download Center. + Back up your important information on the instance, or create an AMI from the instance. For more information about creating an AMI, see [Create an Amazon EBS-backed AMI](creating-an-ami-ebs.md). **Tip** Instead of creating the AMI from the Amazon EC2 console, you can use Systems Manager Automation to create the AMI using the `AWS-CreateImage` runbook. For more information, see [https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-aws-createimage.html](https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-aws-createimage.html) in the *AWS Systems Manager Automation runbook reference User Guide*. If you create an AMI, make sure that you do the following: + Write down your password. + Do not run the Sysprep tool manually or using the EC2Config service. + Set your Ethernet adapter to obtain an IP address automatically using DHCP. **To upgrade Red Hat drivers** 1. Connect to your instance and log in as the local administrator. For more information about connecting to your instance, see [Connect to your Windows instance using RDP](connecting_to_windows_instance.md). 1. In your instance, [download](https://s3.amazonaws.com/ec2-downloads-windows/Drivers/Citrix-Win_PV.zip) the Citrix PV upgrade package. 1. Extract the contents of the upgrade package to a location of your choice. 1. Double-click the **Upgrade.bat** file. If you get a security warning, choose **Run**. 1. In the **Upgrade Drivers** dialog box, review the information and choose **Yes** if you are ready to start the upgrade. 1. In the **Red Hat Paravirtualized Xen Drivers for Windows uninstaller** dialog box, choose **Yes** to remove the Red Hat software. Your instance will be rebooted. **Note** If you do not see the uninstaller dialog box, choose **Red Hat Paravirtualize** in the Windows taskbar. ![\[Red Hat Paravirtualized in taskbar.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/win2003-citrix-taskbar.png) 1. Check that the instance has rebooted and is ready to be used. 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. On the **Instances** page, select **Actions**, then **Monitor and troubleshoot**, and then choose **Get system log**. 1. The upgrade operations should have restarted the server 3 or 4 times. You can see this in the log file by the number of times `Windows is Ready to use` is displayed. ![\[Windows system log.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/win2008-sys-log.png) 1. Connect to your instance and log in as the local administrator. 1. Close the **Red Hat Paravirtualized Xen Drivers for Windows uninstaller** dialog box. 1. Confirm that the installation is complete. Navigate to the `Citrix-WIN_PV` folder that you extracted earlier, open the `PVUpgrade.log` file, and then check for the text `INSTALLATION IS COMPLETE`. ![\[PVUpgrade log file.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/win2008-pvupgrade-log.png) ## Upgrade your Citrix Xen guest agent service If you are using Citrix PV drivers on Windows Server, you can upgrade the Citrix Xen guest agent service. This Windows service handles tasks such as shutdown and restart events from the API. You can run this upgrade package on any version of Windows Server, as long as the instance is running Citrix PV drivers. **Important** For Windows Server 2008 R2 and later, we recommend you upgrade to AWS PV drivers that include the Guest Agent update. Before you start upgrading your drivers, make sure you back up your important information on the instance, or create an AMI from the instance. For more information about creating an AMI, see [Create an Amazon EBS-backed AMI](creating-an-ami-ebs.md). **Tip** Instead of creating the AMI from the Amazon EC2 console, you can use Systems Manager Automation to create the AMI using the `AWS-CreateImage` runbook. For more information, see [https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-aws-createimage.html](https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-aws-createimage.html) in the *AWS Systems Manager Automation runbook reference User Guide*. If you create an AMI, make sure you do the following: + Do not enable the Sysprep tool in the EC2Config service. + Write down your password. + Set your Ethernet adapter to DHCP. **To upgrade your Citrix Xen guest agent service** 1. Connect to your instance and log in as the local administrator. For more information about connecting to your instance, see [Connect to your Windows instance using RDP](connecting_to_windows_instance.md). 1. On your instance, [download](https://s3.amazonaws.com/ec2-downloads-windows/Drivers/Citrix-Win_PV.zip) the Citrix upgrade package. 1. Extract the contents of the upgrade package to a location of your choice. 1. Double-click the **Upgrade.bat** file. If you get a security warning, choose **Run**. 1. In the **Upgrade Drivers** dialog box, review the information and choose **Yes** if you are ready to start the upgrade. 1. When the upgrade is complete, the `PVUpgrade.log` file will open and contain the text `UPGRADE IS COMPLETE`. 1. Reboot your instance. # Troubleshoot PV drivers on Windows instances Troubleshoot PV drivers The following are solutions to issues that you might encounter with older Amazon EC2 images and PV drivers. **Topics** + [ ## Windows Server 2012 R2 loses network and storage connectivity after an instance reboot ](#server2012R2-instance-unavailable) + [ ## TCP offloading ](#citrix-tcp-offloading) + [ ## Time synchronization ](#citrix-time-sync) + [ ## Workloads that leverage more than 20,000 disk IOPS experience degradation due to CPU bottlenecks ](#pvdriver-troubleshooting-cpu-bottlenecks) ## Windows Server 2012 R2 loses network and storage connectivity after an instance reboot **Important** This issue occurs only with AMIs made available before September 2014. Windows Server 2012 R2 Amazon Machine Images (AMIs) made available before September 10, 2014 can lose network and storage connectivity after an instance reboot. The error in the AWS Management Console system log states: “Difficulty detecting PV driver details for Console Output.” The connectivity loss is caused by the Plug and Play Cleanup feature. This features scans for and disables inactive system devices every 30 days. The feature incorrectly identifies the EC2 network device as inactive and removes it from the system. When this happens, the instance loses network connectivity after a reboot. For systems that you suspect could be affected by this issue, you can download and run an in-place driver upgrade. If you are unable to perform the in-place driver upgrade, you can run a helper script. The script determines if your instance is affected. If it is affected, and the Amazon EC2 network device has not been removed, the script disables the Plug and Play Cleanup scan. If the network device was removed, the script repairs the device, disables the Plug and Play Cleanup scan, and enables your instance to reboot with network connectivity enabled. **Topics** + [ ### Choose how to fix problems ](#choose-fix) + [ ### Method 1 - Enhanced networking ](#plug-n-play-fix-method1) + [ ### Method 2 - Registry configuration ](#plug-n-play-fix-method2) + [ ### Run the remediation script ](#plug-n-play-script) ### Choose how to fix problems There are two methods for restoring network and storage connectivity to an instance affected by this issue. Choose one of the following methods: | Method | Prerequisites | Procedure Overview | | --- | --- | --- | | Method 1 - Enhanced networking | Enhanced networking is only available in a virtual private cloud (VPC) which requires a C3 instance type. If the server does not currently use the C3 instance type, then you must temporarily change it. | You change the server instance type to a C3 instance. Enhanced networking then enables you to connect to the affected instance and fix the problem. After you fix the problem, you change the instance back to the original instance type. This method is typically faster than Method 2 and less likely to result in user error. You will incur additional charges as long as the C3 instance is running. | | Method 2 - Registry configuration | Ability to create or access a second server. Ability to change Registry settings. | You detach the root volume from the affected instance, attach it to a different instance, connect, and make changes in the Registry. You will incur additional charges as long as the additional server is running. This method is slower than Method 1, but this method has worked in situations where Method 1 failed to resolve the problem. | ### Method 1 - Enhanced networking 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Locate the affected instance. Select the instance and choose **Instance state**, and then choose **Stop instance**. **Warning** When you stop an instance, the data on instance store volumes is lost. To preserve this data, back it up to persistent storage. 1. After the instance is stopped, create a backup. Select the instance and choose **Actions**, then **Image and templates**, and then choose **Create image**. 1. [Change](ec2-instance-resize.md) the instance type to any C3 instance type. 1. [Start](Stop_Start.md) the instance. 1. Connect to the instance using Remote Desktop and then [download](https://s3.amazonaws.com/ec2-windows-drivers-downloads/AWSPV/Latest/AWSPVDriver.zip) the AWS PV Drivers Upgrade package to the instance. 1. Extract the contents of the folder and run `AWSPVDriverSetup.msi`. After running the MSI, the instance automatically reboots and then upgrades the drivers. The instance will not be available for up to 15 minutes. 1. After the upgrade is complete and the instance passes both health checks in the Amazon EC2 console, connect to the instance using Remote Desktop and verify that the new drivers were installed. In Device Manager, under **Storage Controllers**, locate **AWS PV Storage Host Adapter**. Verify that the driver version is the same as the latest version listed in the Driver Version History table. For more information, see [AWS PV driver package history](xen-drivers-overview.md#pv-driver-history). 1. Stop the instance and change the instance back to its original instance type. 1. Start the instance and resume normal use. ### Method 2 - Registry configuration 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Locate the affected instance. Select the instance, choose **Instance state**, and then choose **Stop instance**. **Warning** When you stop an instance, the data on instance store volumes is lost. To preserve this data, back it up to persistent storage. 1. Choose **Launch instances** and create a temporary Windows Server 2008 or Windows Server 2012 instance in the same Availability Zone as the affected instance. Do not create a Windows Server 2012 R2 instance. **Important** If you do not create the instance in the same Availability Zone as the affected instance you will not be able to attach the root volume of the affected instance to the new instance. 1. In the navigation pane, choose **Volumes**. 1. Locate the root volume of the affected instance. Detach the volume and then attach the volume to the temporary instance that you created earlier. Attach it with the default device name (xvdf). 1. Use Remote Desktop to connect to the temporary instance, and then use the Disk Management utility to make the volume available for use. 1. On the temporary instance, open the **Run** dialog box, type **regedit**, and press Enter. 1. In the Registry Editor navigation pane, choose **HKEY\$1Local\$1Machine**, and then from the **File** menu choose **Load Hive**. 1. In the **Load Hive** dialog box, navigate to *Affected Volume*\$1Windows\$1System32\$1config\$1System and type a temporary name in the **Key Name** dialog box. For example, enter OldSys. 1. In the navigation pane of the Registry Editor, locate the following keys: **HKEY\$1LOCAL\$1MACHINE\$1*your\$1temporary\$1key\$1name*\$1ControlSet001\$1Control\$1Class\$14d36e97d-e325-11ce-bfc1-08002be10318** **HKEY\$1LOCAL\$1MACHINE\$1*your\$1temporary\$1key\$1name*\$1ControlSet001\$1Control\$1Class\$14d36e96a-e325-11ce-bfc1-08002be10318** 1. For each key, double-click **UpperFilters**, enter a value of XENFILT, and then choose **OK**. ![\[Registry key for affected volume.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/troubleshooting-server2012R2-regedit.png) 1. Locate the following key: **HKEY\$1LOCAL\$1MACHINE\$1*your\$1temporary\$1key\$1name*\$1ControlSet001\$1Services\$1XENBUS\$1Parameters** 1. Create a new string (REG\$1SZ) with the name ActiveDevice and the following value: **PCI\$1VEN\$15853&DEV\$10001&SUBSYS\$100015853&REV\$101** 1. Locate the following key: **HKEY\$1LOCAL\$1MACHINE\$1*your\$1temporary\$1key\$1name*\$1ControlSet001\$1Services\$1XENBUS** 1. Change the **Count** from 0 to 1. 1. Locate and delete the following keys: **HKEY\$1LOCAL\$1MACHINE\$1*your\$1temporary\$1key\$1name*\$1ControlSet001\$1Services\$1xenvbd\$1StartOverride** **HKEY\$1LOCAL\$1MACHINE \$1*your\$1temporary\$1key\$1name*\$1ControlSet001\$1Services\$1xenfilt\$1StartOverride** 1. In the Registry Editor navigation pane, choose the temporary key that you created when you first opened the Registry Editor. 1. From the **File** menu, choose **Unload Hive**. 1. In the Disk Management Utility, choose the drive you attached earlier, open the context (right-click) menu, and choose **Offline**. 1. In the Amazon EC2 console, detach the affected volume from the temporary instance and reattach it to your Windows Server 2012 R2 instance with the device name /dev/sda1. You must specify this device name to designate the volume as a root volume. 1. [Start](Stop_Start.md) the instance. 1. Connect to the instance using Remote Desktop and then [download](https://s3.amazonaws.com/ec2-windows-drivers-downloads/AWSPV/Latest/AWSPVDriver.zip) the AWS PV Drivers Upgrade package to the instance. 1. Extract the contents of the folder and run `AWSPVDriverSetup.msi`. After running the MSI, the instance automatically reboots and then upgrades the drivers. The instance will not be available for up to 15 minutes. 1. After the upgrade is complete and the instance passes both health checks in the Amazon EC2 console, connect to the instance using Remote Desktop and verify that the new drivers were installed. In Device Manager, under **Storage Controllers**, locate **AWS PV Storage Host Adapter**. Verify that the driver version is the same as the latest version listed in the Driver Version History table. For more information, see [AWS PV driver package history](xen-drivers-overview.md#pv-driver-history). 1. Delete or stop the temporary instance you created in this procedure. ### Run the remediation script If you are unable to perform an in-place driver upgrade or migrate to a newer instance you can run the remediation script to fix the problems caused by the Plug and Play Cleanup task. **To run the remediation script** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Instances**. 1. Select the instance for which you want to run the remediation script. Choose **Instance state**, and then choose **Stop instance**. **Warning** When you stop an instance, the data on instance store volumes is lost. To preserve this data, back it up to persistent storage. 1. After the instance is stopped, create a backup. Select the instance, choose **Actions**, then **Image and templates**, and then choose **Create image**. 1. Choose **Instance state**, and then choose **Start instance**. 1. Connect to the instance by using Remote Desktop and then [download](https://s3.amazonaws.com/ec2-downloads-windows/Scripts/RemediateDriverIssue.zip) the RemediateDriverIssue.zip folder to the instance. 1. Extract the contents of the folder. 1. Run the remediation script according to the instructions in the Readme.txt file. The file is located in the folder where you extracted RemediateDriverIssue.zip. ## TCP offloading **Important** This issue does not apply to instances running AWS PV or Intel network drivers. By default, TCP offloading is enabled for the Citrix PV drivers in Windows AMIs. If you encounter transport-level errors or packet transmission errors (as visible on the Windows Performance Monitor)—for example, when you're running certain SQL workloads—you may need to disable this feature. **Warning** Disabling TCP offloading may reduce the network performance of your instance. **To disable TCP offloading for Windows Server 2012 and 2008** 1. Connect to your instance and log in as the local administrator. 1. If you're using Windows Server 2012, press **Ctrl\$1Esc** to access the **Start** screen, and then choose **Control Panel**. If you're using Windows Server 2008, choose **Start** and select **Control Panel**. 1. Choose **Network and Internet**, then **Network and Sharing Center**. 1. Choose **Change adapter settings**. 1. Right-click **Citrix PV Ethernet Adapter \$10** and select **Properties**. ![\[Local area connection properties.\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/citrix-local-area-conn.png) 1. In the **Local Area Connection Properties** dialog box, choose **Configure** to open the **Citrix PV Ethernet Adapter \$10 Properties** dialog box. 1. On the **Advanced** tab, disable each of the properties, except for **Correct TCP/UDP Checksum Value**. To disable a property, select it from **Property** and choose **Disabled** from **Value**. 1. Choose **OK**. 1. Run the following commands from a Command Prompt window. ``` netsh int ip set global taskoffload=disabled netsh int tcp set global chimney=disabled netsh int tcp set global rss=disabled netsh int tcp set global netdma=disabled ``` 1. Reboot the instance. ## Time synchronization Prior to the release of the 2013.02.13 Windows AMI, the Citrix Xen guest agent could set the system time incorrectly. This can cause your DHCP lease to expire. If you have issues connecting to your instance, you might need to update the agent. To determine whether you have the updated Citrix Xen guest agent, check whether the `C:\Program Files\Citrix\XenGuestAgent.exe` file is from March 2013. If the date on this file is earlier than that, update the Citrix Xen guest agent service. For more information, see [Upgrade your Citrix Xen guest agent service](Upgrading_PV_drivers.md#citrix-pv-guest-agent-upgrade). ## Workloads that leverage more than 20,000 disk IOPS experience degradation due to CPU bottlenecks You can be affected by this issue if you are using Windows instances running AWS PV drivers that leverage more than 20,000 IOPS, and you experience bug check code `0x9E: USER_MODE_HEALTH_MONITOR`. Disk reads and writes (IOs) in the AWS PV drivers occur in two phases: **IO preparation** and **IO completion**. By default, the preparation phase runs on a single arbitrary core. The completion phase runs on core `0`. The amount of computation required to process an IO varies based on it size and other properties. Some IOs use more computation in the preparation phase, and others in the completion phase. When an instance drives more than 20,000 IOPS, the preparation or completion phase may result in a bottleneck, where the CPU upon which it runs is at 100% capacity. Whether or not the preparation or completion phase becomes a bottleneck depends on the properties of the IOs used by the application. Starting with AWS PV drivers 8.4.0, the load of the preparation phase and the completion phase can be distributed across multiple cores, eliminating bottlenecks. Each application uses different IO properties. Therefore, applying one of the following configurations may raise, lower, or not impact the performance of your application. After you apply any of these configurations, monitor the application to verify that it is meeting your desired performance. 1. **Prerequisites** Before you begin this troubleshooting procedure, verify the following prerequisites: + Your instance uses AWS PV drivers version 8.4.0 or later. To upgrade, see [Upgrade PV drivers on EC2 Windows instances](Upgrading_PV_drivers.md). + You have RDP access to the instance. For steps to connect to your Windows instance using RDP, see [Connect to your Windows instance using an RDP client](connect-rdp.md). + You have administrator access on the instance. 1. **Observe CPU load on your instance** You can use Windows Task Manager to view the load on each CPU to determine potential bottlenecks to disk IO. 1. Verify that your application is running and handling traffic similar to your production workload. 1. Connect to your instance using RDP. 1. Choose the **Start** menu on your instance. 1. Enter `Task Manager` in the **Start** menu to open Task Manager. 1. If Task Manager displays the Summary View, choose **More details** to expand the detailed view. 1. Choose the **Performance** tab. 1. Select **CPU** in the left pane. 1. Right-click on the graph in the main pane and select **Change graph to**>**Logical processors** to display each individual core. 1. Depending on how many cores are on your instance, you may see lines displaying CPU load over time, or you may just see a number. + If you see graphs displaying load over time, look for CPUs where the box is almost entirely shaded. + If you see a number on each core, look for cores that consistently show 95% or greater. 1. Note whether core `0` or a different core is experiencing a heavy load. 1. **Choose which configuration to apply** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/pvdrivers-troubleshooting.html) **Note** We recommend that you do not distribute IO preparation without also distributing IO completion (setting `DpcRedirection` without setting `NotifierDistributed`) because the completion phase is sensitive to overload by the preparation phase when the preparation phase is running in parallel. **Registry key values** + *NotifierDistributed* Value `0` or not present — The completion phase will run on core `0`. Value `1` — The driver chooses to run the completion phase or core `0` or one additional core per attached disk. Value `2` — The driver runs the completion phase on one additional core per attached disk. + *DpcRedirection* Value `0` or not present — The preparation phase will run on a single, arbitrary core. Value `1` — The preparation phase is distributed across multiple cores. **Default configuration** Apply the default configuration with AWS PV driver versions prior to 8.4.0, or if performance or stability degradation is observed after applying one of the other configurations in this section. 1. Connect to your instance using RDP. 1. Open a new PowerShell command prompt as an administrator. 1. Run the following commands to remove the `NotifierDistributed` and `DpcRedirection` registry keys. ``` Remove-ItemProperty -Path HKLM:\System\CurrentControlSet\Services\xenvbd\Parameters -Name NotifierDistributed ``` ``` Remove-ItemProperty -Path HKLM:\System\CurrentControlSet\Services\xenvbd\Parameters -Name DpcRedirection ``` 1. Reboot your instance. **Allow driver to choose whether to distribute completion** Set `NotiferDistributed` registry key to allow the PV storage driver to choose whether or not to distribute IO completion. 1. Connect to your instance using RDP. 1. Open a new PowerShell command prompt as an administrator. 1. Run the following command to set the `NotiferDistributed` registry key. ``` Set-ItemProperty -Type DWORD -Path HKLM:\System\CurrentControlSet\Services\xenvbd\Parameters -Value 0x00000001 -Name NotifierDistributed ``` 1. Reboot your instance. **Distribute both preparation and completion** Set `NotifierDistributed` and `DpcRedirection` registry keys to always distribute both the preparation and completion phases. 1. Connect to your instance using RDP. 1. Open a new PowerShell command prompt as an administrator. 1. Run the following commands to set the `NotifierDistributed` and `DpcRedirection` registry keys. ``` Set-ItemProperty -Type DWORD -Path HKLM:\System\CurrentControlSet\Services\xenvbd\Parameters -Value 0x00000002 -Name NotifierDistributed ``` ``` Set-ItemProperty -Type DWORD -Path HKLM:\System\CurrentControlSet\Services\xenvbd\Parameters -Value 0x00000001 -Name DpcRedirection ``` 1. Reboot your instance.