# Troubleshooting problems with your Storage Gateway deployment Troubleshooting Following, you can find information about best practices and troubleshooting issues related to gateways, host platforms, file shares, high availability, data recovery, and security. The on-premises gateway troubleshooting information covers gateways deployed on supported virtualization platforms. The troubleshooting information for high availability issues covers gateways running on VMware vSphere High Availability (HA) platform. **Topics** + [Troubleshooting: gateway offline issues](troubleshooting-gateway-offline.md) - Learn how to diagnose problems that can cause your gateway to appear offline in the Storage Gateway console. + [Troubleshooting: Active Directory issues](troubleshooting-active-directory.md) - Learn what to do if you receive error messages such as `NETWORK_ERROR`, `TIMEOUT`, or `ACCESS_DENIED` when trying to join your File Gateway to a Microsoft Active Directory domain. + [Troubleshooting: gateway activation issues](troubleshooting-gateway-activation.md) - Learn what to do if you receive an internal error message when attempting to activate your Storage Gateway. + [Troubleshooting: on-premises gateway issues](troubleshooting-on-premises-gateway-issues.md) - Learn about typical issues that you might encounter working with your on-premises gateways, and how to allow Support to connect to your gateway to assist with troubleshooting. + [Troubleshooting: Microsoft Hyper-V setup issues](troubleshooting-hyperv-setup.md) - Learn about typical issues that you might encounter when deploying Storage Gateway on the Microsoft Hyper-V platform. + [Troubleshooting: Amazon EC2 gateway issues](troubleshooting-EC2-gateway-issues.md) - Find information about typical issues that you might encounter when working with gateways deployed on Amazon EC2. + [Troubleshooting: hardware appliance issues](troubleshooting-hardware-appliance-issues.md) - Learn how to resolve issues that you might encounter with the AWS Storage Gateway Hardware Appliance. + [Troubleshooting: File Gateway issues](troubleshooting-file-gateway-issues.md) - Find information that can help you understand the cause of errors and health notifications that appear in your File Gateway's CloudWatch logs. + [Troubleshooting: file share issues](troubleshooting-file-share-issues.md) - Learn about actions you can take if you experience unexpected issues with your file share. + [Troubleshooting: high availability issues](troubleshooting-ha-issues.md) - Learn what to do if you experience issues with gateways that are deployed in a VMware HA environment. # Troubleshooting: gateway offline in the Storage Gateway console Troubleshooting: gateway offline issues Use the following troubleshooting information to determine what to do if the AWS Storage Gateway console shows that your gateway is offline. Your gateway might be showing as offline for one or more of the following reasons: + The gateway can't reach the Storage Gateway service endpoints. + The gateway shut down unexpectedly. + A cache disk associated with the gateway has been disconnected or modified, or has failed. To bring your gateway back online, identify and resolve the issue that caused your gateway to go offline. ## Check the associated firewall or proxy If you configured your gateway to use a proxy, or you placed your gateway behind a firewall, then review the access rules of the proxy or firewall. The proxy or firewall must allow traffic to and from the network ports and service endpoints required by Storage Gateway. For more information, see [Network and firewall requirements](https://docs.aws.amazon.com/filegateway/latest/files3/Requirements.html#networks). ## Check for an ongoing SSL or deep-packet inspection of your gateway's traffic If an SSL or deep-packet inspection is currently being performed on the network traffic between your gateway and AWS, then your gateway might not be able to communicate with the required service endpoints. To bring your gateway back online, you must disable the inspection. ## Check the IOWaitPercent metric after a reboot or software update After a reboot or software update, check to see if the `IOWaitPercent` metric for your File Gateway is 10 or greater. This might cause your gateway to be slow to respond while it rebuilds the index cache to RAM. For more information, see [Troubleshooting: Using CloudWatch metrics](https://docs.aws.amazon.com/filegateway/latest/files3/troubleshooting-file-gateway-issues.html#gateway-not-responding). ## Check for a power outage or hardware failure on the hypervisor host A power outage or hardware failure on the hypervisor host of your gateway can cause your gateway to shut down unexpectedly and become unreachable. After you restore the power and network connectivity, your gateway will become reachable again. After your gateway is back online, be sure to take steps to recover your data. For more information, see [Best practices: recovering your data](https://docs.aws.amazon.com/filegateway/latest/files3/recover-data-from-gateway.html). ## Check for issues with an associated cache disk Your gateway can go offline if at least one of the cache disks associated with your gateway was removed, changed, or resized, or if it is corrupted. **If a working cache disk was removed from the hypervisor host:** 1. Shut down the gateway. 1. Re-add the disk. **Note** Make sure you add the disk to the same disk node. 1. Restart the gateway. **If a cache disk is corrupted, was replaced, or was resized:** + Follow the **Method 2** procedure described in [Replacing your existing S3 File Gateway with a new instance](https://docs.aws.amazon.com/filegateway/latest/files3/migrate-data.html#replace-instance-file-gateway) to set up a new gateway and re-download cache disk information from the AWS cloud. # Troubleshooting: issues joining gateway to Active Directory Troubleshooting: Active Directory issues Use the following troubleshooting information to determine what to do if you receive error messages such as `NETWORK_ERROR`, `TIMEOUT`, or `ACCESS_DENIED` when trying to join your File Gateway to a Microsoft Active Directory domain. To resolve these errors, perform the following checks and configurations. ## Confirm that the gateway can reach the domain controller by running an nping test **To run an nping test:** 1. Connect to the gateway local console using your hypervisor management software (VMware, Hyper-V, or KVM) for on-premises gateways, or using ssh for Amazon EC2 gateways. 1. Enter the corresponding numeral to select **Gateway Console**, and then enter `h` to list all available commands. To test the connectivity between the Storage Gateway virtual machine and the domain, run the following command: `nping -d corp.domain.com -p 389 -c 1 -t tcp` **Note** Replace `corp.domain.com` with your Active Directory domain DNS name and replace `389` with the LDAP port for your environment. Verify that you have opened the required ports within your firewall. The following is an example of a successful nping test where the gateway was able to reach the domain controller: ``` nping -d corp.domain.com -p 389 -c 1 -t tcp Starting Nping 0.6.40 ( http://nmap.org/nping ) at 2022-06-30 16:24 UTC SENT (0.0553s) TCP 10.10.10.21:9783 > 10.10.10.10:389 S ttl=64 id=730 iplen=40 seq=2597195024 win=1480 RCVD (0.0556s) TCP 10.10.10.10:389 > 10.10.10.21:9783 SA ttl=128 id=22332 iplen=44 seq=4170716243 win=8192 Max rtt: 0.310ms | Min rtt: 0.310ms | Avg rtt: 0.310ms Raw packets sent: 1 (40B) | Rcvd: 1 (44B) | Lost: 0 (0.00%) Nping done: 1 IP address pinged in 1.09 seconds
``` The following is an example of an nping test where there is no connectivity to or response from the `corp.domain.com` destination: ``` nping -d corp.domain.com -p 389 -c 1 -t tcp Starting Nping 0.6.40 ( http://nmap.org/nping ) at 2022-06-30 16:26 UTC SENT (0.0421s) TCP 10.10.10.21:47196 > 10.10.10.10:389 S ttl=64 id=30318 iplen=40 seq=1762671338 win=1480 Max rtt: N/A | Min rtt: N/A | Avg rtt: N/A Raw packets sent: 1 (40B) | Rcvd: 0 (0B) | Lost: 1 (100.00%) Nping done: 1 IP address pinged in 1.07 seconds ``` ## Check the DHCP options set for the VPC of your Amazon EC2 gateway instance If the File Gateway is running on an Amazon EC2 instance, then you must make sure a DHCP options set is properly configured and attached to the Amazon Virtual Private Cloud (VPC) that contains the gateway instance. For more information, see [DHCP option sets in Amazon VPC](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html). ## Confirm that the gateway can resolve the domain by running a dig query If the domain isn't resolvable by the gateway, then the gateway can't join the domain. **To run a dig query:** 1. Connect to the gateway local console using your hypervisor management software (VMware, Hyper-V, or KVM) for on-premises gateways, or using ssh for Amazon EC2 gateways. 1. Enter the corresponding numeral to select **Gateway Console**, and then enter `h` to list all available commands. To test whether the gateway can resolve the domain, run the following command: `dig -d corp.domain.com` **Note** Replace `corp.domain.com` with your Active Directory domain DNS name. The following is an example of a successful response: ``` ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.5.2 <<>> corp.domain.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24817 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ;; QUESTION SECTION: ;corp.domain.com. IN A ;; ANSWER SECTION: corp.domain.com. 600 IN A 10.10.10.10 corp.domain.com. 600 IN A 10.10.20.10 ;; Query time: 0 msec ;; SERVER: 10.10.20.228#53(10.10.20.228) ;; WHEN: Thu Jun 30 16:36:32 UTC 2022 ;; MSG SIZE rcvd: 78 ``` ## Check the domain controller settings and roles Verify that the domain controller isn't set to read-only, and that the domain controller has enough roles for computers to join. To test this, try joining other servers from the same VPC subnet as the gateway VM to the domain. ## Check that the gateway is joined to the nearest domain controller As a best practice, we recommend joining your gateway to a domain controller that is geographically close to the gateway appliance. If the gateway appliance can't communicate with the domain controller within 20 seconds due to network latency, then the domain joining process can time out. For example, the process might time out if the gateway appliance is in the US East (N. Virginia) AWS Region and the domain controller is in the Asia Pacific (Singapore) AWS Region. **Note** To increase the default timeout value of 20 seconds, you can run the [join-domain command](https://docs.aws.amazon.com/cli/latest/reference/storagegateway/join-domain.html) in the AWS Command Line Interface (AWS CLI) and include the `--timeout-in-seconds` option to increase the time. You can also use the [JoinDomain API call](https://amazonaws.com/storagegateway/latest/APIReference/API_JoinDomain.html) and include the `TimeoutInSeconds` parameter to increase the time. The maximum timeout value is 3,600 seconds. If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version. ## Confirm that Active Directory creates new computer objects in the default organizational unit (OU) Make sure Microsoft Active Directory does not have any Group Policy Objects that create new computer objects in any location other than the default OU. Before you can join your gateway to the Active Directory domain, a new computer object must exist in the default OU. Some Active Directory environments are customized to have different OUs for newly created objects. To guarantee that a new computer object for the gateway VM exists in the default OU, try creating the computer object manually on your domain controller before you join the gateway to the domain. You can also run the [join-domain command](https://docs.aws.amazon.com/cli/latest/reference/storagegateway/join-domain.html) using the AWS CLI. Then, specify the option for `--organizational-unit`. **Note** The process of creating the computer object is called pre-staging. ## Check your domain controller event logs If you can't join the gateway to the domain after trying all other checks and configurations described in the previous sections, we recommend examining your domain controller event logs. Check for any errors in the event viewer of the domain controller. Verify that the gateway queries have reached the domain controller. # Troubleshooting: internal error during gateway activation Troubleshooting: gateway activation issues Storage Gateway activation requests traverse two network paths. Incoming activation requests sent by a client connect to the gateway's virtual machine (VM) or Amazon Elastic Compute Cloud (Amazon EC2) instance over port 80. If the gateway successfully receives the activation request, then the gateway communicates with the Storage Gateway endpoints to receive an activation key. If the gateway can't reach the Storage Gateway endpoints, then the gateway responds to the client with an internal error message. Use the following troubleshooting information to determine what to do if you receive an internal error message when attempting to activate your AWS Storage Gateway. **Note** Make sure you deploy new gateways using the latest virtual machine image file or Amazon Machine Image (AMI) version. You will receive an internal error if you attempt to activate a gateway that uses an outdated AMI. Make sure that you select the correct gateway type that you intend to deploy before you download the AMI. The .ova files and AMIs for each gateway type are different, and they are not interchangeable. ## Resolve errors when activating your gateway using a public endpoint To resolve activation errors when activating your gateway using a public endpoint, perform the following checks and configurations. ### Check the required ports For gateways deployed on-premises, check that the ports are open on your local firewall. For gateways deployed on an Amazon EC2 instance, check that the ports are open on the instance's security group. To confirm that the ports are open, run a telnet command on the public endpoint from a server. This server must be in the same subnet as the gateway. For example, the following telnet commands test the connection to port 443: ``` telnet d4kdq0yaxexbo.cloudfront.net 443 telnet storagegateway.region.amazonaws.com 443 telnet dp-1.storagegateway.region.amazonaws.com 443 telnet proxy-app.storagegateway.region.amazonaws.com 443 telnet client-cp.storagegateway.region.amazonaws.com 443 telnet anon-cp.storagegateway.region.amazonaws.com 443 ``` To confirm that the gateway itself can reach the endpoint, access the gateway's local VM console (for gateways deployed on-premises). Or, you can SSH to the gateway's instance (for gateways deployed on Amazon EC2). Then, run a network connectivity test. Confirm that the test returns `[PASSED]`. For more information, see [Testing your gateway's network connectivity](https://docs.aws.amazon.com/filegateway/latest/files3/manage-on-premises-fgw.html#MaintenanceTestGatewayConnectivity-fgw). **Note** The default login user name for the gateway console is `admin`, and the default password is `password`. ### Make sure firewall security does not modify packets sent from the gateway to the public endpoints SSL inspections, deep packet inspections, or other forms of firewall security can interfere with packets sent from the gateway. The SSL handshake fails if the SSL certificate is modified from what the activation endpoint expects. To confirm that there's no SSL inspection in progress, run an OpenSSL command on the main activation endpoint ( `anon-cp.storagegateway.region.amazonaws.com`) on port 443. You must run this command from a machine that's in the same subnet as the gateway: ``` $ openssl s_client -connect anon-cp.storagegateway.region.amazonaws.com:443 -servername anon-cp.storagegateway.region.amazonaws.com ``` **Note** Replace *region* with your AWS Region. If there's no SSL inspection in progress, then the command returns a response similar to the following: ``` $ openssl s_client -connect anon-cp.storagegateway.us-east-2.amazonaws.com:443 -servername anon-cp.storagegateway.us-east-2.amazonaws.com CONNECTED(00000003) depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 CN = anon-cp.storagegateway.us-east-2.amazonaws.com verify return:1 --- Certificate chain 0 s:/CN=anon-cp.storagegateway.us-east-2.amazonaws.com i:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon 1 s:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon i:/C=US/O=Amazon/CN=Amazon Root CA 1 2 s:/C=US/O=Amazon/CN=Amazon Root CA 1 i:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2 3 s:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2 i:/C=US/O=Starfield Technologies, Inc./OU=Starfield Class 2 Certification Authority --- ``` If there is an ongoing SSL inspection, then the response shows an altered certificate chain, similar to the following: ``` $ openssl s_client -connect anon-cp.storagegateway.ap-southeast-1.amazonaws.com:443 -servername anon-cp.storagegateway.ap-southeast-1.amazonaws.com CONNECTED(00000003) depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.ap-southeast-1.amazonaws.com verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.ap-southeast-1.amazonaws.com verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/DC=com/DC=amazonaws/OU=AWS/CN=anon-cp.storagegateway.ap-southeast-1.amazonaws.com i:/C=IN/O=Company/CN=Admin/ST=KA/L=New town/OU=SGW/emailAddress=admin@company.com --- ``` The activation endpoint accepts SSL handshakes only if it recognizes the SSL certificate. This means that the gateway's outbound traffic to the endpoints must be exempt from inspections performed by firewalls in your network. These inspections might be an SSL inspection or a deep packet inspection. ### Check gateway time synchronization Excessive time skews can cause SSL handshake errors. For on-premises gateways, you can use the gateway's local VM console to check your gateway's time synchronization. The time skew should be no larger than 60 seconds. For more information, see [Synchronizing Your Gateway VM Time](https://docs.aws.amazon.com/filegateway/latest/files3/MaintenanceTimeSync-hyperv.html). The **System Time Management** option isn't available on gateways that are hosted on Amazon EC2 instances. To make sure Amazon EC2 gateways can properly synchronize time, confirm that the Amazon EC2 instance can connect to the following NTP server pool list over ports UDP and TCP 123: + time.aws.com + 0.amazon.pool.ntp.org + 1.amazon.pool.ntp.org + 2.amazon.pool.ntp.org + 3.amazon.pool.ntp.org ## Resolve errors when activating your gateway using an Amazon VPC endpoint To resolve activation errors when activating your gateway using an Amazon Virtual Private Cloud (Amazon VPC) endpoint, perform the following checks and configurations. ### Check the required ports Make sure the required ports within your local firewall (for gateways deployed on-premises) or security group (for gateways deployed in Amazon EC2) are open. The ports required for connecting a gateway to a Storage Gateway VPC endpoint differ from those required when connecting a gateway to public endpoints. The following ports are required for connecting to a Storage Gateway VPC endpoint: + TCP 443 + TCP 1026 + TCP 1027 + TCP 1028 + TCP 1031 + TCP 2222 For more information, see [Creating a VPC endpoint for Storage Gateway](https://docs.aws.amazon.com/filegateway/latest/files3/gateway-private-link.html#create-vpc-endpoint). Additionally, check the security group that's attached to your Storage Gateway VPC endpoint. The default security group attached to the endpoint might not allow the required ports. Create a new security group that allows traffic from your gateway's IP address range over the required ports. Then, attach that security group to the VPC endpoint. **Note** Use the [Amazon VPC console](https://console.aws.amazon.com//vpc/) to verify the security group that's attached to the VPC endpoint. View your Storage Gateway VPC endpoint from the console, and then choose the **Security Groups** tab. To confirm that the required ports are open, you can run telnet commands on the Storage Gateway VPC Endpoint. You must run these commands from a server that's in the same subnet as the gateway. You can run the tests on the first DNS name that doesn't specify an Availability Zone. For example, the following telnet commands test the required port connections using the DNS name vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com: ``` telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 443 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1026 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1027 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1028 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1031 telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 2222 ``` ### Make sure firewall security does not modify packets sent from the gateway to your Storage Gateway Amazon VPC endpoint SSL inspections, deep packet inspections, or other forms of firewall security can interfere with packets sent from the gateway. The SSL handshake fails if the SSL certificate is modified from what the activation endpoint expects. To confirm that there's no SSL inspection in progress, run an OpenSSL command on your Storage Gateway VPC endpoint. You must run this command from a machine that's in the same subnet as the gateway. Run the command for each required port: ``` $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:443 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1026 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1028 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1031 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com $ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:2222 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com ``` If there's no SSL inspection in progress, then the command returns a response similar to the following: ``` openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com CONNECTED(00000005) depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 CN = anon-cp.storagegateway.us-east-1.amazonaws.com verify return:1 --- Certificate chain 0 s:CN = anon-cp.storagegateway.us-east-1.amazonaws.com i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon 1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon i:C = US, O = Amazon, CN = Amazon Root CA 1 2 s:C = US, O = Amazon, CN = Amazon Root CA 1 i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2 i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority --- ``` If there is an ongoing SSL inspection, then the response shows an altered certificate chain, similar to the following: ``` openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com CONNECTED(00000005) depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.us-east-1.amazonaws.com verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/DC=com/DC=amazonaws/OU=AWS/CN=anon-cp.storagegateway.us-east-1.amazonaws.com i:/C=IN/O=Company/CN=Admin/ST=KA/L=New town/OU=SGW/emailAddress=admin@company.com --- ``` The activation endpoint accepts SSL handshakes only if it recognizes the SSL certificate. This means that the gateway's outbound traffic to your VPC endpoint over required ports is exempt from inspections performed by your network firewalls. These inspections might be SSL inspections or deep packet inspections. ### Check gateway time synchronization Excessive time skews can cause SSL handshake errors. For on-premises gateways, you can use the gateway's local VM console to check your gateway's time synchronization. The time skew should be no larger than 60 seconds. For more information, see [Synchronizing Your Gateway VM Time](https://docs.aws.amazon.com/filegateway/latest/files3/MaintenanceTimeSync-hyperv.html). The **System Time Management** option isn't available on gateways that are hosted on Amazon EC2 instances. To make sure Amazon EC2 gateways can properly synchronize time, confirm that the Amazon EC2 instance can connect to the following NTP server pool list over ports UDP and TCP 123: + 0.amazon.pool.ntp.org + 1.amazon.pool.ntp.org + 2.amazon.pool.ntp.org + 3.amazon.pool.ntp.org ### Check for an HTTP proxy and confirm associated security group settings Before activation, check if you have an HTTP proxy on Amazon EC2 configured on the on-premises gateway VM as a Squid proxy on port 3128. In this case, confirm the following: + The security group attached to the HTTP proxy on Amazon EC2 must have an inbound rule. This inbound rule must allow Squid proxy traffic on port 3128 from the gateway VM's IP address. + The security group attached to the Amazon EC2 VPC endpoint must have inbound rules. These inbound rules must allow traffic on ports 1026-1028, 1031, 2222, and 443 from the IP address of the HTTP proxy on Amazon EC2. ## Resolve errors when activating your gateway using a public endpoint and there is a Storage Gateway VPC endpoint in the same VPC To resolve errors when activating your gateway using a public endpoint when there is a Amazon Virtual Private Cloud (Amazon VPC) enpoint in the same VPC, perform the following checks and configurations. ### Confirm that the **Enable Private DNS Name** setting isn't enabled on your Storage Gateway VPC endpoint If **Enable Private DNS Name** is enabled, you can't activate any gateways from that VPC to the public endpoint. **To disable the private DNS name option:** 1. Open the [Amazon VPC console](https://console.aws.amazon.com//vpc/). 1. In the navigation pane, choose **Endpoints**. 1. Choose your Storage Gateway VPC endpoint. 1. Choose **Actions**. 1. Choose **Manage Private DNS Names**. 1. For **Enable Private DNS Name**, clear **Enable for this Endpoint**. 1. Choose **Modify Private DNS Names** to save the setting. # Troubleshooting: on-premises gateway issues Troubleshooting: on-premises gateway issues You can find information following about typical issues that you might encounter working with your on-premises gateways, and how to allow Support to connect to your gateway to assist with troubleshooting. The following table lists typical issues that you might encounter working with your on-premises gateways. | Issue | Action to Take | | --- | --- | | You cannot find the IP address of your gateway. | Use the hypervisor client to connect to your host to find the gateway IP address. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/files3/troubleshooting-on-premises-gateway-issues.html) If you are still having trouble finding the gateway IP address: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/files3/troubleshooting-on-premises-gateway-issues.html) | | You're having network or firewall problems. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/files3/troubleshooting-on-premises-gateway-issues.html) | | Your gateway's activation fails when you click the **Proceed to Activation** button in the Storage Gateway Management Console. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/files3/troubleshooting-on-premises-gateway-issues.html) | | You need to improve bandwidth between your gateway and AWS. | You can improve the bandwidth from your gateway to AWS by setting up your internet connection to AWS on a network adapter (NIC) separate from that connecting your applications and the gateway VM. Taking this approach is useful if you have a high-bandwidth connection to AWS and you want to avoid bandwidth contention, especially during a snapshot restore. For high-throughput workload needs, you can use [Direct Connect](https://aws.amazon.com/directconnect/) to establish a dedicated network connection between your on-premises gateway and AWS. To measure the bandwidth of the connection from your gateway to AWS, use the `CloudBytesDownloaded` and `CloudBytesUploaded` metrics of the gateway. For more on this subject, see [Performance and optimization](Performance.md). Improving your internet connectivity helps to ensure that your upload buffer does not fill up. | | Throughput to or from your gateway drops to zero. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/files3/troubleshooting-on-premises-gateway-issues.html) You can view the throughput to and from your gateway from the Amazon CloudWatch console. For more information about measuring throughput to and from your gateway to AWS, see [Performance and optimization](Performance.md). | | You are having trouble importing (deploying) Storage Gateway on Microsoft Hyper-V. | See [Troubleshooting: Microsoft Hyper-V setup](troubleshooting-hyperv-setup.md), which discusses some of the common issues of deploying a gateway on Microsoft Hyper-V. | | You receive a message that says: "The data that has been written to the volume in your gateway isn't securely stored at AWS". | You receive this message if your gateway VM was created from a clone or snapshot of another gateway VM. If this isn’t the case, contact Support. | ## Troubleshooting: Security scans show open NFS ports Troubleshooting: Open NFS ports Certain NFS ports are enabled by default, even on gateways that you only use with SMB file shares. If you use third-party security software such as Qualys to scan the network where your File Gateway is deployed, the scan results might report these open NFS ports as a potential security vulnerability. If you only use your gateway with SMB file shares and you want to disable the unused NFS ports for security reasons, use the following procedure: **To disable NFS ports on a File Gateway:** 1. Access the gateway local console command prompt using the procedure outlined in [Running Storage Gateway commands on the local console](MaintenanceGatewayConsole-fgw.md). 1. Enter the following commands to disable NFS traffic: **IPv4** ``` iptables -I INPUT -p udp -m udp --dport 111 -j DROP iptables -I INPUT -p udp -m udp --dport 2049 -j DROP iptables -I INPUT -p udp -m udp --dport 20048 -j DROP iptables -I INPUT -p tcp -m tcp --dport 111 -j DROP iptables -I INPUT -p tcp -m tcp --dport 2049 -j DROP iptables -I INPUT -p tcp -m tcp --dport 20048 -j DROP ``` **IPv6** ``` ip6tables -I INPUT -p udp -m udp --dport 111 -j DROP ip6tables -I INPUT -p udp -m udp --dport 2049 -j DROP ip6tables -I INPUT -p udp -m udp --dport 20048 -j DROP ip6tables -I INPUT -p tcp -m tcp --dport 111 -j DROP ip6tables -I INPUT -p tcp -m tcp --dport 2049 -j DROP ip6tables -I INPUT -p tcp -m tcp --dport 20048 -j DROP ``` 1. Enter the following command to confirm that the blocked NFS ports appear in the IP tables: **IPv4** ``` iptables -n -L -v --line-numbers ``` **IPv6** ``` ip6tables -n -L -v --line-numbers ``` ## Turning on Support access to help troubleshoot your gateway hosted on-premises Turning on Support access to help troubleshoot your gateway Storage Gateway provides a local console you can use to perform several maintenance tasks, including allowing Support to access your gateway to assist you with troubleshooting gateway issues. By default, Support access to your gateway is turned off. You turn on this access through the host's local console. To give Support access to your gateway, you first log in to the local console for the host, navigate to the Storage Gateway's console, and then connect to the support server. **To turn on Support access to your gateway** 1. Log in to your host's local console. + VMware ESXi – for more information, see [Accessing the Gateway Local Console with VMware ESXi](accessing-local-console.md#MaintenanceConsoleWindowVMware-common). + Microsoft Hyper-V – for more information, see [Access the Gateway Local Console with Microsoft Hyper-V](accessing-local-console.md#MaintenanceConsoleWindowHyperV-common). 1. At the prompt, enter the corresponding numeral to select **Gateway Console**. 1. Enter **h** to open the list of available commands. 1. Do one of the following: + If your gateway is using a public endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel** to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. + If your gateway is using a VPC endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel**. If your gateway is not activated, provide the VPC endpoint or IP address to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. **Note** The channel number is not a Transmission Control Protocol/User Datagram Protocol (TCP/UDP) port number. Instead, the gateway makes a Secure Shell (SSH) (TCP 22) connection to Storage Gateway servers and provides the support channel for the connection. 1. After the support channel is established, provide your support service number to Support so Support can provide troubleshooting assistance. 1. When the support session is completed, enter **q** to end it. Don't close the session until Amazon Web Services Support notifies you that the support session is complete. 1. Enter **exit** to log out of the Storage Gateway console. 1. Follow the prompts to exit the local console. # Troubleshooting: Microsoft Hyper-V setup Troubleshooting: Microsoft Hyper-V setup issues The following table lists typical issues that you might encounter when deploying Storage Gateway on the Microsoft Hyper-V platform. | Issue | Action to Take | | --- | --- | | You try to import a gateway and receive the following error message: "A server error occurred while attempting to import the virtual machine. Import failed. Unable to find virtual machine import files under location [...]. You can import a virtual machine only if you used Hyper-V to create and export it." | This error can occur for the following reasons: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/filegateway/latest/files3/troubleshooting-hyperv-setup.html) | | You try to import a gateway and receive the following error message: "A server error occurred while attempting to import the virtual machine. Import failed. Import task failed to copy file from [...]: The file exists. (0x80070050)" | If you have already deployed a gateway and you try to reuse the default folders that store the virtual hard disk files and virtual machine configuration files, then this error will occur. To fix this problem, specify new locations under **Server** in the panel on the left side of the **Hyper-V Settings** dialog box. | | You try to import a gateway and receive the following error message: "A server error occurred while attempting to import the virtual machine. Import failed. Import failed because the virtual machine must have a new identifier. Select a new identifier and try the import again." | When you import the gateway make sure you select **Copy the virtual machine** and check the **Duplicate all files** box in the **Import Virtual Machine** dialog box to create a new unique ID for the VM. | | You try to start a gateway VM and receive the following error message: "An error occurred while attempting to start the selected virtual machine(s). The child partition processor setting is incompatible with parent partition. 'AWS-Storage-Gateway' could not initialize. (Virtual machine ID [...])" | This error is likely caused by a CPU discrepancy between the required CPUs for the gateway and the available CPUs on the host. Ensure that the VM CPU count is supported by the underlying hypervisor. For more information about the requirements for Storage Gateway, see [File Gateway setup requirements](Requirements.md). | | You try to start a gateway VM and receive the following error message: "An error occurred while attempting to start the selected virtual machine(s). 'AWS-Storage-Gateway' could not initialize. (Virtual machine ID [...]) Failed to create partition: Insufficient system resources exist to complete the requested service. (0x800705AA)" | This error is likely caused by a RAM discrepancy between the required RAM for the gateway and the available RAM on the host. For more information about the requirements for Storage Gateway, see [File Gateway setup requirements](Requirements.md). | | Your snapshots and gateway software updates are occurring at slightly different times than expected. | The gateway VM's clock might be offset from the actual time, known as clock drift. Check and correct the VM's time using local gateway console's time synchronization option. For more information, see [Configuring a Network Time Protocol (NTP) server for your gateway](MaintenanceTimeSync-fgw.md). | | You need to put the unzipped Microsoft Hyper-V Storage Gateway files on the host file system. | Access the host as you do a typical Microsoft Windows server. For example, if the hypervisor host is name `hyperv-server`, then you can use the following UNC path `\\hyperv-server\c$`, which assumes that the name `hyperv-server` can be resolved or is defined in your local hosts file. | | You are prompted for credentials when connecting to hypervisor. | Add your user credentials as a local administrator for the hypervisor host by using the Sconfig.cmd tool. | | You may notice poor network performance if you turn on virtual machine queue (VMQ) for a Hyper-V host that's using a Broadcom network adapter. | For information about a workaround, see the Microsoft documentation, see [Poor network performance on virtual machines on a Windows Server 2012 Hyper-V host if VMQ is turned on](https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/poor-network-performance-hyper-v-host-vm). | # Troubleshooting: Amazon EC2 gateway issues Troubleshooting: Amazon EC2 gateway issues In the following sections, you can find typical issues that you might encounter working with your gateway deployed on Amazon EC2. For more information about the difference between an on-premises gateway and a gateway deployed in Amazon EC2, see [Deploy a default Amazon EC2 host for S3 File GatewayDeploy a customized Amazon EC2 host for S3 File Gateway](ec2-gateway-file.md). For information about using ephemeral storage, see [Using ephemeral storage with EC2 gateways](ephemeral-disk-cache.md). **Topics** + [ ## Your gateway activation hasn't occurred after a few moments ](#activation-issues) + [ ## You can't find your EC2 gateway instance in the instance list ](#find-instance) + [ ## You want to connect to your gateway instance using the Amazon EC2 serial console ](#ec2-serial-console) + [ ## You want Support to help troubleshoot your Amazon EC2 gateway ](#EC2-EnableAWSSupportAccess) ## Your gateway activation hasn't occurred after a few moments Gateway activation hasn't occurred after a few moments Check the following in the Amazon EC2 console: + Port 80 is open in the security group that you associated with the instance. For more information about adding a security group rule, see [Adding a security group rule](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html#adding-security-group-rule) in the *Amazon EC2 User Guide*. + The gateway instance is marked as running. In the Amazon EC2 console, the **State** value for the instance should be RUNNING. + Make sure that your Amazon EC2 instance type meets the minimum requirements, as described in [Storage requirements](Requirements.md#requirements-storage). After correcting the problem, try activating the gateway again. To do this, open the Storage Gateway console, choose **Deploy a new Gateway on Amazon EC2**, and re-enter the IP address of the instance. ## You can't find your EC2 gateway instance in the instance list Can't find the EC2 gateway instance in the instance list If you didn't give your instance a resource tag and you have many instances running, it can be hard to tell which instance you launched. In this case, you can take the following actions to find the gateway instance: + Check the name of the Amazon Machine Image (AMI) on the **Description** tab of the instance. An instance based on the Storage Gateway AMI should start with the text **aws-storage-gateway-ami**. + If you have several instances based on the Storage Gateway AMI, check the instance launch time to find the correct instance. ## You want to connect to your gateway instance using the Amazon EC2 serial console Connect to your Amazon EC2 gateway using the serial console You can use the Amazon EC2 serial console to troubleshoot boot, network configuration, and other issues. For instructions and troubleshooting tips, see [Amazon EC2 Serial Console](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html) in the *Amazon Elastic Compute Cloud User Guide*. ## You want Support to help troubleshoot your Amazon EC2 gateway Turning on Support access to help troubleshoot the gateway Storage Gateway provides a local console you can use to perform several maintenance tasks, including allowing Support to access your gateway to assist you with troubleshooting gateway issues. By default, Support access to your gateway is turned off. You turn on this access through the Amazon EC2 local console. You log in to the Amazon EC2 local console through a Secure Shell (SSH). To successfully log in through SSH, your instance's security group must have a rule that opens TCP port 22. **Note** If you add a new rule to an existing security group, the new rule applies to all instances that use that security group. For more information about security groups and how to add a security group rule, see [Amazon EC2 security groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html) in the *Amazon EC2 User Guide*. To let Support connect to your gateway, you first log in to the local console for the Amazon EC2 instance, navigate to the Storage Gateway's console, and then provide the access. **To turn on Support access for a gateway deployed on an Amazon EC2 instance** 1. Log in to the local console for your Amazon EC2 instance. For instructions, go to [Connect to your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html) in the *Amazon EC2 User Guide*. You can use the following command to log in to the EC2 instance's local console. ``` ssh –i PRIVATE-KEY admin@INSTANCE-PUBLIC-DNS-NAME ``` **Note** The *PRIVATE-KEY* is the `.pem` file containing the private certificate of the EC2 key pair that you used to launch the Amazon EC2 instance. For more information, see [Retrieving the public key for your key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#retriving-the-public-key) in the *Amazon EC2 User Guide*. The *INSTANCE-PUBLIC-DNS-NAME* is the public Domain Name System (DNS) name of your Amazon EC2 instance that your gateway is running on. You obtain this public DNS name by selecting the Amazon EC2 instance in the EC2 console and clicking the **Description** tab. 1. At the prompt, enter **6 - Command Prompt** to open the Support Channel console. 1. Enter **h** to open the **AVAILABLE COMMANDS** window. 1. Do one of the following: + If your gateway is using a public endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel** to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. + If your gateway is using a VPC endpoint, in the **AVAILABLE COMMANDS** window, enter **open-support-channel**. If your gateway is not activated, provide the VPC endpoint or IP address to connect to customer support for Storage Gateway. Allow TCP port 22 so you can open a support channel to AWS. When you connect to customer support, Storage Gateway assigns you a support number. Make a note of your support number. **Note** The channel number is not a Transmission Control Protocol/User Datagram Protocol (TCP/UDP) port number. Instead, the gateway makes a Secure Shell (SSH) (TCP 22) connection to Storage Gateway servers and provides the support channel for the connection. 1. After the support channel is established, provide your support service number to Support so Support can provide troubleshooting assistance. 1. When the support session is completed, enter **q** to end it. Don't close the session until Amazon Web Services Support notifies you that the support session is complete. 1. Enter **exit** to exit the Storage Gateway console. 1. Follow the console menus to log out of the Storage Gateway instance. # Troubleshooting: hardware appliance issues Troubleshooting: hardware appliance issues **Note** End of availability notice: As of May 12, 2025, the AWS Storage Gateway Hardware Appliance will no longer be offered. Existing customers with the AWS Storage Gateway Hardware Appliance can continue to use and receive support until May 2028. As an alternative, you can use the AWS Storage Gateway service to give your applications on-premises and in-cloud access to virtually unlimited cloud storage. The following topics discuss issues that you might encounter with the AWS Storage Gateway Hardware Appliance, and suggestions on troubleshooting these. **Topics** + [ ## You can't determine the service IP address ](#service_ip_address) + [ ## How do you perform a factory reset? ](#factory_reset) + [ ## How do you perform a remote restart? ](#remote-restart) + [ ## Where do you obtain Dell iDRAC support? ](#iDRAC_support) + [ ## You can't find the hardware appliance serial number ](#appliance_serial_number) + [ ## Where to obtain hardware appliance support ](#appliance_support) ## You can't determine the service IP address How to determine service IP address When attempting to connect to your service, make sure that you are using the service's IP address and not the host IP address. Configure the service IP address in the service console, and the host IP address in the hardware console. You see the hardware console when you start the hardware appliance. To go to the service console from the hardware console, choose **Open Service Console**. ## How do you perform a factory reset? How to perform a factory reset If you need to perform a factory reset on your appliance, contact the AWS Storage Gateway Hardware Appliance team for support, as described in the Support section following. ## How do you perform a remote restart? How to perform a remote restart If you need to perform a remote restart of your appliance, you can do so using the Dell iDRAC management interface. For more information, see [iDRAC9 Virtual Power Cycle: Remotely power cycle Dell EMC PowerEdge Servers](https://infohub.delltechnologies.com/en-us/p/idrac9-virtual-power-cycle-remotely-power-cycle-dell-emc-poweredge-servers/) on the Dell Technologies InfoHub website. ## Where do you obtain Dell iDRAC support? How to obtain Dell iDRAC support The Dell PowerEdge server comes with the Dell iDRAC management interface. We recommend the following: + If you use the iDRAC management interface, you should change the default password. For more information about the iDRAC credentials, see [Dell PowerEdge - What is the default sign-in credentials for iDRAC?](https://www.dell.com/support/article/en-us/sln306783/dell-poweredge-what-is-the-default-username-and-password-for-idrac?lang=en). + Make sure that the firmware is up-to-date to prevent security breaches. + Moving the iDRAC network interface to a normal (`em`) port can cause performance issues or prevent the normal functioning of the appliance. ## You can't find the hardware appliance serial number How to find the hardware appliance serial number You can find the serial number for your AWS Storage Gateway Hardware Appliance using the Storage Gateway console. **To find the hardware appliance serial number:** 1. Open the Storage Gateway console at [https://console.aws.amazon.com/storagegateway/home](https://console.aws.amazon.com/storagegateway/). 1. Choose **Hardware** from the navigation menu on the left side of the page. 1. Select your hardware appliance from the list. 1. Locate the **Serial Number** field on the **Details** tab for your appliance. ## Where to obtain hardware appliance support How to get hardware appliance support To contact AWS about technical support for your hardware appliance, see [Support](https://aws.amazon.com/contact-us). The Support team might ask you to activate the support channel to troubleshoot your gateway issues remotely. You don't need this port to be open for the normal operation of your gateway, but it is required for troubleshooting. You can activate the support channel from the hardware console as shown in the procedure following. **To open a support channel for AWS** 1. Open the hardware console. 1. Choose **Open Support Channel** at the bottom of the main page of the hardware console, and then press `Enter`. The assigned port number should appear within 30 seconds if there are no network connectivity or firewall issues. For example: **Status: Open on port 19599** 1. Note the port number and provide it to Support. # Troubleshooting: File Gateway issues Troubleshooting: File Gateway issues You can configure your File Gateway to write log entries to a Amazon CloudWatch log group. If you do, you receive notifications about gateway health status and about any errors that the gateway encounters. You can find information about these error and health notifications in CloudWatch Logs. In the following sections, you can find information that can help you understand the cause of each error and health notification and how to fix issues. **Topics** + [ ## Error: 1344 (0x00000540) ](#troubleshoot-copying-files-to-s3) + [ ## Error: GatewayClockOutOfSync ](#troubleshoot-logging-errors-gatewayclockoutofsync) + [ ## Error: InaccessibleStorageClass ](#troubleshoot-logging-errors-inaccessiblestorageclass) + [ ## Error: InvalidObjectState ](#troubleshoot-logging-errors-invalidobjectstate) + [ ## Error: ObjectMissing ](#troubleshoot-logging-errors-objectmissing) + [ ## Error: RoleTrustRelationshipInvalid ](#misconfig-trust) + [ ## Error: S3AccessDenied ](#troubleshoot-logging-errors-s3accessdenied) + [ ## Error: DroppedNotifications ](#troubleshoot-logging-errors-droppednotifications) + [ ## Notification: HardReboot ](#troubleshoot-hardreboot-notification) + [ ## Notification: Reboot ](#troubleshoot-reboot-notification) + [ ## Troubleshooting: Security scans show open NFS ports ](#troubleshoot-open-nfs-ports) + [ ## Troubleshooting: Using CloudWatch metrics ](#troubleshooting-with-cw-metrics) ## Error: 1344 (0x00000540) Error: 1344 (0x00000540) While migrating files to Amazon S3 you may encounter an `ERROR 1344 (0x00000540)` if you are trying to copy files with more than 10 Access Control Entries (ACEs) into Amazon S3. Access Control Entries are listed in the Access Control List (ACL). The Amazon S3 File Gateway can only preserve 10 ACE entries per given file or folder. **To resolve an Error 1344: Copying NTFS Security to Destination Directory.** Reduce the number of entries in Windows Permissions for files or folders that contain more than 10 entries. A common approach is to create a group containing the full list of entries, then replacing the list of entries with that single group. Once the number of entries is less the 10, you can retry copying the files or folders to the gateway. ## Error: GatewayClockOutOfSync Error: GatewayClockOutOfSync You can get a `GatewayClockOutOfSync` error when the gateway detects a difference of 5 minutes or more between the local system time and the time reported by the AWS Storage Gateway servers. Clock synchronization issues can negatively impact connectivity between the gateway and AWS. If the gateway clock is out of sync, I/O errors might occur for NFS and SMB connections, and SMB users might experience authentication errors. **To resolve a GatewayClockOutOfSync error** + Check the network configuration between the gateway and the NTP server. For more information about synchronizing the gateway VM time and updating the NTP server configuration, see [Configuring a Network Time Protocol (NTP) server for your gateway](https://docs.aws.amazon.com/filegateway/latest/files3/manage-on-premises-fgw.html#MaintenanceTimeSync-fgw). ## Error: InaccessibleStorageClass Error: InaccessibleStorageClass You can get an `InaccessibleStorageClass` error when an object has moved out of the Amazon S3 Standard storage class. Your File Gateway usually encounters this error when it tries to either upload an object to or read an object from the Amazon S3 bucket. Generally, this error means the object has moved to Amazon Glacier and is in either the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage class. Your S3 File Gateway can generate a cache report that lists all files in the gateway cache that are currently failing to upload to Amazon S3 due to this error. The information in this report can help you work with Support to resolve issues with your gateway, Amazon S3, or IAM configuration. For more information, see [Create a cache report](https://docs.aws.amazon.com/filegateway/latest/files3/create-cache-report.html). **To resolve an InaccessibleStorageClass error** + Restore the object from the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage class back to its original storage class in S3. If you restore the object to the S3 bucket to fix an upload error, the file is eventually uploaded. If you restore the object to fix a read error, the File Gateway's SMB or NFS client can then read the file. ## Error: InvalidObjectState Error: InvalidObjectState You can get an `InvalidObjectState` error when a writer other than the specified File Gateway modifies the specified file in the specified Amazon S3 bucket. As a result, the state of the file for the File Gateway doesn't match its state in Amazon S3. Any subsequent uploads of the file to Amazon S3 or retrievals of the file from Amazon S3 fail. Your S3 File Gateway can generate a cache report that lists all files in the gateway cache that are currently failing to upload to Amazon S3 due to this error. The information in this report can help you work with Support to resolve issues with your gateway, Amazon S3, or IAM configuration. For more information, see [Create a cache report](https://docs.aws.amazon.com/filegateway/latest/files3/create-cache-report.html). **To resolve an InvalidObjectState error** If the operation that modifies the file is `S3Upload` or `S3GetObject`, do the following: 1. Save the latest copy of the file to the local file system of your SMB or NFS client (you need this file copy in step 4). If the version of the file in Amazon S3 is the latest, download that version. You can do this using the AWS Management Console or AWS CLI. 1. Delete the file in Amazon S3 using the AWS Management Console or AWS CLI. 1. Delete the file from the File Gateway using your SMB or NFS client. 1. Copy the latest version of the file that you saved in step 1 to Amazon S3 using your SMB or NFS client. Do this through your File Gateway. ## Error: ObjectMissing Error: ObjectMissing You can get an `ObjectMissing` error when a writer other than the specified File Gateway deletes the specified file from the S3 bucket. Any subsequent uploads to Amazon S3 or retrievals from Amazon S3 for the object fail. Your S3 File Gateway can generate a cache report that lists all files in the gateway cache that are currently failing to upload to Amazon S3 due to this error. The information in this report can help you work with Support to resolve issues with your gateway, Amazon S3, or IAM configuration. For more information, see [Create a cache report](https://docs.aws.amazon.com/filegateway/latest/files3/create-cache-report.html). **To resolve an ObjectMissing error** If the operation that modifies the file is `S3Upload` or `S3GetObject`, do the following: 1. Save the latest copy of the file to the local file system of your SMB or NFS client (you need this file copy in step 3). 1. Delete the file from the File Gateway using your SMB or NFS client. 1. Copy the latest version of the file that you saved in step 1 using your SMB or NFS client. Do this through your File Gateway. ## Error: RoleTrustRelationshipInvalid Error: RoleTrustRelationshipInvalid You get this error when the IAM role for a file share has a misconfigured IAM trust relationship (that is, the IAM role does not trust the Storage Gateway principal named `storagegateway.amazonaws.com`). As a result, the File Gateway would not be able to get the credentials to run any operations on the S3 bucket that backs the file share. **To resolve an RoleTrustRelationshipInvalid error** + Use the IAM console or IAM API to include `storagegateway.amazonaws.com` as a principal that is trusted by your file share's IAMrole. For information about IAM role, see [Tutorial: delegate access across AWS accounts using IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html). ## Error: S3AccessDenied Error: S3AccessDenied You can get an `S3AccessDenied` error for a file share's Amazon S3 bucket access AWS Identity and Access Management (IAM) role. In this case, the S3 bucket access IAM role that is specified by `roleArn` in the error doesn't allow the operation involved. The operation isn't allowed because of the permissions for the objects in the directory specified by the Amazon S3 prefix. Your S3 File Gateway can generate a cache report that lists all files in the gateway cache that are currently failing to upload to Amazon S3 due to this error. The information in this report can help you work with Support to resolve issues with your gateway, Amazon S3, or IAM configuration. For more information, see [Create a cache report](https://docs.aws.amazon.com/filegateway/latest/files3/create-cache-report.html). **To resolve an S3AccessDenied error** + Modify the Amazon S3 access policy that is attached to `roleArn` in the File Gateway health log to allow permissions for the Amazon S3 operation. Make sure that the access policy allows permission for the operation that caused the error. Also, allow permission for the directory specified in the log for `prefix`. For information about Amazon S3 permissions, see [Specifying permissions in a policy](https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html) in *Amazon Simple Storage Service User Guide.* These operations can cause an `S3AccessDenied` error to occur: + `S3HeadObject` + `S3GetObject` + `S3ListObjects` + `S3DeleteObject` + `S3PutObject` ## Error: DroppedNotifications Error: DroppedNotifications You might see a `DroppedNotifications` error instead of other expected types of CloudWatch log entries when free storage space on your gateway's root disk is less than 1 GB, or if more than 100 health notifications are generated within a 1 minute interval. In these circumstances, the gateway stops generating detailed CloudWatch log notifications as a precautionary measure. **To resolve a DroppedNotifications error** 1. Check the `Root Disk Usage` metric on the **Monitoring** tab for your gateway in the Storage Gateway console to determine whether available root disk space is running low. 1. Increase the size of the gateway's root storage disk if available space is less than 1 GB. Refer to your virtual machine hypervisor's documentation for instructions. To increase root disk size for Amazon EC2 gateways, see [Request modifications to your EBS volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requesting-ebs-volume-modifications.html) in the *Amazon Elastic Compute Cloud User Guide*. **Note** It is not possible to increase the root disk size for the AWS Storage Gateway Hardware Appliance. 1. Restart your gateway. ## Notification: HardReboot Notification: HardReboot You can get a `HardReboot` notification when the gateway VM is restarted unexpectedly. Such a restart can be due to loss of power, a hardware failure, or another event. For VMware gateways, a reset by vSphere High Availability Application Monitoring can cause this event. When your gateway runs in such an environment, check for the presence of the `HealthCheckFailure` notification and consult the VMware events log for the VM. ## Notification: Reboot Notification: Reboot You can get a reboot notification when the gateway VM is restarted. You can restart a gateway VM by using the VM Hypervisor Management console or the Storage Gateway console. You can also restart by using the gateway software during the gateway's maintenance cycle. If the time of the reboot is within 10 minutes of the gateway's configured [maintenance start time](MaintenanceManagingUpdate-common.md), this reboot is probably a normal occurrence and not a sign of any problem. If the reboot occurred significantly outside the maintenance window, check whether the gateway was restarted manually. ## Troubleshooting: Security scans show open NFS ports Troubleshooting: Open NFS ports Certain NFS ports are enabled by default, even on gateways that you only use with SMB file shares. If you use third-party security software such as Qualys to scan the network where your File Gateway is deployed, the scan results might report these open NFS ports as a potential security vulnerability. If you only use your gateway with SMB file shares and you want to disable the unused NFS ports for security reasons, use the following procedure: **To disable NFS ports on a File Gateway:** 1. Access the gateway local console command prompt using the procedure outlined in [Running Storage Gateway commands on the local console](MaintenanceGatewayConsole-fgw.md). 1. Enter the following commands to disable NFS traffic: **IPv4** ``` iptables -I INPUT -p udp -m udp --dport 111 -j DROP iptables -I INPUT -p udp -m udp --dport 2049 -j DROP iptables -I INPUT -p udp -m udp --dport 20048 -j DROP iptables -I INPUT -p tcp -m tcp --dport 111 -j DROP iptables -I INPUT -p tcp -m tcp --dport 2049 -j DROP iptables -I INPUT -p tcp -m tcp --dport 20048 -j DROP ``` **IPv6** ``` ip6tables -I INPUT -p udp -m udp --dport 111 -j DROP ip6tables -I INPUT -p udp -m udp --dport 2049 -j DROP ip6tables -I INPUT -p udp -m udp --dport 20048 -j DROP ip6tables -I INPUT -p tcp -m tcp --dport 111 -j DROP ip6tables -I INPUT -p tcp -m tcp --dport 2049 -j DROP ip6tables -I INPUT -p tcp -m tcp --dport 20048 -j DROP ``` 1. Enter the following command to confirm that the blocked NFS ports appear in the IP tables: **IPv4** ``` iptables -n -L -v --line-numbers ``` **IPv6** ``` ip6tables -n -L -v --line-numbers ``` ## Troubleshooting: Using CloudWatch metrics Troubleshooting with CloudWatch metrics You can find information following about actions to address issues using Amazon CloudWatch metrics with Storage Gateway. **Topics** + [ ### Your gateway reacts slowly when browsing directories ](#slow-gateway) + [ ### Your gateway isn't responding ](#gateway-not-responding) + [ ### Your gateway is slow transferring data to Amazon S3 ](#slow-data-transfer-to-S3) + [ ### Your gateway is performing more Amazon S3 operations than expected ](#gateway-performing-more-s3-operations) + [ ### You do not see files in your Amazon S3 bucket ](#files-missing-s3-bucket) + [ ### Your gateway backup job fails or there are errors when writing to your gateway ](#backup-job-fails) ### Your gateway reacts slowly when browsing directories Gateway reacts slowly when browsing directories If your File Gateway reacts slowly when you run the **ls** command or browse directories, check the `IndexFetch` and `IndexEviction` CloudWatch metrics: + If the `IndexFetch` metric is greater than 0 when you run an `ls` command or browse directories, your File Gateway started without information on the contents of the directory affected and had to access Amazon S3. Subsequent efforts to list the contents of that directory should go faster. + If the `IndexEviction` metric is greater than 0, it means that your File Gateway has reached the limit of what it can manage in its cache at that time. In this case, your File Gateway has to free some storage space from the least recently accessed directory to list a new directory. If this occurs frequently and there is a performance impact, contact Support. Discuss with Support the contents of the related S3 bucket and recommendations to improve performance based on your use case. ### Your gateway isn't responding Gateway isn't responding If your File Gateway isn't responding, do the following: + If there was a recent reboot or software update, then check the `IOWaitPercent` metric. This metric shows the percentage of time that the CPU is idle when there is an outstanding disk I/O request. In some cases, this might be high (10 or greater) and might have risen after the server was rebooted or updated. In these cases, then your File Gateway might be bottlenecked by a slow root disk as it rebuilds the index cache to RAM. You can address this issue by using a faster physical disk for the root disk. + If the `MemUsedBytes` metric is at or nearly the same as the `MemTotalBytes` metric, then your File Gateway is running out of available RAM. Make sure that your File Gateway has at least the minimum required RAM. If it already does, consider adding more RAM to your File Gateway based on your workload and use case. If the file share is SMB, the issue might also be due to the number of SMB clients connected to the file share. To see the number of clients connected at any given time, check the `SMBV(1/2/3)Sessions` metric. If there are many clients connected, you might need to add more RAM to your File Gateway. ### Your gateway is slow transferring data to Amazon S3 Gateway is slow transferring data to Amazon S3 If your File Gateway is slow transferring data to Amazon S3, do the following: + If the `CachePercentDirty` metric is 80 or greater, your File Gateway is writing data faster to disk than it can upload the data to Amazon S3. Consider increasing the bandwidth for upload from your File Gateway, adding one or more cache disks, or slowing down client writes. + If the `CachePercentDirty` metric is low, check the `IoWaitPercent` metric. If `IoWaitPercent` is greater than 10, your File Gateway might be bottlenecked by the speed of the local cache disk. We recommend local solid state drive (SSD) disks for your cache, preferably NVM Express (NVMe). If such disks aren't available, try using multiple cache disks from separate physical disks for a performance improvement. + If `S3PutObjectRequestTime`, `S3UploadPartRequestTime`, or `S3GetObjectRequestTime` are high, there might be a network bottleneck. Try analyzing your network to verify that the gateway has the expected bandwidth. ### Your gateway is performing more Amazon S3 operations than expected Gateway performing too many Amazon S3 operations If your File Gateway is performing more Amazon S3 operations than expected, check the `FilesRenamed` metric. Rename operations are expensive to run in Amazon S3. Optimize your workflow to minimize the number of rename operations. ### You do not see files in your Amazon S3 bucket Files missing from Amazon S3 bucket If you notice that files on the gateway are not reflected in the Amazon S3 bucket, check the `FilesFailingUpload` metric. If the metric reports that some files are failing upload, check your health notifications. When files fail to upload, the gateway generates a health notification containing more details on the issue. ### Your gateway backup job fails or there are errors when writing to your gateway Gateway backup job fails or errors when writing to the gateway If your File Gateway backup job fails or there are errors when writing to your File Gateway, do the following: + If the `CachePercentDirty` metric is 90 percent or greater, your File Gateway can't accept new writes to disk because there is not enough available space on the cache disk. To see how fast your File Gateway is uploading to Amazon S3, view the `CloudBytesUploaded` metric. Compare that metric with the `WriteBytes` metric, which shows how fast the client is writing files to your File Gateway. If the SMB client is writing to your File Gateway faster than it can upload to Amazon S3, add more cache disks to cover the size of the backup job at a minimum. Or, increase the upload bandwidth. + If a large file copy such as backup job fails but the `CachePercentDirty` metric is less than 80 percent, your File Gateway might be hitting a client-side session timeout. For SMB, you can increase this timeout using the PowerShell command `Set-SmbClientConfiguration -SessionTimeout 300`. Running this command sets the timeout to 300 seconds. For NFS, make sure that the client is mounted using a hard mount instead of a soft mount. # Troubleshooting: file share issues Troubleshooting: file share issues You can find information following about actions to take if you experience unexpected issues with your file share. **Topics** + [ ## File share stuck in CREATING, UPDATING, or DELETING state ](#troubleshooting-file-share-stuck-states) + [ ## You can't create a file share ](#create-file-troubleshoot) + [ ## SMB file shares don't allow multiple different access methods ](#smb-fileshare-troubleshoot) + [ ## Multiple file shares can't write to the mapped S3 bucket ](#multiwrite) + [ ## Notification for deleted log group when using audit logs ](#multiwrite) + [ ## Can't upload files into your S3 bucket ](#access-s3bucket) + [ ## Can't change the default encryption to use SSE-KMS to encrypt objects stored in my S3 bucket ](#encryption-issues) + [ ## Changes made directly in an S3 bucket with object versioning turned on may affect what you see in your file share ](#s3-object-versioning-file-share-issue) + [ ## When writing to an S3 bucket with versioning turned on, the Amazon S3 File Gateway may create multiple versions of Amazon S3 objects ](#s3-object-versioning-file-gateway-issue) + [ ## Changes to an S3 bucket are not reflected in Storage Gateway ](#s3-changes-issue) + [ ## ACL permissions aren't working as expected ](#smb-acl-issues) + [ ## Your gateway performance declined after you performed a recursive operation ](#recursive-operation-issues) ## File share stuck in CREATING, UPDATING, or DELETING state File share stuck in transitional state The file share status summarizes the health of your file share. If your S3 File Gateway file share is stuck in the `CREATING`, `UPDATING`, or `DELETING` state, use the following troubleshooting steps to identify and resolve the issue. ### Confirm IAM role permissions and trust relationship The AWS Identity and Access Management (IAM) role associated with your file share must have sufficient permissions to access the Amazon S3 bucket. Additionally, the role's trust policy must grant the Storage Gateway service permissions to assume the role. **To verify IAM role permissions:** 1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Roles**. 1. Choose the IAM role that's associated with your file share. 1. Choose the **Trust relationships** tab. 1. Confirm that Storage Gateway is listed as a trusted entity. If Storage Gateway isn't a trusted entity, choose **Edit trust relationship**, and then add the following policy: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "storagegateway.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` 1. Verify that the IAM role has the correct permissions and that the Amazon S3 bucket is listed as a resource in the IAM policy. For more information, see [Granting access to an Amazon S3 bucket](grant-access-s3.md). **Note** To avoid cross-service confused deputy prevention issues, use a trust relationship policy that includes condition context keys. For more information, see [Cross-service confused deputy prevention](cross-service-confused-deputy-prevention.md). ### Verify AWS STS is activated in your Region File shares can become stuck in the `CREATING` or `UPDATING` state if AWS Security Token Service (AWS STS) is deactivated in your AWS Region. **To verify AWS STS status:** 1. Open the AWS Identity and Access Management console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the navigation pane, choose **Account settings**. 1. In the **Security Token Service (STS)** section, verify that the **Status** is **Active** for the AWS Region where you want to create the file share. 1. If the status is **Inactive**, choose **Activate** to enable AWS STS in that Region. ### Verify S3 bucket exists and follows naming rules Your file share requires a valid Amazon S3 bucket that follows Amazon S3 naming conventions. **To verify your S3 bucket:** 1. Open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. Confirm that the Amazon S3 bucket mapped to your file share exists. If the bucket doesn't exist, create it. After you create the bucket, the file share status should change to `AVAILABLE`. For more information, see [Create a bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html) in the *Amazon Simple Storage Service User Guide*. 1. Verify that your bucket name complies with the [rules for bucket naming](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules) in the *Amazon Simple Storage Service User Guide*. **Note** S3 File Gateway does not support Amazon S3 buckets with periods (`.`) in the bucket name. ### Force delete a file share stuck in DELETING state When you delete a file share, the gateway removes the share from the associated Amazon S3 bucket. However, data that's currently uploading continues to upload before the deletion completes. During this process, the file share shows a `DELETING` status. **Important** Check the Amazon CloudWatch metric `CachePercentDirty` for your gateway to determine how much data is pending upload. For more information about Storage Gateway metrics, see [Monitoring your S3 File Gateway](monitoring-file-gateway.md). If you don't want to wait for all in-progress uploads to finish, you can force delete the file share. **To force delete a file share:** 1. Open the Storage Gateway console at [https://console.aws.amazon.com/storagegateway/](https://console.aws.amazon.com/storagegateway/). 1. In the navigation pane, choose **File shares**. 1. Select the file share that you want to delete. 1. Choose the **Details** tab, and review the **This file share is being deleted** message. 1. Verify the ID of the file share in the message, and then select the confirmation box. **Note** You can't undo the force delete operation. 1. Choose **Force delete now**. Alternatively, you can use the AWS CLI [delete-file-share](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/storagegateway/delete-file-share.html) command with the `--force-delete` parameter set to `true`. **Important** Before force deleting a file share, confirm that your gateway isn't in an `OFFLINE` state. If the gateway is offline, first resolve the offline issue. For more information, see [Troubleshooting: gateway offline in the Storage Gateway console](troubleshooting-gateway-offline.md). If the gateway virtual machine (VM) is already deleted, you must delete the gateway from the Storage Gateway console to remove all associated file shares, including those stuck in the `DELETING` state. For more information, see [Deleting your gateway and removing associated resources](deleting-gateway-common.md). ### Troubleshoot network connectivity issues Network issues can prevent your file share from transitioning out of the `CREATING`, `UPDATING`, or `DELETING` state. Common network issues include: + Your gateway is offline or the gateway VM is deleted. + Network access between Storage Gateway and the Amazon S3 service endpoint is blocked. + The Amazon S3 Amazon VPC endpoint that the gateway uses to communicate with Amazon S3 was deleted. + Required network ports aren't open or network routing is improperly configured. #### Test S3 connectivity from the gateway local console **To test S3 connectivity:** 1. Log in to your gateway's local console. For more information, see [Logging in to the File Gateway local console](LocalConsole-login-fgw.md). 1. In the **Storage Gateway - Configuration** main menu, enter the number corresponding to **Test S3 Connectivity**. 1. Choose the Amazon S3 endpoint type: + For Amazon S3 traffic that flows through an Internet Gateway, NAT Gateway, Transit Gateway, or Amazon S3 Gateway Amazon VPC endpoint, choose **Public**. + For Amazon S3 traffic that flows through an Amazon S3 interface Amazon VPC endpoint, choose **VPC (PrivateLink)**. + For a FIPS endpoint, choose the FIPS option. 1. Enter the Amazon S3 bucket Region. 1. If using a Amazon VPC endpoint, enter the Amazon S3 Amazon VPC endpoint DNS name (for example, `vpce-0329c2790456f2d01-0at85l34`). The gateway automatically performs a connectivity test that validates both the network connection and SSL connection. If the test fails: + **Network Test failure** - Usually caused by firewall rules, security group configurations, or improper network routing. Verify that required ports are open and network routing is configured correctly. + **SSL Test failure** - Indicates that SSL inspection or deep packet inspection is occurring between your gateway VM and Amazon S3 service endpoints. Disable SSL and deep packet inspection for Storage Gateway traffic. #### Verify proxy configuration If your gateway uses a proxy server, verify that the proxy isn't blocking network communication. **To check proxy configuration:** 1. In the **Storage Gateway - Configuration** main menu, enter the number corresponding to **HTTP/SOCKS Proxy Configuration**. 1. Select the option to view the current network proxy configuration. 1. If a proxy is configured, verify that Amazon S3 traffic can flow from Storage Gateway to the proxy server over port 3128 (or your configured listener port), and then to the Amazon S3 endpoint over port 443. 1. Confirm that the proxy or firewall allows traffic to and from the network ports and service endpoints required by Storage Gateway. For more information, see the required network ports. If issues persist, you can temporarily remove the proxy configuration to determine if the proxy is causing the problem. #### Verify security groups and network routing + **For gateways on Amazon EC2** - Confirm that the security group has port 443 open to Amazon S3 endpoints. Verify that the Amazon EC2 subnet's route table properly routes Amazon S3 traffic to Amazon S3 endpoints. For more information, see the required network ports. + **For on-premises gateways** - Confirm that firewall rules allow the required ports and that local route tables properly route Amazon S3 traffic to Amazon S3 endpoints. For more information, see the required network ports. + **VPC endpoints** - Verify that the Amazon S3 Amazon VPC endpoint used by the gateway hasn't been deleted. If the Amazon VPC endpoint is deleted and the gateway has no public IP address, the gateway can't communicate with Amazon S3. ## You can't create a file share Can't create a file share 1. If you can't create a file share because your file share is stuck in CREATING status, verify that the S3 bucket you mapped your file share to exists. For information on how to do so, see [File share stuck in CREATING, UPDATING, or DELETING state](#troubleshooting-file-share-stuck-states), preceding. 1. If the S3 bucket exists, then verify that AWS Security Token Service is activated in the region where you are creating the file share. If a security token is not active, you should activate it. For information about how to activate a token using AWS Security Token Service, see [Activating and deactivating AWS STS in an AWS Region](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) in the *IAM User Guide*. ## SMB file shares don't allow multiple different access methods SMB file shares don't allow multiple different access methods SMB file shares have the following restrictions: 1. When the same client attempts to mount both an Active Directory and Guest access SMB file share the following error message is displayed: `Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all previous connections to the server or shared resource and try again.` 1. A Windows user cannot remain connected to two Guest Access SMB file shares, and may be disconnected when a new Guest Access connection is established. 1. A Windows client can't mount both a Guest Access and an Active Directory SMB file share that is exported by the same gateway. ## Multiple file shares can't write to the mapped S3 bucket Multiple file shares can't write to the mapped S3 bucket We don't recommend configuring your S3 bucket to allow multiple file shares to write to one S3 bucket. This approach can cause unpredictable results. Instead, we recommend that you allow only one file share to write to each S3 bucket. You create a bucket policy to allow only the role associated with your file share to write to the bucket. For more information, see [Best Practices for File Gateway](https://docs.aws.amazon.com/filegateway/latest/files3/best-practices.html). ## Notification for deleted log group when using audit logs Notification for deleted log group when using audit logs If the log group does not exist, the user could select the log group link below that message to go either create a new log group or use an existing log group to use as the target for audit logs ## Can't upload files into your S3 bucket Can't upload files into S3 bucket If you can't upload files into your S3 bucket, do the following: 1. Make sure you have granted the required access for the Amazon S3 File Gateway to upload files into your S3 bucket. For more information, see [Granting access to an Amazon S3 bucket](grant-access-s3.md). 1. Make sure the role that created the bucket has permission to write to the S3 bucket. For more information, see [Best Practices for File Gateway](https://docs.aws.amazon.com/filegateway/latest/files3/best-practices.html). 1. If your File Gateway uses SSE-KMS or DSSE-KMS for encryption, make sure the IAM role associated with the file share includes *kms:Encrypt*, *kms:Decrypt*, *kms:ReEncrypt\$1*, *kms:GenerateDataKey*, and *kms:DescribeKey* permissions. For more information, see [Using Identity-Based Policies (IAM Policies) for Storage Gateway](https://docs.aws.amazon.com/filegateway/latest/files3/using-identity-based-policies.html). ## Can't change the default encryption to use SSE-KMS to encrypt objects stored in my S3 bucket Can't change default encryption to SSE-KMS If you change the default encryption and make SSE-KMS (server-side encryption with AWS KMS–managed keys) the default for your S3 bucket, objects that a Amazon S3 File Gateway stores in the bucket are not encrypted with SSE-KMS. By default, a S3 File Gateway uses server-side encryption managed with Amazon S3 (SSE-S3) when it writes data to an S3 bucket. Changing the default won't automatically change your encryption. To change the encryption to use SSE-KMS with your own AWS KMS key, you must turn on SSE-KMS encryption. To do so, you provide the Amazon Resource Name (ARN) of the KMS key when you create your file share. You can also update KMS settings for your file share by using the `UpdateNFSFileShare` or `UpdateSMBFileShare` API operation. This update applies to objects stored in the S3 buckets after the update. For more information, see [Data encryption using AWS KMS](encryption.md). ## Changes made directly in an S3 bucket with object versioning turned on may affect what you see in your file share If your S3 bucket has objects written to it by another client, your view of the S3 bucket might not be up-to-date as a result of S3 bucket object versioning. You should always refresh your cache before examining files of interest. *Object versioning *is an optional S3 bucket feature that helps protect data by storing multiple copies of the same-named object. Each copy has a separate ID value, for example `file1.jpg`: `ID="xxx"` and `file1.jpg`: `ID="yyy"`. The number of identically named objects and their lifetimes is controlled by Amazon S3 lifecycle policies. For more details on these Amazon S3 concepts, see [Using versioning](https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html) and [Object lifecycle management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) in the *Amazon S3 Developer Guide. * When you delete a versioned object, that object is flagged with a delete marker but retained. Only an S3 bucket owner can permanently delete an object with versioning turned on. In your S3 File Gateway, files shown are the most recent versions of objects in an S3 bucket at the time the object was fetched or the cache was refreshed. S3 File Gateways ignore any older versions or any objects marked for deletion. When reading a file, you read data from the latest version. When you write a file in your file share, your S3 File Gateway creates a new version of a named object with your changes, and that version becomes the latest version. Your S3 File Gateway continues to read from the earlier version, and updates that you make are based on the earlier version should a new version be added to the S3 bucket outside of your application. To read the latest version of an object, use the [RefreshCache](https://docs.aws.amazon.com/storagegateway/latest/APIReference/API_RefreshCache.html) API action or refresh from the console as described in [Refreshing Amazon S3 bucket object cache](refresh-cache.md). **Important** We don't recommend that objects or files be written to your S3 File Gateway S3 bucket from outside of the file share. ## When writing to an S3 bucket with versioning turned on, the Amazon S3 File Gateway may create multiple versions of Amazon S3 objects With object versioning turned on, you may have multiple versions of an object created in Amazon S3 on every update to a file from your NFS or SMB client. Here are scenarios that can result in multiple versions of an object being created in your S3 bucket: + When a file is modified in the Amazon S3 File Gateway by an NFS or SMB client after it has been uploaded to Amazon S3, the S3 File Gateway uploads the new or modified data instead of uploading the whole file. The file modification results in a new version of the Amazon S3 object being created. + When a file is written to the S3 File Gateway by an NFS or SMB client, the S3 File Gateway uploads the file's data to Amazon S3 followed by its metadata, (ownerships, timestamps, etc.). Uploading the file data creates an Amazon S3 object, and uploading the metadata for the file updates the metadata for the Amazon S3 object. This process creates another version of the object, resulting in two versions of an object. + When the S3 File Gateway is uploading larger files, it might need to upload smaller chunks of the file before the client is done writing to the File Gateway. Some reasons for this include to free up cache space or a high rate of writes to a file. This can result in multiple versions of an object in the S3 bucket. You should monitor your S3 bucket to determine how many versions of an object exist before setting up lifecycle policies to move objects to different storage classes. You should configure lifecycle expiration for previous versions to minimize the number of versions you have for an object in your S3 bucket. The use of Same-Region replication (SRR) or Cross-Region replication (CRR) between S3 buckets will increase the storage used. For more information about replication, see [Replication](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.html). **Important** Do not configure replication between S3 buckets until you understand how much storage is being used when object versioning is turned on. Use of versioned S3 buckets can greatly increase the amount of storage in Amazon S3 because each modification to a file creates a new version of the S3 object. By default, Amazon S3 continues to store all of these versions unless you specifically create a policy to override this behavior and limit the number of versions that are kept. If you notice unusually large storage usage with object versioning turned on, check that you have your storage policies set appropriately. An increase in the number of `HTTP 503-slow down` responses for browser requests can also be the result of problems with object versioning. If you turn on object versioning after installing a S3 File Gateway, all unique objects are retained (`ID=”NULL”`) and you can see them all in the file system. New versions of objects are assigned a unique ID (older versions are retained). Based on the object's timestamp only the newest versioned object is viewable in the NFS file system. After you turn on object versioning, your S3 bucket can't be returned to a nonversioned state. You can, however, suspend versioning. When you suspend versioning, a new object is assigned an ID. If the same named object exists with an `ID=”NULL”` value, the older version is overwritten. However, any version that contains a non-`NULL` ID is retained. Timestamps identify the new object as the current one, and that is the one that appears in the NFS file system. ## Changes to an S3 bucket are not reflected in Storage Gateway Storage Gateway updates the file share cache automatically when you write files to the cache locally using the file share. However, Storage Gateway doesn't automatically update the cache when you upload a file directly to Amazon S3. When you do this, you must perform a `RefreshCache` operation to see the changes on the file share. If you have more than one file share, then you must run the `RefreshCache` operation on each file share. You can refresh the cache using the Storage Gateway console and the AWS Command Line Interface (AWS CLI): + To refresh the cache using the Storage Gateway console, see Refreshing objects in your Amazon S3 bucket. + To refresh the cache using the AWS CLI: 1. Run the command `aws storagegateway list-file-shares` 1. Copy the Amazon Resource Number (ARN) of the file share with the cache that you want to refresh. 1. Run the `refresh-cache` command with your ARN as the value for `--file-share-arn`: `aws storagegateway refresh-cache --file-share-arn arn:aws:storagegateway:eu-west-1:12345678910:share/share-FFDEE12` To automate the `RefreshCache` operation, see [ How can I automate the RefreshCache operation on Storage Gateway?](https://aws.amazon.com/premiumsupport/knowledge-center/storage-gateway-automate-refreshcache/) ## ACL permissions aren't working as expected ACL permissions aren't working as expected If access control list (ACL) permissions aren't working as you expect with your SMB file share, you can perform a test. To do this, first test the permissions on a Microsoft Windows file server or a local Windows file share. Then compare the behavior to your gateway's file share. ## Your gateway performance declined after you performed a recursive operation Gateway performance declined after a recursive operation In some cases, you might perform a recursive operation, such as renaming a directory or turning on inheritance for an ACL, and force it down the tree. If you do this, your S3 File Gateway recursively applies the operation to all objects in the file share. For example, suppose that you apply inheritance to existing objects in an S3 bucket. Your S3 File Gateway recursively applies inheritance to all objects in the bucket. Such operations can cause your gateway performance to decline. ## High Availability Health Notifications When running your gateway on the VMware vSphere High Availability (HA) platform, you may receive health notifications. For more information about health notifications, see [Troubleshooting: high availability issues](troubleshooting-ha-issues.md). # Troubleshooting: high availability issues Troubleshooting: high availability issues You can find information following about actions to take if you experience availability issues. **Topics** + [ ## Health notifications ](#ha-health-notifications) + [ ## Metrics ](#ha-health-notification-metrics) ## Health notifications Health notifications When you run your gateway on VMware vSphere HA, all gateways produce the following health notifications to your configured Amazon CloudWatch log group. These notifications go into a log stream called `AvailabilityMonitor`. **Topics** + [ ### Notification: Reboot ](#troubleshoot-reboot-notification) + [ ### Notification: HardReboot ](#troubleshoot-hardreboot-notification) + [ ### Notification: HealthCheckFailure ](#troubleshoot-healthcheckfailure-notification) + [ ### Notification: AvailabilityMonitorTest ](#troubleshoot-availabilitymonitortest-notification) ### Notification: Reboot Notification: Reboot You can get a reboot notification when the gateway VM is restarted. You can restart a gateway VM by using the VM Hypervisor Management console or the Storage Gateway console. You can also restart by using the gateway software during the gateway's maintenance cycle. **Action to Take** If the time of the reboot is within 10 minutes of the gateway's configured [maintenance start time](MaintenanceManagingUpdate-common.md), this is probably a normal occurrence and not a sign of any problem. If the reboot occurred significantly outside the maintenance window, check whether the gateway was restarted manually. ### Notification: HardReboot Notification: HardReboot You can get a `HardReboot` notification when the gateway VM is restarted unexpectedly. Such a restart can be due to loss of power, a hardware failure, or another event. For VMware gateways, a reset by vSphere High Availability Application Monitoring can cause this event. **Action to Take** When your gateway runs in such an environment, check for the presence of the `HealthCheckFailure` notification and consult the VMware events log for the VM. ### Notification: HealthCheckFailure Notification: HealthCheckFailure For a gateway on VMware vSphere HA, you can get a `HealthCheckFailure` notification when a health check fails and a VM restart is requested. This event also occurs during a test to monitor availability, indicated by an `AvailabilityMonitorTest` notification. In this case, the `HealthCheckFailure` notification is expected. **Note** This notification is for VMware gateways only. **Action to Take** If this event repeatedly occurs without an `AvailabilityMonitorTest` notification, check your VM infrastructure for issues (storage, memory, and so on). If you need additional assistance, contact Support. ### Notification: AvailabilityMonitorTest Notification: AvailabilityMonitorTest For a gateway on VMware vSphere HA, you can get an `AvailabilityMonitorTest` notification when you [run a test](vmware-ha.md#vmware-ha-test-failover) of the [Availability and application monitoring](https://docs.aws.amazon.com/storagegateway/latest/APIReference/API_StartAvailabilityMonitorTest.html) system in VMware. ## Metrics Metrics The `AvailabilityNotifications` metric is available on all gateways. This metric is a count of the number of availability-related health notifications generated by the gateway. Use the `Sum` statistic to observe whether the gateway is experiencing any availability-related events. Consult with your configured CloudWatch log group for details about the events.