# Best Practices for Windows Windows This guide provides advice about running windows containers and nodes. **Topics** + [ # Amazon EKS optimized Windows AMI management ](windows-ami.md) + [ # Configure gMSA for Windows Pods and containers ](windows-gmsa.md) + [ # Windows worker nodes hardening ](windows-hardening.md) + [ # Container image scanning ](windows-images.md) + [ # Windows Server version and License ](windows-licensing.md) + [ # Logging ](windows-logging.md) + [ # Monitoring ](windows-monitoring.md) + [ # Windows Networking ](windows-networking.md) + [ # Avoiding OOM errors ](windows-oom.md) + [ # Patching Windows Servers and Containers ](windows-patching.md) + [ # Running Heterogeneous workloads ](windows-scheduling.md) + [ # Pod Security Contexts ](windows-security.md) + [ # Persistent storage options ](windows-storage.md) + [ # Hardening Windows container images ](windows-hardening-containers-images.md) # Amazon EKS optimized Windows AMI management AMI Management Windows Amazon EKS optimized AMIs are built on top of Windows Server 2019 and Windows Server 2022. They are configured to serve as the base image for Amazon EKS nodes. By default, the AMIs include the following components: + [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) + [kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) + [AWS IAM Authenticator for Kubernetes](https://github.com/kubernetes-sigs/aws-iam-authenticator) + [csi-proxy](https://github.com/kubernetes-csi/csi-proxy) + [containerd](https://containerd.io/) You can programmatically retrieve the Amazon Machine Image (AMI) ID for Amazon EKS optimized AMIs by querying the AWS Systems Manager Parameter Store API. This parameter eliminates the need for you to manually look up Amazon EKS optimized AMI IDs. For more information about the Systems Manager Parameter Store API, see [GetParameter](https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_GetParameter.html). Your user account must have the ssm:GetParameter IAM permission to retrieve the Amazon EKS optimized AMI metadata. The following example retrieves the AMI ID for the latest Amazon EKS optimized AMI for Windows Server 2019 LTSC Core. The version number listed in the AMI name relates to the corresponding Kubernetes build it is prepared for. ``` aws ssm get-parameter --name /aws/service/ami-windows-latest/Windows_Server-2019-English-Core-EKS_Optimized-1.21/image_id --region us-east-1 --query "Parameter.Value" --output text ``` Example output: ``` ami-09770b3eec4552d4e ``` ## Managing your own Amazon EKS optimized Windows AMI An essential step towards production environments is maintaining the same Amazon EKS optimized Windows AMI and kubelet version across the Amazon EKS cluster. Using the same version across the Amazon EKS cluster reduces the time during troubleshooting and increases cluster consistency. [Amazon EC2 Image Builder](https://aws.amazon.com/image-builder/) helps create and maintain custom Amazon EKS optimized Windows AMIs to be used across an Amazon EKS cluster. Use Amazon EC2 Image Builder to select between Windows Server versions, AWS Windows Server AMI release dates, and/or OS build version. The build components step, allows you to select between existing EKS Optimized Windows Artifacts as well as the kubelet versions. For more information: https://docs.aws.amazon.com/eks/latest/userguide/eks-custom-ami-windows.html ![\[build components\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/build-components.png) **NOTE:** Prior to selecting a base image, consult the [Windows Server Version and License](windows-licensing.md) section for important details pertaining to release channel updates. ## Configuring faster launching for custom EKS optimized AMIs When using a custom Windows Amazon EKS optimized AMI, Windows worker nodes can be launched up to 65% faster by enabling the Fast Launch feature. This feature maintains a set of pre-provisioned snapshots which have the *Sysprep specialize*, *Windows Out of Box Experience (OOBE)* steps and required reboots already completed. These snapshots are then used on subsequent launches, reducing the time to scale-out or replace nodes. Fast Launch can be only enabled for AMIs *you own* through the EC2 console or in the AWS CLI and the number of snapshots maintained is configurable. **NOTE:** Fast Launch is not compatible with the default Amazon-provided EKS optimized AMI, create a custom AMI as above before attempting to enable it. For more information: [AWS Windows AMIs - Configure your AMI for faster launching](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/windows-ami-version-history.html#win-ami-config-fast-launch) ## Caching Windows base layers on custom AMIs Windows container images are larger than their Linux counterparts. If you are running any containerized .NET Framework-based application, the average image size is around 8.24GB. During pod scheduling, the container image must be fully pulled and extracted in the disk before the pod reaches Running status. During this process, the container runtime (containerd) pulls and extracts the entire container image in the disk. The pull operation is a parallel process, meaning the container runtime pulls the container image layers in parallel. In contrast, the extraction operation occurs in a sequential process, and it is I/O intensive. Due to that, the container image can take more than 8 minutes to be fully extracted and ready to be used by the container runtime (containerd), and as a result, the pod startup time can take several minutes. As mentioned in the **Patching Windows Server and Container** topic, there is an option to build a custom AMI with EKS. During the AMI preparation, you can add an additional EC2 Image builder component to pull all the necessary Windows container images locally and then generate the AMI. This strategy will drastically reduce the time a pod reaches the status **Running**. On Amazon EC2 Image Builder, create a [component](https://docs.aws.amazon.com/imagebuilder/latest/userguide/manage-components.html) to download the necessary images and attach it to the Image recipe. The following example pulls a specific image from a ECR repository. ``` name: ContainerdPull description: This component pulls the necessary containers images for a cache strategy. schemaVersion: 1.0 phases: - name: build steps: - name: containerdpull action: ExecutePowerShell inputs: commands: - Set-ExecutionPolicy Unrestricted -Force - (Get-ECRLoginCommand).Password | docker login --username AWS --password-stdin 111000111000.dkr.ecr.us-east-1.amazonaws.com - ctr image pull mcr.microsoft.com/dotnet/framework/aspnet:latest - ctr image pull 111000111000.dkr.ecr.us-east-1.amazonaws.com/myappcontainerimage:latest ``` To make sure the following component works as expected, check if the IAM role used by EC2 Image builder (EC2InstanceProfileForImageBuilder) has the attached policies: ![\[permissions policies\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/permissions-policies.png) ## Blog post In the following blog post, you will find a step by step on how to implement caching strategy for custom Amazon EKS Windows AMIs: [Speeding up Windows container launch times with EC2 Image builder and image cache strategy](https://aws.amazon.com/blogs/containers/speeding-up-windows-container-launch-times-with-ec2-image-builder-and-image-cache-strategy/) # Configure gMSA for Windows Pods and containers gMSA for Windows Containers ## What is a gMSA account Windows-based applications such as .NET applications often use Active Directory as an identity provider, providing authorization/authentication using NTLM or Kerberos protocol. An application server to exchange Kerberos tickets with Active Directory requires to be domain-joined. Windows containers don’t support domain joins and would not make much sense as containers are ephemeral resources, creating a burden on the Active Directory RID pool. However, administrators can leverage [gMSA Active Directory](https://docs.microsoft.com/en-us/windows-server/security/group-managed-service-accounts/group-managed-service-accounts-overview) accounts to negotiate a Windows authentication for resources such as Windows containers, NLB, and server farms. ## Windows container and gMSA use case Applications that leverage on Windows authentication, and run as Windows containers, benefit from gMSA because the Windows Node is used to exchange the Kerberos ticket on behalf of the container.There are two options available to setup the Windows worker node to support gMSA integration: In this setup, the Windows worker node is domain-joined in the Active Directory domain, and the AD Computer account of the Windows worker nodes is used to authenticate against Active Directory and retrieve the gMSA identity to be used with the pod. In the domain-joined approach, you can easily manage and harden your Windows worker nodes using existing Active Directory GPOs; however, it generates additional operational overhead and delays during Windows worker node joining in the Kubernetes cluster, as it requires additional reboots during node startup and Active Directory garage cleaning after the Kubernetes cluster terminates nodes. In the following blog post, you will find a detailed step-by-step on how to implement the Domain-joined Windows worker node approach: [Windows Authentication on Amazon EKS Windows pods](https://aws.amazon.com/blogs/containers/windows-authentication-on-amazon-eks-windows-pods/) In this setup, the Windows worker node isn’t joined in the Active Directory domain, and a "portable" identity (user/password) is used to authenticate against Active Directory and retrieve the gMSA identity to be used with the pod. ![\[domainless gmsa\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/domainless_gmsa.png) The portable identity is an Active Directory user; the identity (user/password) is stored on AWS Secrets Manager or AWS System Manager Parameter Store, and an AWS-developed plugin called ccg\$1plugin will be used to retrieve this identity from AWS Secrets Manager or AWS System Manager Parameter Store and pass it to containerd to retrieve the gMSA identity and made it available for the pod. In this domainless approach, you can benefit from not having any Active Directory interaction during Windows worker node startup when using gMSA and reducing the operational overhead for Active Directory administrators. In the following blog post, you will find a detailed step-by-step on how to implement the Domainless Windows worker node approach: [Domainless Windows Authentication for Amazon EKS Windows pods](https://aws.amazon.com/blogs/containers/domainless-windows-authentication-for-amazon-eks-windows-pods/) Despite the pod being able to use a gMSA account, it is necessary to also setup the application or service accordingly to support Windows authentication, for instance, in order to setup Microsoft IIS to support Windows authentication, you should prepared it via dockerfile: ``` RUN Install-WindowsFeature -Name Web-Windows-Auth -IncludeAllSubFeature RUN Import-Module WebAdministration; Set-ItemProperty 'IIS:\AppPools\SiteName' -name processModel.identityType -value 2 RUN Import-Module WebAdministration; Set-WebConfigurationProperty -Filter '/system.webServer/security/authentication/anonymousAuthentication' -Name Enabled -Value False -PSPath 'IIS:\' -Location 'SiteName' RUN Import-Module WebAdministration; Set-WebConfigurationProperty -Filter '/system.webServer/security/authentication/windowsAuthentication' -Name Enabled -Value True -PSPath 'IIS:\' -Location 'SiteName' ``` # Windows worker nodes hardening Windows Server Hardening OS Hardening is a combination of OS configuration, patching, and removing unnecessary software packages, which aim to lock down a system and reduce the attack surface. It is a best practice to prepare your own EKS Optimized Windows AMI with the hardening configurations required by your company. AWS provides a new EKS Optimized Windows AMI every month containing the latest Windows Server Security Patches. However, it is still the user’s responsibility to harden their AMI by applying the necessary OS configurations regardless of whether they use self-managed or managed node groups. Microsoft offers a range of tools like [Microsoft Security Compliance Toolkit](https://www.microsoft.com/en-us/download/details.aspx?id=55319) and [Security Baselines](https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-security-baselines) that helps you to achieve hardening based on your security policies needs. [CIS Benchmarks](https://learn.cisecurity.org/benchmarks) are also available and should be implemented on top of an Amazon EKS Optimized Windows AMI for production environments. ## Reducing attack surface with Windows Server Core Windows Server Core is a minimal installation option that is available as part of the [EKS Optimized Windows AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-windows-ami.html). Deploying Windows Server Core has a couple of benefits. First, it has a relatively small disk footprint, being 6GB on Server Core against 10GB on Windows Server with Desktop experience. Second, it has a smaller attack surface because of its smaller code base and available APIs. AWS provides customers with new Amazon EKS Optimized Windows AMIs every month, containing the latest Microsoft security patches, regardless of the Amazon EKS-supported version. As a best practice, Windows worker nodes must be replaced with new ones based on the latest Amazon EKS-optimized AMI. Any node running for more than 45 days without an update in place or node replacement lacks security best practices. ## Avoiding RDP connections Remote Desktop Protocol (RDP) is a connection protocol developed by Microsoft to provide users with a graphical interface to connect to another Windows computer over a network. As a best practice, you should treat your Windows worker nodes as if they were ephemeral hosts. That means no management connections, no updates, and no troubleshooting. Any modification and update should be implemented as a new custom AMI and replaced by updating an Auto Scaling group. See **Patching Windows Servers and Containers** and **Amazon EKS optimized Windows AMI management**. Disable RDP connections on Windows nodes during the deployment by passing the value **false** on the ssh property, as the example below: ``` nodeGroups: - name: windows-ng instanceType: c5.xlarge minSize: 1 volumeSize: 50 amiFamily: WindowsServer2019CoreContainer ssh: allow: false ``` If access to the Windows node is needed, use [AWS System Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) to establish a secure PowerShell session through the AWS Console and SSM agent. To see how to implement the solution watch [Securely Access Windows Instances Using AWS Systems Manager Session Manager](https://www.youtube.com/watch?v=nt6NTWQ-h6o) In order to use System Manager Session Manager an additional IAM policy must be applied to the IAM role used to launch the Windows worker node. Below is an example where the **AmazonSSMManagedInstanceCore** is specified in the `eksctl` cluster manifest: ``` nodeGroups: - name: windows-ng instanceType: c5.xlarge minSize: 1 volumeSize: 50 amiFamily: WindowsServer2019CoreContainer ssh: allow: false iam: attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore ``` ## Amazon Inspector [Amazon Inspector](https://aws.amazon.com/inspector/) is an automated security assessment service that helps improve the security and compliance of applications deployed on AWS. Amazon Inspector automatically assesses applications for exposure, vulnerabilities, and deviations from best practices. After performing an assessment, Amazon Inspector produces a detailed list of security findings prioritized by level of severity. These findings can be reviewed directly or as part of detailed assessment reports which are available via the Amazon Inspector console or API. Amazon Inspector can be used to run CIS Benchmark assessment on the Windows worker node and it can be installed on a Windows Server Core by performing the following tasks: 1. Download the following .exe file: https://inspector-agent.amazonaws.com/windows/installer/latest/AWSAgentInstall.exe 1. Transfer the agent to the Windows worker node. 1. Run the following command on PowerShell to install the Amazon Inspector Agent: `.\AWSAgentInstall.exe /install` Below is the ouput after the first run. As you can see, it generated findings based on the [CVE](https://cve.mitre.org/) database. You can use this to harden your Worker nodes or create an AMI based on the hardened configurations. ![\[inspector agent\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/inspector-agent.png) For more information on Amazon Inspector, including how to install Amazon Inspector agents, set up the CIS Benchmark assessment, and generate reports, watch the [Improving the security and compliance of Windows Workloads with Amazon Inspector](https://www.youtube.com/watch?v=nIcwiJ85EKU) video. ## Amazon GuardDuty [Amazon GuardDuty](https://aws.amazon.com/guardduty/) is a threat detection service that continuously monitors for malicious activity and unauthorized behavior to protect your AWS accounts, workloads, and data stored in Amazon S3. With the cloud, the collection and aggregation of account and network activities is simplified, but it can be time consuming for security teams to continuously analyze event log data for potential threats. By using Amazon GuardDuty you have visilitiby on malicious actitivy against Windows worker nodes, like RDP brute force and Port Probe attacks. Watch the [Threat Detection for Windows Workloads using Amazon GuardDuty](https://www.youtube.com/watch?v=ozEML585apQ) video to learn how to implement and run CIS Benchmarks on Optimized EKS Windows AMI ## Security in Amazon EC2 for Windows Read up on the [Security best practices for Amazon EC2 Windows instances](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-security.html) to implement security controls at every layer. # Container image scanning Scanning Windows Images Image Scanning is an automated vulnerability assessment feature that helps improve the security of your application’s container images by scanning them for a broad range of operating system vulnerabilities. Currently, the Amazon Elastic Container Registry (ECR) is only able to scan Linux container image for vulnerabilities. However; there are third-party tools which can be integrated with an existing CI/CD pipeline for Windows container image scanning. + [Anchore](https://anchore.com/blog/scanning-windows-container-images/) + [PaloAlto Prisma Cloud](https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/vulnerability_management/windows_image_scanning.html) + [Trend Micro - Deep Security Smart Check](https://www.trendmicro.com/en_us/business/products/hybrid-cloud/smart-check-image-scanning.html) To learn more about how to integrate these solutions with Amazon Elastic Container Repository (ECR), check: + [Anchore, scanning images on Amazon Elastic Container Registry (ECR)](https://anchore.com/blog/scanning-images-on-amazon-elastic-container-registry/) + [PaloAlto, scanning images on Amazon Elastic Container Registry (ECR)](https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/vulnerability_management/registry_scanning0/scan_ecr.html) + [TrendMicro, scanning images on Amazon Elastic Container Registry (ECR)](https://cloudone.trendmicro.com/docs/container-security/sc-about/) # Windows Server version and License Windows Versions and Licensing ## Windows Server version An Amazon EKS Optimized Windows AMI is based on Windows Server 2019 and 2022 Datacenter edition on the Long-Term Servicing Channel (LTSC). The Datacenter version doesn’t have a limitation on the number of containers running on a worker node. For more information: https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/faq ### Long-Term Servicing Channel (LTSC) Formerly called the "Long-Term Servicing Branch", this is the release model you are already familiar with, where a new major version of Windows Server is released every 2-3 years. Users are entitled to 5 years of mainstream support and 5 years of extended support. ## Licensing When launching an Amazon EC2 instance with a Windows Server-based AMI, Amazon covers licensing costs and license compliance for you. # Logging Containerized applications typically direct application logs to STDOUT. The container runtime traps these logs and does something with them - typically writes to a file. Where these files are stored depends on the container runtime and configuration. One fundamental difference with Windows pods is they do not generate STDOUT. You can run [LogMonitor](https://github.com/microsoft/windows-container-tools/tree/master/LogMonitor) to retrieve the ETW (Event Tracing for Windows), Windows Event Logs and other application specific logs from running Windows containers and pipes formatted log output to STDOUT. These logs can then be streamed using fluent-bit or fluentd to your desired destination such as Amazon CloudWatch. The Log collection mechanism retrieves STDOUT/STDERR logs from Kubernetes pods. A [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) is a common way to collect logs from containers. It gives you the ability to manage log routing/filtering/enrichment independently of the application. A fluentd DaemonSet can be used to stream these logs and any other application generated logs to a desired log aggregator. More detailed information about log streaming from Windows workloads to CloudWatch is explained [here](https://aws.amazon.com/blogs/containers/streaming-logs-from-amazon-eks-windows-pods-to-amazon-cloudwatch-logs-using-fluentd/) ## Logging Recomendations The general logging best practices are no different when operating Windows workloads in Kubernetes. + Always log **structured log entries** (JSON/SYSLOG) which makes handling log entries easier as there are many pre-written parsers for such structured formats. + **Centralize** logs - dedicated logging containers can be used specifically to gather and forward log messages from all containers to a destination + Keep **log verbosity** down except when debugging. Verbosity places a lot of stress on the logging infrastructure and significant events can be lost in the noise. + Always log the **application information** along with **transaction/request id** for traceability. Kubernetes objects do-not carry the application name, so for example a pod name `windows-twryrqyw` may not carry any meaning when debugging logs. This helps with traceability and troubleshooting applications with your aggregated logs. How you generate these transaction/correlation id’s depends on the programming construct. But a very common pattern is to use a logging Aspect/Interceptor, which can use [MDC](https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/MDC.html) (Mapped diagnostic context) to inject a unique transaction/correlation id to every incoming request, like so: ``` import org.slf4j.MDC; import java.util.UUID; Class LoggingAspect { //interceptor @Before(value = "execution(* *.*(..))") func before(...) { transactionId = generateTransactionId(); MDC.put(CORRELATION_ID, transactionId); } func generateTransactionId() { return UUID.randomUUID().toString(); } } ``` # Monitoring Monitoring Windows Containers Prometheus, a [graduated CNCF project](https://www.cncf.io/projects/) is by far the most popular monitoring system with native integration into Kubernetes. Prometheus collects metrics around containers, pods, nodes, and clusters. Additionally, Prometheus leverages AlertsManager which lets you program alerts to warn you if something in your cluster is going wrong. Prometheus stores the metric data as a time series data identified by metric name and key/value pairs. Prometheus includes away to query using a language called PromQL, which is short for Prometheus Query Language. The high level architecture of Prometheus metrics collection is shown below: ![\[Prometheus Metrics collection\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/prom.png) Prometheus uses a pull mechanism and scrapes metrics from targets using exporters and from the Kubernetes API using the [kube state metrics](https://github.com/kubernetes/kube-state-metrics). This means applications and services must expose a HTTP(S) endpoint containing Prometheus formatted metrics. Prometheus will then, as per its configuration, periodically pull metrics from these HTTP(S) endpoints. An exporter lets you consume third party metrics as Prometheus formatted metrics. A Prometheus exporter is typically deployed on each node. For a complete list of exporters please refer to the Prometheus [exporters](https://prometheus.io/docs/instrumenting/exporters/). While [node exporter](https://github.com/prometheus/node_exporter) is suited for exporting host hardware and OS metrics for linux nodes, it wont work for Windows nodes. In a **mixed node EKS cluster with Windows nodes** when you use the stable [Prometheus helm chart](https://github.com/prometheus-community/helm-charts), you will see failed pods on the Windows nodes, as this exporter is not intended for Windows. You will need to treat the Windows worker pool separate and instead install the [Windows exporter](https://github.com/prometheus-community/windows_exporter) on the Windows worker node group. In order to setup Prometheus monitoring for Windows nodes, you need to download and install the WMI exporter on the Windows server itself and then setup the targets inside the scrape configuration of the Prometheus configuration file. The [releases page](https://github.com/prometheus-community/windows_exporter/releases) provides all available .msi installers, with respective feature sets and bug fixes. The installer will setup the windows\$1exporter as a Windows service, as well as create an exception in the Windows firewall. If the installer is run without any parameters, the exporter will run with default settings for enabled collectors, ports, etc. You can check out the **scheduling best practices** section of this guide which suggests the use of taints/tolerations or RuntimeClass to selectively deploy node exporter only to linux nodes, while the Windows exporter is installed on Windows nodes as you bootstrap the node or using a configuration management tool of your choice (example chef, Ansible, SSM etc). Note that, unlike the linux nodes where the node exporter is installed as a daemonset , on Windows nodes the WMI exporter is installed on the host itself. The exporter will export metrics such as the CPU usage, the memory and the disk I/O usage and can also be used to monitor IIS sites and applications, the network interfaces and services. The windows\$1exporter will expose all metrics from enabled collectors by default. This is the recommended way to collect metrics to avoid errors. However, for advanced use the windows\$1exporter can be passed an optional list of collectors to filter metrics. The collect[] parameter, in the Prometheus configuration lets you do that. The default install steps for Windows include downloading and starting the exporter as a service during the bootstrapping process with arguments, such as the collectors you want to filter. ``` > Powershell Invoke-WebRequest https://github.com/prometheus-community/windows_exporter/releases/download/v0.13.0/windows_exporter-0.13.0-amd64.msi -OutFile > msiexec /i ENABLED_COLLECTORS="cpu,cs,logical_disk,net,os,system,container,memory" ``` By default, the metrics can be scraped at the /metrics endpoint on port 9182. At this point, Prometheus can consume the metrics by adding the following scrape\$1config to the Prometheus configuration ``` scrape_configs: - job_name: "prometheus" static_configs: - targets: ['localhost:9090'] ... - job_name: "wmi_exporter" scrape_interval: 10s static_configs: - targets: [':9182', ':9182', ...] ``` Prometheus configuration is reloaded using ``` > ps aux | grep prometheus > kill HUP ``` A better and recommended way to add targets is to use a Custom Resource Definition called ServiceMonitor, which comes as part of the [Prometheus operator](https://github.com/prometheus-operator/kube-prometheus/releases)] that provides the definition for a ServiceMonitor Object and a controller that will activate the ServiceMonitors we define and automatically build the required Prometheus configuration. The ServiceMonitor, which declaratively specifies how groups of Kubernetes services should be monitored, is used to define an application you wish to scrape metrics from within Kubernetes. Within the ServiceMonitor we specify the Kubernetes labels that the operator can use to identify the Kubernetes Service which in turn identifies the Pods, that we wish to monitor. In order to leverage the ServiceMonitor, create an Endpoint object pointing to specific Windows targets, a headless service and a ServiceMontor for the Windows nodes. ``` apiVersion: v1 kind: Endpoints metadata: labels: k8s-app: wmiexporter name: wmiexporter namespace: kube-system subsets: - addresses: - ip: NODE-ONE-IP targetRef: kind: Node name: NODE-ONE-NAME - ip: NODE-TWO-IP targetRef: kind: Node name: NODE-TWO-NAME - ip: NODE-THREE-IP targetRef: kind: Node name: NODE-THREE-NAME ports: - name: http-metrics port: 9182 protocol: TCP --- apiVersion: v1 kind: Service ##Headless Service metadata: labels: k8s-app: wmiexporter name: wmiexporter namespace: kube-system spec: clusterIP: None ports: - name: http-metrics port: 9182 protocol: TCP targetPort: 9182 sessionAffinity: None type: ClusterIP --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor ##Custom ServiceMonitor Object metadata: labels: k8s-app: wmiexporter name: wmiexporter namespace: monitoring spec: endpoints: - interval: 30s port: http-metrics jobLabel: k8s-app namespaceSelector: matchNames: - kube-system selector: matchLabels: k8s-app: wmiexporter ``` For more details on the operator and the usage of ServiceMonitor, checkout the official [operator](https://github.com/prometheus-operator/kube-prometheus) documentation. Note that Prometheus does support dynamic target discovery using many [service discovery](https://prometheus.io/blog/2015/06/01/advanced-service-discovery/) options. # Windows Networking ## Windows Container Networking Overview Windows containers are fundamentally different than Linux containers. Linux containers use Linux constructs like namespaces, the union file system, and cgroups. On Windows, those constructs are abstracted from containerd by the [Host Compute Service (HCS)](https://github.com/microsoft/hcsshim). HCS acts as an API layer that sits above the container implementation on Windows. Windows containers also leverage the Host Network Service (HNS) that defines the network topology on a node. ![\[windows networking\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/windows-networking.png) From a networking perspective, HCS and HNS make Windows containers function like virtual machines. For example, each container has a virtual network adapter (vNIC) that is connected to a Hyper-V virtual switch (vSwitch) as shown in the diagram above. ## IP Address Management A node in Amazon EKS uses it’s Elastic Network Interface (ENI) to connect to an AWS VPC network. Presently, **only a single ENI per Windows worker node is supported**. The IP address management for Windows nodes is performed by [VPC Resource Controller](https://github.com/aws/amazon-vpc-resource-controller-k8s) which runs in control plane. More details about the workflow for IP address management of Windows nodes can be found [here](https://github.com/aws/amazon-vpc-resource-controller-k8s#windows-ipv4-address-management). The number of pods that a Windows worker node can support is dictated by the size of the node and the number of available IPv4 addresses. You can calculate the IPv4 address available on the node as below: + By default, only secondary IPv4 addresses are assigned to the ENI. In such a case: ``` Total IPv4 addresses available for Pods = Number of supported IPv4 addresses in the primary interface - 1 ``` We subtract one from the total count since one IPv4 addresses will be used as the primary address of the ENI and hence cannot be allocated to the Pods. + If the cluster has been configured for high pod density by enabling [prefix delegation feature](prefix-mode-win.md) then- ``` Total IPv4 addresses available for Pods = (Number of supported IPv4 addresses in the primary interface - 1) * 16 ``` Here, instead of allocating secondary IPv4 addresses, VPC Resource Controller will allocate `/28 prefixes` and therefore, the overall number of available IPv4 addresses will be boosted 16 times. Using the formula above, we can calculate max pods for an Windows worker noded based on a m5.large instance as below: + By default, when running in secondary IP mode- ``` 10 secondary IPv4 addresses per ENI - 1 = 9 available IPv4 addresses ``` + When using `prefix delegation`- ``` (10 secondary IPv4 addresses per ENI - 1) * 16 = 144 available IPv4 addresses ``` For more information on how many IP addresses an instance type can support, see [IP addresses per network interface per instance type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI). Another key consideration is the flow of network traffic. With Windows there is a risk of port exhaustion on nodes with more than 100 services. When this condition arises, the nodes will start throwing errors with the following message: **"Policy creation failed: hcnCreateLoadBalancer failed in Win32: The specified port already exists."** To address this issue, we leverage Direct Server Return (DSR). DSR is an implementation of asymmetric network load distribution. In other words, the request and response traffic use different network paths. This feature speeds up communication between pods and reduces the risk of port exhaustion. We therefore recommend enabling DSR on Windows nodes. DSR is enabled by default in Windows Server SAC EKS Optimized AMIs. For Windows Server 2019 LTSC EKS Optimized AMIs, you will need to enable it during instance provisioning using the script below and by using Windows Server 2019 Full or Core as the amiFamily in the `eksctl` nodeGroup. See [eksctl custom AMI](https://eksctl.io/usage/custom-ami-support/) for additional information. ``` nodeGroups: - name: windows-ng instanceType: c5.xlarge minSize: 1 volumeSize: 50 amiFamily: WindowsServer2019CoreContainer ssh: allow: false ``` In order to utilize DSR in Windows Server 2019 and above, you will need to specify the following [https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#load-balancing-and-services](https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#load-balancing-and-services) flags during instance startup. You can do this by adjusting the userdata script associated with the [self-managed node groups Launch Template](https://docs.aws.amazon.com/eks/latest/userguide/launch-windows-workers.html). ``` [string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS" [string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1' [string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName" (Get-Content $EKSBootstrapScriptFile).replace('"--proxy-mode=kernelspace",', '"--proxy-mode=kernelspace", "--feature-gates WinDSR=true", "--enable-dsr",') | Set-Content $EKSBootstrapScriptFile & $EKSBootstrapScriptFile -EKSClusterName "eks-windows" -APIServerEndpoint "https://" -Base64ClusterCA "" -DNSClusterIP "172.20.0.10" -KubeletExtraArgs "--node-labels=alpha.eksctl.io/cluster-name=eks-windows,alpha.eksctl.io/nodegroup-name=windows-ng-ltsc2019 --register-with-taints=" 3>&1 4>&1 5>&1 6>&1 ``` DSR enablement can be verified following the instructions in the [Microsoft Networking blog](https://techcommunity.microsoft.com/t5/networking-blog/direct-server-return-dsr-in-a-nutshell/ba-p/693710) and the [Windows Containers on AWS Lab](https://catalog.us-east-1.prod.workshops.aws/workshops/1de8014a-d598-4cb5-a119-801576492564/en-US/module1-eks/lab3-handling-mixed-clusters). ![\[dsr\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/dsr.png) If preserving your available IPv4 addresses and minimizing wastage is crucial for your subnet, it is generally recommended to avoid using prefix delegation mode as mentioned in [Prefix Mode for Windows - When to avoid](prefix-mode-win.md#windows-prefix-avoid). If using prefix delegation is still desired, you can take steps to optimize IPv4 address utilization in your subnet. See [Configuring Parameters for Prefix Delegation](prefix-mode-win.md#windows-network-conserve) for detailed instructions on how to fine-tune the IPv4 address request and allocation process. Adjusting these configurations can help you strike a balance between conserving IPv4 addresses and pod density benefits of prefix delegation. When using the default setting of assigning secondary IPv4 addresses, there are currently no supported configurations to manipulate how the VPC Resource Controller requests and allocates IPv4 addresses. More specifically, `minimum-ip-target` and `warm-ip-target` are only supported for prefix delegation mode. Also take note that in secondary IP mode, depending on the available IP addresses on the interface, the VPC Resource Controller will typically allocate 3 unused IPv4 addresses on the node on your behalf to maintain warm IPs for faster pod startup times. If you would like to minimize IP wastage of unused warm IP addresses, you could aim to schedule more pods on a given Windows node such that you use as much IP address capacity of the ENI as possible. More explicitly, you could avoid having warm unused IPs if all IP addresses on the ENI are already in use by the node and running pods. Another workaround to help you resolve constraints with IP address availability in your subnet(s) could be to explore [increasing your subnet size](https://docs.aws.amazon.com/vpc/latest/userguide/modify-subnets.html) or separating your Windows nodes into their own dedicated subnets. Additionally, it’s important to note that IPv6 is not supported on Windows nodes at the moment. ## Container Network Interface (CNI) options The AWSVPC CNI is the de facto CNI plugin for Windows and Linux worker nodes. While the AWSVPC CNI satisfies the needs of many customers, still there may be times when you need to consider alternatives like an overlay network to avoid IP exhaustion. In these cases, the Calico CNI can be used in place of the AWSVPC CNI. [Project Calico](https://www.projectcalico.org/) is open source software that was developed by [Tigera](https://www.tigera.io/). That software includes a CNI that works with EKS. Instructions for installing Calico CNI in EKS can be found on the [Project Calico EKS installation](https://docs.projectcalico.org/getting-started/kubernetes/managed-public-cloud/eks) page. ## Network Polices It is considered a best practice to change from the default mode of open communication between pods on your Kubernetes cluster to limiting access based on network polices. The open source [Project Calico](https://www.tigera.io/tigera-products/calico/) has strong support for network polices that work with both Linux and Windows nodes. This feature is separate and not dependent on using the Calico CNI. We therefore recommend installing Calico and using it for network policy management. Instructions for installing Calico in EKS can be found on the [Installing Calico on Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/calico.html) page. In addition, the advice provided in the [Amazon EKS Best Practices Guide for Security - Network Section](https://docs.aws.amazon.com/eks/latest/best-practices/network-security.html) applies equally to EKS clusters with Windows worker nodes, however, some features like "Security Groups for Pods" are not supported by Windows at this time. # Avoiding OOM errors Memory and Systems Management Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. The net effect is that Windows won’t reach out of memory conditions the same way Linux does. Processes will page to disk instead of being subject to out of memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance. ## Reserving system and kubelet memory Different from Linux where `--kubelet-reserve` **capture** resource reservation for kubernetes system daemons like kubelet, container runtime, etc; and `--system-reserve` **capture** resource reservation for OS system daemons like sshd, udev and etc. On **Windows** these flags do not **capture** and **set** memory limits on **kubelet** or **processes** running on the node. However, you can combine these flags to manage **NodeAllocatable** to reduce Capacity on the node with Pod manifest **memory resource limit** to control memory allocation per pod. Using this strategy you have a better control of memory allocation as well as a mechanism to minimize out-of-memory (OOM) on Windows nodes. On Windows nodes, a best practice is to reserve at least 2GB of memory for the OS and process. Use `--kubelet-reserve` and/or `--system-reserve` to reduce NodeAllocatable. Following the [Amazon EKS Self-managed Windows nodes](https://docs.aws.amazon.com/eks/latest/userguide/launch-windows-workers.html) documentation, use the CloudFormation template to launch a new Windows node group with customizations to kubelet configuration. The CloudFormation has an element called `BootstrapArguments` which is the same as `KubeletExtraArgs`. Use with the following flags and values: ``` --kube-reserved memory=0.5Gi,ephemeral-storage=1Gi --system-reserved memory=1.5Gi,ephemeral-storage=1Gi --eviction-hard memory.available<200Mi,nodefs.available<10%" ``` If eksctl is the deployment tool, check the following documentation to customize the kubelet configuration https://eksctl.io/usage/customizing-the-kubelet/ ## Windows container memory requirements As per [Microsoft documentation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/system-requirements), a Windows Server base image for NANO requires at least 30MB, whereas Server Core requires 45MB. These numbers grow as you add Windows components such as the .NET Framework, Web Services as IIS and applications. It is essential for you to know the minimum amount of memory required by your Windows container image, i.e. the base image plus its application layers, and set it as the container’s resources/requests in the pod specification. You should also set a limit to avoid pods to consume all the available node memory in case of an application issue. In the example below, when the Kubernetes scheduler tries to place a pod on a node, the pod’s requests are used to determine which node has sufficient resources available for scheduling. ``` spec: - name: iis image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 resources: limits: cpu: 1 memory: 800Mi requests: cpu: .1 memory: 128Mi ``` ## Conclusion Using this approach minimizes the risks of memory exhaustion but does not prevent it happen. Using Amazon CloudWatch Metrics, you can set up alerts and remediations in case of memory exhaustion occurs. # Patching Windows Servers and Containers Infrastructure Management Patching Windows Server is a standard management task for Windows Administrators. This can be accomplished using different tools like Amazon System Manager - Patch Manager, WSUS, System Center Configuration Manager, and many others. However, Windows nodes in an Amazon EKS cluster should not be treated as an ordinary Windows servers. They should be treated as an immutable server. Simply put, avoid updating an existing node, just launch a new one based on an new updated AMI. Using [EC2 Image Builder](https://aws.amazon.com/image-builder/) you can automate AMIs build, by creating recipes and adding components. The following example shows **components**, which can be pre-existing ones built by AWS (Amazon-managed) as well as the components you create (Owned by me). Pay close attention to the Amazon-managed component called **update-windows**, this updates Windows Server before generating the AMI through the EC2 Image Builder pipeline. ![\[associated components\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/associated-components.png) EC2 Image Builder allows you to build AMI’s based off Amazon Managed Public AMIs and customize them to meet your business requirements. You can then associate those AMIs with Launch Templates which allows you to link a new AMI to the Auto Scaling Group created by the EKS Nodegroup. After that is complete, you can begin terminating the existing Windows Nodes and new ones will be launched based on the new updated AMI. ## Pushing and pulling Windows images Amazon publishes EKS optimized AMIs that include two cached Windows container images. ``` mcr.microsoft.com/windows/servercore mcr.microsoft.com/windows/nanoserver ``` ![\[images\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/images.png) Cached images are updated following the updates on the main OS. When Microsoft releases a new Windows update that directly affects the Windows container base image, the update will be launched as an ordinary Windows Update on the main OS. Keeping the environment up-to-date offers a more secure environment at the Node and Container level. The size of a Windows container image influences push/pull operations which can lead to slow container startup times. [Caching Windows container images](https://aws.amazon.com/blogs/containers/speeding-up-windows-container-launch-times-with-ec2-image-builder-and-image-cache-strategy/) allows the expensive I/O operations (file extraction) to occur on the AMI build creation instead of the container launch. As a result, all the necessary image layers will be extracted on the AMI and will be ready to be used, speeding up the time a Windows container launches and can start accepting traffic. During a push operation, only the layers that compose your image are uploaded to the repository. The following example shows that on the Amazon ECR the **fluentd-windows-sac2004** images have only **390.18MB**. This is the amount of upload that happened during the push operation. The following example shows a [fluentd Windows ltsc](https://github.com/fluent/fluentd-docker-image/blob/master/v1.14/windows-ltsc2019/Dockerfile) image pushed to an Amazon ECR repository. The size of the layer stored in ECR is **533.05MB**. ![\[ecr image\]](http://docs.aws.amazon.com/eks/latest/best-practices/images/windows/ecr-image.png) The output below from `docker image ls` , the size of the fluentd v1.14-windows-ltsc2019-1 is **6.96GB** on disk, but that doesn’t mean it downloaded and extracted that amount of data. In practice, during the pull operation only the **compressed 533.05MB** will be downloaded and extracted. ``` REPOSITORY TAG IMAGE ID CREATED SIZE 111122223333.dkr.ecr.us-east-1.amazonaws.com/fluentd-windows-coreltsc latest 721afca2c725 7 weeks ago 6.96GB fluent/fluentd v1.14-windows-ltsc2019-1 721afca2c725 7 weeks ago 6.96GB amazonaws.com/eks/pause-windows latest 6392f69ae6e7 10 months ago 255MB ``` The size column shows the overall size of image, 6.96GB. Breaking it down: + Windows Server Core 2019 LTSC Base image = 5.74GB + Fluentd Uncompressed Base Image = 6.96GB + Difference on disk = 1.2GB + Fluentd [compressed final image ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-info.html) = 533.05MB The base image already exists on the local disk, resulting in the total amount on disk being 1.2GB additional. The next time you see the amount of GBs in the size column, don’t worry too much, likely more than 70% is already on disk as a cached container image. ## Reference [Speeding up Windows container launch times with EC2 Image builder and image cache strategy](https://aws.amazon.com/blogs/containers/speeding-up-windows-container-launch-times-with-ec2-image-builder-and-image-cache-strategy/) # Running Heterogeneous workloads Scheduling Kubernetes has support for heterogeneous clusters where you can have a mixture of Linux and Windows nodes in the same cluster. Within that cluster, you can have a mixture of Pods that run on Linux and Pods that run on Windows. You can even run multiple versions of Windows in the same cluster. However, there are several factors (as mentioned below) that will need to be accounted for when making this decision. ## Assigning PODs to Nodes Best practices In order to keep Linux and Windows workloads on their respective OS-specific nodes, you need to use some combination of node selectors and taints/tolerations. The main goal of scheduling workloads in a heterogeneous environment is to avoid breaking compatibility for existing Linux workloads. ## Ensuring OS-specific workloads land on the appropriate container host Users can ensure Windows containers can be scheduled on the appropriate host using nodeSelectors. All Kubernetes nodes today have the following default labels: ``` kubernetes.io/os = [windows|linux] kubernetes.io/arch = [amd64|arm64|...] ``` If a Pod specification does not include a nodeSelector like `"kubernetes.io/os": windows`, the Pod may be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. In Enterprise environments, it’s not uncommon to have a large number of pre-existing deployments for Linux containers, as well as an ecosystem of off-the-shelf configurations, like Helm charts. In these situations, you may be hesitant to make changes to a deployment’s nodeSelectors. **The alternative is to use Taints**. For example: `--register-with-taints='os=windows:NoSchedule'` If you are using EKS, eksctl offers ways to apply taints through clusterConfig: ``` NodeGroups: - name: windows-ng amiFamily: WindowsServer2022FullContainer ... labels: nodeclass: windows2022 taints: os: "windows:NoSchedule" ``` Adding a taint to all Windows nodes, the scheduler will not schedule pods on those nodes unless they tolerate the taint. Pod manifest example: ``` nodeSelector: kubernetes.io/os: windows tolerations: - key: "os" operator: "Equal" value: "windows" effect: "NoSchedule" ``` ## Handling multiple Windows build in the same cluster The Windows container base image used by each pod must match the same kernel build version as the node. If you want to use multiple Windows Server builds in the same cluster, then you should set additional node labels, nodeSelectors or leverage a label called **windows-build**. Kubernetes 1.17 automatically adds a new label **node.kubernetes.io/windows-build** to simplify the management of multiple Windows build in the same cluster. If you’re running an older version, then it’s recommended to add this label manually to Windows nodes. This label reflects the Windows major, minor, and build number that need to match for compatibility. Below are values used today for each Windows Server version. It’s important to note that Windows Server is moving to the Long-Term Servicing Channel (LTSC) as the primary release channel. The Windows Server Semi-Annual Channel (SAC) was retired on August 9, 2022. There will be no future SAC releases of Windows Server. | Product Name | Build Number(s) | | --- | --- | | Server full 2022 LTSC | 10.0.20348 | | Server core 2019 LTSC | 10.0.17763 | It is possible to check the OS build version through the following command: ``` kubectl get nodes -o wide ``` The KERNEL-VERSION output matches the Windows OS build version. ``` NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-10-2-235.ec2.internal Ready 23m v1.24.7-eks-fb459a0 10.10.2.235 3.236.30.157 Windows Server 2022 Datacenter 10.0.20348.1607 containerd://1.6.6 ip-10-10-31-27.ec2.internal Ready 23m v1.24.7-eks-fb459a0 10.10.31.27 44.204.218.24 Windows Server 2019 Datacenter 10.0.17763.4131 containerd://1.6.6 ip-10-10-7-54.ec2.internal Ready 31m v1.24.11-eks-a59e1f0 10.10.7.54 3.227.8.172 Amazon Linux 2 5.10.173-154.642.amzn2.x86_64 containerd://1.6.19 ``` The example below applies an additional nodeSelector to the pod manifest in order to match the correct Windows-build version when running different Windows node groups OS versions. ``` nodeSelector: kubernetes.io/os: windows node.kubernetes.io/windows-build: '10.0.20348' tolerations: - key: "os" operator: "Equal" value: "windows" effect: "NoSchedule" ``` ## Simplifying NodeSelector and Toleration in Pod manifests using RuntimeClass You can also make use of RuntimeClass to simplify the process of using taints and tolerations. This can be accomplished by creating a RuntimeClass object which is used to encapsulate these taints and tolerations. Create a RuntimeClass by running the following manifest: ``` apiVersion: node.k8s.io/v1beta1 kind: RuntimeClass metadata: name: windows-2022 handler: 'docker' scheduling: nodeSelector: kubernetes.io/os: 'windows' kubernetes.io/arch: 'amd64' node.kubernetes.io/windows-build: '10.0.20348' tolerations: - effect: NoSchedule key: os operator: Equal value: "windows" ``` Once the Runtimeclass is created, assign it using as a Spec on the Pod manifest: ``` apiVersion: apps/v1 kind: Deployment metadata: name: iis-2022 labels: app: iis-2022 spec: replicas: 1 template: metadata: name: iis-2022 labels: app: iis-2022 spec: runtimeClassName: windows-2022 containers: - name: iis ``` ## Managed Node Group Support To help customers run their Windows applications in a more streamlined manner, AWS launched the support for Amazon [EKS Managed Node Group (MNG) support for Windows containers](https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-eks-automated-provisioning-lifecycle-management-windows-containers/) on December 15, 2022. To help align operations teams, [Windows MNGs](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html) are enabled using the same workflows and tools as [Linux MNGs](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html). Full and core AMI (Amazon Machine Image) family versions of Windows Server 2019 and 2022 are supported. Following AMI families are supported for Managed Node Groups(MNG)s. | AMI Family | | --- | | WINDOWS\$1CORE\$12019\$1x86\$164 | | WINDOWS\$1FULL\$12019\$1x86\$164 | | WINDOWS\$1CORE\$12022\$1x86\$164 | | WINDOWS\$1FULL\$12022\$1x86\$164 | ## Additional documentations AWS Official Documentation: https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html To better understand how Pod Networking (CNI) works, check the following link: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html AWS Blog on Deploying Managed Node Group for Windows on EKS: https://aws.amazon.com/blogs/containers/deploying-amazon-eks-windows-managed-node-groups/ # Pod Security Contexts Pod Security for Windows Containers **Pod Security Policies (PSP)** and **Pod Security Standards (PSS)** are two main ways of enforcing security in Kubernetes. Note that PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25 and Pod Security Standard (PSS) is the Kubernetes recommended approach for enforcing security going forward. A Pod Security Policy (PSP) is a native solution in Kubernetes to implement security policies. PSP is a cluster-level resource that controls security-sensitive aspects of the Pod specification. Using Pod Security Policy you can define a set of conditions that Pods must meet to be accepted by the cluster. The PSP feature has been available from the early days of Kubernetes and is designed to block misconfigured pods from being created on a given cluster. For more information on Pod Security Policies please reference the Kubernetes [documentation](https://kubernetes.io/docs/concepts/policy/pod-security-policy/). According to the [Kubernetes deprecation policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/), older versions will stop getting support nine months after the deprecation of the feature. On the other hand, Pod Security Standards (PSS) which is the recommended security approach and typically implemented using Security Contexts are defined as part of the Pod and container specifications in the Pod manifest. PSS is the official standard that the Kubernetes project team has defined to address the security-related best practices for Pods. It defines policies such as baseline (minimally restrictive, default), privileged (unrestrictive) and restricted (most restrictive). We recommend starting with the baseline profile. PSS baseline profile provides a solid balance between security and potential friction, requiring a minimal list of exceptions, it serves as a good starting point for workload security. If you are currently using PSPs we recommend switching to PSS. More details on the PSS policies can be found in the Kubernetes [documentation](https://kubernetes.io/docs/concepts/security/pod-security-standards/). These policies can be enforced with several tools including those from [OPA](https://www.openpolicyagent.org/) and [Kyverno](https://kyverno.io/). For example, Kyverno provides the full collection of PSS policies [here](https://kyverno.io/policies/pod-security/). Security context settings allow one to give privileges to select processes, use program profiles to restrict capabilities to individual programs, allow privilege escalation, filter system calls, among other things. Windows pods in Kubernetes have some limitations and differentiators from standard Linux-based workloads when it comes to security contexts. Windows uses a Job object per container with a system namespace filter to contain all processes in a container and provide logical isolation from the host. There is no way to run a Windows container without the namespace filtering in place. This means that system privileges cannot be asserted in the context of the host, and thus privileged containers are not available on Windows. The following `windowsOptions` are the only documented [Windows Security Context options](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#windowssecuritycontextoptions-v1-core) while the rest are general [Security Context options](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#securitycontext-v1-core) For a list of security context attributes that are supported in Windows vs linux, please refer to the official documentation [here](https://kubernetes.io/docs/setup/production-environment/windows/_print/#v1-container). The Pod specific settings are applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. For example, runAsUserName setting for Pods and containers which is a Windows option is a rough equivalent of the Linux-specific runAsUser setting and in the following manifest, the pod specific security context is applied to all containers ``` apiVersion: v1 kind: Pod metadata: name: run-as-username-pod-demo spec: securityContext: windowsOptions: runAsUserName: "ContainerUser" containers: - name: run-as-username-demo nodeSelector: kubernetes.io/os: windows ``` Whereas in the following, the container level security context overrides the pod level security context. ``` apiVersion: v1 kind: Pod metadata: name: run-as-username-container-demo spec: securityContext: windowsOptions: runAsUserName: "ContainerUser" containers: - name: run-as-username-demo .. securityContext: windowsOptions: runAsUserName: "ContainerAdministrator" nodeSelector: kubernetes.io/os: windows ``` Examples of acceptable values for the runAsUserName field: ContainerAdministrator, ContainerUser, NT AUTHORITY\$1NETWORK SERVICE, NT AUTHORITY\$1LOCAL SERVICE It is generally a good idea to run your containers with ContainerUser for Windows pods. The users are not shared between the container and host but the ContainerAdministrator does have additional privileges with in the container. Note that, there are username [limitations](https://kubernetes.io/docs/tasks/configure-pod-container/configure-runasusername/#windows-username-limitations) to be aware of. A good example of when to use ContainerAdministrator is to set PATH. You can use the USER directive to do that, like so: ``` USER ContainerAdministrator RUN setx /M PATH "%PATH%;C:/your/path" USER ContainerUser ``` Also note that, secrets are written in clear text on the node’s volume (as compared to tmpfs/in-memory on linux). This means you have to do two things + Use file ACLs to secure the secrets file location + Use volume-level encryption using [BitLocker](https://docs.microsoft.com/en-us/windows/security/information-protection/bitlocker/bitlocker-how-to-deploy-on-windows-server) # Persistent storage options Storage Options ## What is an in-tree vs. out-of-tree volume plugin? Before the introduction of the Container Storage Interface (CSI), all volume plugins were in-tree meaning they were built, linked, compiled, and shipped with the core Kubernetes binaries and extend the core Kubernetes API. This meant that adding a new storage system to Kubernetes (a volume plugin) required checking code into the core Kubernetes code repository. Out-of-tree volume plugins are developed independently of the Kubernetes code base, and are deployed (installed) on Kubernetes clusters as extensions. This gives vendors the ability to update drivers out-of-band, i.e. separately from the Kubernetes release cycle. This is largely possible because Kubernetes has created a storage interface or CSI that provides vendors a standard way of interfacing with k8s. You can check more about Amazon Elastic Kubernetes Services (EKS) storage classes and CSI Drivers on https://docs.aws.amazon.com/eks/latest/userguide/storage.html ## In-tree Volume Plugin for Windows Kubernetes volumes enable applications, with data persistence requirements, to be deployed on Kubernetes. The management of persistent volumes consists of provisioning/de-provisioning/resizing of volumes, attaching/detaching a volume to/from a Kubernetes node, and mounting/dismounting a volume to/from individual containers in a pod. The code for implementing these volume management actions for a specific storage back-end or protocol is shipped in the form of a Kubernetes volume plugin **(In-tree Volume Plugins)**. On Amazon Elastic Kubernetes Services (EKS) the following class of Kubernetes volume plugins are supported on Windows: *In-tree Volume Plugin:* [awsElasticBlockStore](https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore) In order to use In-tree volume plugin on Windows nodes, it is necessary to create an additional StorageClass to use NTFS as the fsType. On EKS, the default StorageClass uses ext4 as the default fsType. A StorageClass provides a way for administrators to describe the "classes" of storage they offer. Different classes might map to quality-of-service levels, backup policies, or arbitrary policies determined by the cluster administrators. Kubernetes is unopinionated about what classes represent. This concept is sometimes called "profiles" in other storage systems. You can check it by running the following command: ``` kubectl describe storageclass gp2 ``` Output: ``` Name: gp2 IsDefaultClass: Yes Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClas ","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType" "ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"} ,storageclass.kubernetes.io/is-default-class=true Provisioner: kubernetes.io/aws-ebs Parameters: fsType=ext4,type=gp2 AllowVolumeExpansion: MountOptions: ReclaimPolicy: Delete VolumeBindingMode: WaitForFirstConsumer Events: ``` To create the new StorageClass to support **NTFS**, use the following manifest: ``` kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: gp2-windows provisioner: kubernetes.io/aws-ebs parameters: type: gp2 fsType: ntfs volumeBindingMode: WaitForFirstConsumer ``` Create the StorageClass by running the following command: ``` kubectl apply -f NTFSStorageClass.yaml ``` The next step is to create a Persistent Volume Claim (PVC). A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using PVC. It is a resource in the cluster just like a node is a cluster resource. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system. A PersistentVolumeClaim (PVC) is a request for storage by a user. Claims can request specific size and access modes (e.g., they can be mounted ReadWriteOnce, ReadOnlyMany or ReadWriteMany). Users need PersistentVolumes with different attributes, such as performance, for different use cases. Cluster administrators need to be able to offer a variety of PersistentVolumes that differ in more ways than just size and access modes, without exposing users to the details of how those volumes are implemented. For these needs, there is the StorageClass resource. In the example below, the PVC has been created within the namespace windows. ``` apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ebs-windows-pv-claim namespace: windows spec: accessModes: - ReadWriteOnce storageClassName: gp2-windows resources: requests: storage: 1Gi ``` Create the PVC by running the following command: ``` kubectl apply -f persistent-volume-claim.yaml ``` The following manifest creates a Windows Pod, setup the VolumeMount as `C:\Data` and uses the PVC as the attached storage on `C:\Data`. ``` apiVersion: apps/v1 kind: Deployment metadata: name: windows-server-ltsc2019 namespace: windows spec: selector: matchLabels: app: windows-server-ltsc2019 tier: backend track: stable replicas: 1 template: metadata: labels: app: windows-server-ltsc2019 tier: backend track: stable spec: containers: - name: windows-server-ltsc2019 image: mcr.microsoft.com/windows/servercore:ltsc2019 ports: - name: http containerPort: 80 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "C:\\data" name: test-volume volumes: - name: test-volume persistentVolumeClaim: claimName: ebs-windows-pv-claim nodeSelector: kubernetes.io/os: windows node.kubernetes.io/windows-build: '10.0.17763' ``` Test the results by accessing the Windows pod via PowerShell: ``` kubectl exec -it podname powershell -n windows ``` Inside the Windows Pod, run: `ls` Output: ``` PS C:\> ls Directory: C:\ Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 3/8/2021 1:54 PM data d----- 3/8/2021 3:37 PM inetpub d-r--- 1/9/2021 7:26 AM Program Files d----- 1/9/2021 7:18 AM Program Files (x86) d-r--- 1/9/2021 7:28 AM Users d----- 3/8/2021 3:36 PM var d----- 3/8/2021 3:36 PM Windows -a---- 12/7/2019 4:20 AM 5510 License.txt ``` The **data directory** is provided by the EBS volume. ## Out-of-tree for Windows Code associated with CSI plugins ship as out-of-tree scripts and binaries that are typically distributed as container images and deployed using standard Kubernetes constructs like DaemonSets and StatefulSets. CSI plugins handle a wide range of volume management actions in Kubernetes. CSI plugins typically consist of node plugins (that run on each node as a DaemonSet) and controller plugins. CSI node plugins (especially those associated with persistent volumes exposed as either block devices or over a shared file-system) need to perform various privileged operations like scanning of disk devices, mounting of file systems, etc. These operations differ for each host operating system. For Linux worker nodes, containerized CSI node plugins are typically deployed as privileged containers. For Windows worker nodes, privileged operations for containerized CSI node plugins is supported using [csi-proxy](https://github.com/kubernetes-csi/csi-proxy), a community-managed, stand-alone binary that needs to be pre-installed on each Windows node. [The Amazon EKS Optimized Windows AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-windows-ami.html) includes CSI-proxy starting from April 2022. Customers can use the [SMB CSI Driver](https://github.com/kubernetes-csi/csi-driver-smb) on Windows nodes to access [Amazon FSx for Windows File Server](https://aws.amazon.com/fsx/windows/), [Amazon FSx for NetApp ONTAP SMB Shares](https://aws.amazon.com/fsx/netapp-ontap/), and/or [AWS Storage Gateway — File Gateway](https://aws.amazon.com/storagegateway/file/). The following [blog](https://aws.amazon.com/blogs/modernizing-with-aws/using-smb-csi-driver-on-amazon-eks-windows-nodes/) has implementation details on how to setup SMB CSI Driver to use Amazon FSx for Windows File Server as a persistent storage for Windows Pods. ## Amazon FSx for Windows File Server An option is to use Amazon FSx for Windows File Server through an SMB feature called [SMB Global Mapping](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/persistent-storage) which makes it possible to mount a SMB share on the host, then pass directories on that share into a container. The container doesn’t need to be configured with a specific server, share, username or password - that’s all handled on the host instead. The container will work the same as if it had local storage. The SMB Global Mapping is transparent to the orchestrator, and it is mounted through HostPath which can **imply in secure concerns**. In the example below, the path `G:\Directory\app-state` is an SMB share on the Windows Node. ``` apiVersion: v1 kind: Pod metadata: name: test-fsx spec: containers: - name: test-fsx image: mcr.microsoft.com/windows/servercore:ltsc2019 command: - powershell.exe - -command - "Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe' -OutFile 'C:\\ServiceMonitor.exe'; echo '

Hello EKS!!!

' > C:\\inetpub\\wwwroot\\default.html; C:\\ServiceMonitor.exe 'w3svc'; " volumeMounts: - mountPath: C:\dotnetapp\app-state name: test-mount volumes: - name: test-mount hostPath: path: G:\Directory\app-state type: Directory nodeSelector: beta.kubernetes.io/os: windows beta.kubernetes.io/arch: amd64 ``` The following [blog](https://aws.amazon.com/blogs/containers/using-amazon-fsx-for-windows-file-server-on-eks-windows-containers/) has implementation details on how to setup Amazon FSx for Windows File Server as a persistent storage for Windows Pods. # Hardening Windows container images Hardening Windows containers images Are you hardening your Windows container images? Over the years, I’ve worked with customers globally to help them migrate legacy workloads to containers, particularly Windows workloads. With more than 20 years of experience, I’ve seen organizations dedicate substantial effort and resources to hardening their Windows Servers, implementing everything from CIS Benchmarks to runtime antivirus protection to safeguard sensitive data. However, a concerning trend has emerged. As these highly secure virtual machines are modernized into containers, many critical hardening practices are being overlooked. Windows security best practices, from the base image (OS) to web services such as IIS, are often neglected, with most of the focus placed solely on securing the container host. It’s vital to recognize that while containers operate in isolated namespaces, they still share kernel primitives with the host. Attackers are typically more interested in lateral movement rather than targeting the container host directly, allowing them to exploit weak container security settings and access sensitive data. The goal of the documentations is to highlight a few essential security settings you should implement specifically for Windows containers hosting `ASP.NET` websites on IIS. We’ll focus on four key areas: + Account security policies + Audit policies + IIS security best practices + Principle of least privilege We’ll start by delving into why each of these security configurations is vital for protecting your Windows containers, examining the specific risks they mitigate and the security benefits they provide. Next, we’ll walk through a code snippet that demonstrates how to implement these configurations correctly in your Dockerfile, ensuring your container is hardened against potential threats. Finally, we’ll break down each setting in detail, offering a comprehensive explanation of its function, impact on container security, and how it contributes to safeguarding your applications. This approach will not only show you how to apply these best practices but also give you the insight to understand why they are essential for maintaining a robust security posture in containerized environments. ## 1. Configure Account Policies (Password or Lockout) using Local Security Policies and Registry Windows Server Core is a minimal installation option that is available as part of the [EKS Optimized Windows AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-windows-ami.html). Configuring Account Policies (Password or Lockout) using Local Security Policies and the Registry strengthens system security by enforcing robust password and lockout rules. These policies require users to create strong passwords with a defined minimum length and complexity, protecting against common password-related attacks. By setting a maximum password age, users are prompted to regularly update their passwords, reducing the likelihood of compromised credentials. Lockout policies add an extra layer of protection by temporarily locking accounts after a specified number of failed login attempts, helping to prevent brute-force attacks. Configuring these settings via the Windows Registry allows administrators to enforce these security measures at the system level, ensuring uniformity and compliance throughout the organization. Applying these Account Policies in a Windows Container is essential for maintaining security consistency, even though containers are often ephemeral and intended for isolated workloads: ### Security Consistency + Compliance: Enforcing consistent password policies and lockout rules in containers helps maintain security compliance, especially in environments that require strict access controls (e.g., regulatory compliance such as HIPAA, PCI-DSS). + Hardened Containers: Applying these settings ensures that your Windows container is hardened against unauthorized access or password-based attacks, aligning the security posture of your container with the broader system security policies. ### Protection Against Brute Force Attacks + Account Lockout: These settings help defend against brute force login attempts by locking accounts after a specific number of failed login attempts. This prevents attackers from trying an unlimited number of passwords. + Password Complexity: Requiring complex passwords with sufficient length reduces the likelihood of weak passwords being exploited, even in isolated containerized environments. ### Multi-User Scenarios + If your containerized application is designed to handle multiple users or requires user authentication, enforcing password policies ensures that user accounts within the container adhere to strict security rules, limiting access to only authorized users. ### Persistent Windows Containers + While containers are generally considered ephemeral, certain Windows containers can run long-term services or handle user management, making it important to enforce proper security policies similar to a regular Windows server. ### Consistency in Hybrid Environments + If you are running both virtual machines and containers in your infrastructure, applying the same security policies (e.g., password/lockout policies) across all environments ensures uniform security standards, simplifying governance and management. In summary, applying these account policies within Windows containers ensures that your containers are not a weak point in your security strategy, protecting against password attacks and enforcing consistency across your entire environment. Dockerfile: ``` # Configure account policies for password complexity and lockout RUN powershell -Command \ "Write-Output 'Configuring Account Policies (Password/Lockout)...'; \ NET ACCOUNTS /MINPWLEN:14 /MAXPWAGE:60 /MINPWAGE:14 /LOCKOUTTHRESHOLD:5 ``` **Explanation:** This section configures account policies for password and lockout settings via the Windows Registry. These policies help enforce security by controlling password requirements and account lockout thresholds. 1. **MinimumPasswordLength (MINPWLEN) = 14** This setting defines the minimum number of characters for a password. The range is 0-14 characters; the default is six characters. 1. **MaximumPasswordAge (MAXPWAGE) = 60** This setting defines the maximum number of days that a password is valid. No limit is specified by using UNLIMITED. /MAXPWAGE can’t be less than /MINPWAGE. The range is 1-999; the default is 90 days 1. **Lockout Threshold (LOCKOUTTHRESHOLD) = 5** This setting defines the threshold for failed login attempts. After 5 incorrect attempts, the account will be locked. These settings help improve password security and prevent brute force attacks by enforcing strong password policies and locking out accounts after a certain number of failed login attempts. ## 2. Audit policies Audit Policies are important for Windows Containers because they provide critical visibility into security events, such as login attempts and privilege use, helping to detect unauthorized access, monitor user activity, and ensure compliance with regulatory standards. Even in the ephemeral nature of containers, audit logs are essential for incident investigation, proactive threat detection, and maintaining a consistent security posture across containerized environments. **Security Monitoring and Compliance:** + Track User Activities: Audit policies allow administrators to monitor user activities, such as login attempts and privilege use, within the container. This is critical for detecting unauthorized access or suspicious behavior. + Regulatory Compliance: Many organizations are required to log security events for compliance with regulations such as HIPAA, PCI-DSS, and GDPR. Enabling audit policies in containers ensures you meet these requirements, even in containerized environments. **Incident Investigation:** + Forensics and Analysis: If a containerized application or service is compromised, audit logs can provide valuable insights for post-incident analysis. They help security teams trace the actions taken by attackers or identify how a breach occurred. + Real-time Detection: Audit logs allow administrators to set up real-time alerts for critical events (e.g., failed login attempts, privilege escalations). This proactive monitoring helps detect attacks early and enables faster response times. **Consistency Across Environments:** + Uniform Security Posture: By applying audit policies in containers via the registry, you ensure consistent security practices across both containerized and non-containerized environments. This avoids containers becoming a blind spot for security monitoring. + Visibility in Hybrid Environments: For organizations running both traditional Windows servers and containers, auditing policies provide similar visibility and control across all platforms, making management easier and more effective. **Tracking Privileged Operations:** + Privilege Use Auditing: In container environments where applications run with elevated privileges or where administrative tasks are performed, auditing privileged operations ensures accountability. You can log who accessed sensitive resources or performed critical tasks inside the container. + Prevent Abuse of Privileges: By monitoring privilege use, you can detect when unauthorized users try to elevate their privileges or access restricted areas within the container, which helps prevent internal or external attacks. **Detecting Unauthorized Access Attempts:** + Failed Logon Attempts: Enabling audit policies for failed login attempts helps identify brute-force attacks or unauthorized attempts to access containerized applications. This provides visibility into who is trying to gain access to the system and how often. + Account Lockout Monitoring: Auditing account lockout events allows administrators to detect and investigate potential lockouts caused by suspicious or malicious activity. **Persistent Security Even in Ephemeral Environments:** + Ephemeral Yet Secure: While containers are ephemeral, meaning they can be deleted and recreated frequently, auditing still plays a key role in ensuring that security events are captured while the container is running. This ensures that critical security events are logged for the duration of the container’s lifecycle. **Centralized Logging:** + Forwarding Logs to Centralized Systems: Containers can be integrated with centralized logging systems (e.g., ELK stack, AWS CloudWatch) to capture audit logs from multiple container instances. This allows for better analysis and correlation of security events across your infrastructure. Dockerfile: ``` # Configure audit policies for logging security events RUN powershell -Command \ "Write-Host 'Configuring Audit Policy..'; \ Set-ItemProperty -Path 'HKLM:\\SYSTEM\\CurrentControlSet\\Control\\Lsa' -Name 'SCENoApplyLegacyAuditPolicy' -Value 0; \ auditpol /set /category:"Logon/Logoff" /subcategory:"Logon" /failure:enable # Creates STDOUT on Windows Containers (check GitHub LogMonitor:: https://github.com/microsoft/windows-container-tools/blob/main/LogMonitor/README.md) COPY LogMonitor.exe LogMonitorConfig.json 'C:\\LogMonitor\\' WORKDIR /LogMonitor ``` **Explanation:** This section configures audit policies by using registry modifications. Audit policies control what security events are logged by Windows, which helps in monitoring and detecting unauthorized access attempts. 1. **SCENoApplyLegacyAuditPolicy = 0** This disables the legacy audit policy format, enabling more granular auditing policies introduced in later versions of Windows. This is important for modern audit configurations. 1. **Auditpol Subcategory: "Logon" ** This setting enables auditing for both success and failure logon events. The value 3 means that Windows will log both successful and failed logon attempts. This helps in monitoring who is accessing the system and catching failed login attempts. These audit policies are critical for security monitoring and compliance, as they provide detailed logs of important security events such as login attempts and the use of privileged operations. ## 3. IIS Security best practices for Windows containers Implementing IIS best practices in Windows Containers is important for several reasons, ensuring that your applications are secure, high performance, and scalable. Although containers provide isolation and a lightweight environment, they still require proper configuration to avoid vulnerabilities and operational issues. Here’s why following best practices for IIS in Windows Containers is crucial: **Security** + Preventing Common Vulnerabilities: IIS is often a target for attacks such as cross-site scripting (XSS), clickjacking, and information disclosure. Implementing security headers (e.g., X-Content-Type-Options, X-Frame-Options, and Strict-Transport-Security) helps protect your application from these threats. + Isolation Isn’t Enough: Containers are isolated, but a misconfigured IIS instance can expose sensitive information, such as server version details, directory listings, or unencrypted communications. By disabling features such as directory browsing and removing the IIS version header, you minimize the attack surface. + Encryption and HTTPS: Best practices, such as enforcing HTTPS-only connections, ensure that data in transit is encrypted, protecting sensitive information from being intercepted. **Performance** + Efficient Resource Usage: IIS best practices such as enabling dynamic and static compression reduce bandwidth usage and improve load times. These optimizations are especially important in containerized environments, where resources are shared across containers and the host system. + Optimized Logging: Properly configuring logging (e.g., including the X-Forwarded-For header) ensures that you can trace client activity while minimizing unnecessary logging overhead. This helps you gather relevant data for troubleshooting without degrading performance. **Scalability and Maintainability** + Consistency Across Environments: By following best practices, you ensure that your IIS configuration is consistent across multiple container instances. This simplifies scaling and makes sure that when new containers are deployed, they adhere to the same security and performance guidelines. + Automated Configurations: Best practices in Dockerfiles, such as setting folder permissions and disabling unnecessary features, ensure that each new container is automatically configured correctly. This reduces manual intervention and lowers the risk of human error. **Compliance** + Meeting Regulatory Requirements: Many industries have strict regulatory requirements (e.g., PCI-DSS, HIPAA) that mandate specific security measures, such as encrypted communications (HTTPS) and logging of client requests. Following IIS best practices in containers helps ensure compliance with these standards. + Auditability: Implementing audit policies and secure logging allows for the traceability of events, which is critical in audits. For example, logging the X-Forwarded-For header ensures that client IP addresses are recorded correctly in proxy-based architectures. **Minimizing Risk in Shared Environments** + Avoiding Misconfigurations: Containers share the host’s kernel, and while they are isolated from one another, a poorly configured IIS instance could expose vulnerabilities or create performance bottlenecks. Best practices ensure that each IIS instance runs optimally, reducing the risk of cross-container issues. + Least Privilege Access: Setting proper permissions for folders and files within the container (e.g., using Set-Acl in PowerShell) ensures that users and processes within the container only have the necessary access, reducing the risk of privilege escalation or data tampering. **Resilience in Ephemeral Environments** + Ephemeral Nature of Containers: Containers are often short-lived and rebuilt frequently. Applying IIS best practices ensures that each container is configured securely and consistently, regardless of how many times it is redeployed. This prevents misconfigurations from being introduced over time. + Mitigating Potential Misconfigurations: By automatically enforcing best practices (e.g., disabling weak protocols or headers), the risk of a misconfiguration during container restarts or updates is minimized. Dockerfile: ``` # Enforce HTTPS (disable HTTP) -- Only if container is target for SSL termination RUN powershell -Command \ "$httpBinding = Get-WebBinding -Name 'Default Web Site' -Protocol http | Where-Object { $_.bindingInformation -eq '*:80:' }; \ if ($httpBinding) { Remove-WebBinding -Name 'Default Web Site' -Protocol http -Port 80; } \ $httpsBinding = Get-WebBinding -Name 'Default Web Site' -Protocol https | Where-Object { $_.bindingInformation -eq '*:443:' }; \ if (-not $httpsBinding) { New-WebBinding -Name 'Default Web Site' -Protocol https -Port 443 -IPAddress '*'; }" # Use secure headers RUN powershell -Command \ "Write-Host 'Adding security headers...'; \ Add-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter 'system.applicationHost/sites/siteDefaults/logFile/customFields' -name "." -value @{logFieldName='X-Forwarded-For';sourceName='X-Forwarded-For';sourceType='RequestHeader'}; \ Add-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.webServer/httpProtocol/customHeaders" -name "." -value @{name='Strict-Transport-Security';value='max-age=31536000; includeSubDomains'}; \ Add-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.webServer/httpProtocol/customHeaders" -name "." -value @{name='X-Content-Type-Options';value='nosniff'}; \ Add-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.webServer/httpProtocol/customHeaders" -name "." -value @{name='X-XSS-Protection';value='1; mode=block'}; \ Add-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.webServer/httpProtocol/customHeaders" -name "." -value @{name='X-Frame-Options';value='DENY'};" # Disable IIS version disclosure RUN powershell -Command \ "Write-Host 'Disabling IIS version disclosure...'; \ Import-Module WebAdministration; \ Set-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.webServer/security/requestFiltering" -name "removeServerHeader" -value "true";" # Set IIS Logging Best Practices RUN powershell -Command \ Set-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.webServer/directoryBrowse" -name "enabled" -value "false"; \ Set-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.webServer/httpErrors" -name "existingResponse" -value "PassThrough"; \ # Enable IIS dynamic and static compression to optimize performance RUN powershell -Command \ "Write-Host 'Enabling IIS compression...'; \ Enable-WindowsOptionalFeature -Online -FeatureName IIS-HttpCompressionDynamic; \ Import-Module WebAdministration; \ Set-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.webServer/urlCompression" -name "doDynamicCompression" -value "true"; \ Set-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.webServer/urlCompression" -name "doStaticCompression" -value "true" # Ensure proper folder permissions using PowerShell's Set-Acl RUN powershell -Command \ "Write-Host 'Setting folder permissions for IIS...'; \ $path = 'C:\\inetpub\\wwwroot'; \ $acl = Get-Acl $path; \ $iusr = New-Object System.Security.Principal.NTAccount('IIS_IUSRS'); \ $rule = New-Object System.Security.AccessControl.FileSystemAccessRule($iusr, 'ReadAndExecute', 'ContainerInherit, ObjectInherit', 'None', 'Allow'); \ $acl.SetAccessRule($rule); \ $users = New-Object System.Security.Principal.NTAccount('Users'); \ $rule2 = New-Object System.Security.AccessControl.FileSystemAccessRule($users, 'ReadAndExecute', 'ContainerInherit, ObjectInherit', 'None', 'Allow'); \ $acl.SetAccessRule($rule2); \ Set-Acl -Path $path -AclObject $acl" ``` Explanation: This command configures IIS to log the X-Forwarded-For header, which is commonly used to capture the original client IP address when a request passes through a proxy or load balancer. By default, IIS only logs the IP address of the load balancer or reverse proxy, so adding this custom log field helps track the true client IP for security auditing, analytics, and troubleshooting. 1. **X-Forwarded-For header** which is commonly used to capture the original client IP address when a request passes through a proxy or load balancer. By default, IIS only logs the IP address of the load balancer or reverse proxy, so adding this custom log field helps track the true client IP for security auditing, analytics, and troubleshooting. 1. **Strict-Transport-Security (HSTS)** Ensures browsers only communicate over HTTPS. The max-age=31536000 specifies that this policy is enforced for 1 year, and includeSubDomains applies the policy to all subdomains. 1. **X-Content-Type-Options** Prevents browsers from "MIME-sniffing" a response away from the declared Content-Type. This helps prevent some types of attacks. 1. **X-XSS-Protection** Enables Cross-Site Scripting (XSS) protection in browsers. 1. **X-Frame-Options** Prevents the page from being embedded in iframes, protecting against clickjacking attacks. 1. **Disable IIS version disclosure** This command disables the Server header in HTTP responses, which by default reveals the version of IIS being used. Hiding this information helps reduce the risk of attackers identifying and targeting vulnerabilities specific to the IIS version. 1. **Enable HTTPS-only connections** This (commented-out) section enforces HTTPS connections and disables HTTP. If uncommented, the Dockerfile will configure IIS to listen only on port 443 (HTTPS) and remove the default HTTP binding on port 80. This is useful when terminating SSL inside the container and ensures that all traffic is encrypted. 1. **Disable Directory Browsing** Prevents IIS from showing a directory listing when no default document is present. This avoids exposing the internal file structure to users. 1. **Pass Through Custom Error Pages** Ensures that if the application has its own error handling, IIS will let the application’s error pages pass through instead of showing default IIS error pages. 1. **Detailed Error Mode** Configures IIS to display detailed error messages for local requests only, helping developers diagnose issues without exposing sensitive information to external users. 1. **Ensure Proper Folder Permissions** This block configures folder permissions for the IIS web root (C:\$1inetpub\$1wwwroot). It sets Read and Execute permissions for the IIS\$1IUSRS and Users groups, ensuring that these users can access the folder but not modify files. Setting the correct permissions minimizes the risk of unauthorized access or tampering with the files hosted by the web server. Following IIS best practices in Windows Containers ensures that your containerized applications are secure, high performance, and scalable. These practices help prevent vulnerabilities, optimize resource usage, ensure compliance, and maintain consistency across container instances. Even though containers are designed to be isolated, proper configuration is necessary to minimize risks and ensure the reliability of your application in production environments. ## 4. Principle of Least Privilege The Principle of Least Privilege (PoLP) is crucial for Windows containers for several important reasons, particularly in enhancing security and minimizing risks within containerized environments. This principle dictates that a system or application should operate with the minimum level of permissions necessary to function properly. Here’s why it’s important in Windows containers: **Minimizing Attack Surface** + Containers often run applications that interact with various system components, and the more privileges an application has, the broader its access to those components. By limiting the container’s permissions to only what’s necessary, PoLP significantly reduces the attack surface, making it harder for an attacker to exploit the container if it becomes compromised. **Limiting the Impact of Compromised Containers** + If a Windows container is compromised, running applications with excessive privileges (e.g., Administrator or root-level access) could allow an attacker to gain control over critical system files or escalate privileges across the container host. By enforcing PoLP, even if a container is breached, the attacker is limited in what they can do, preventing further escalation and access to sensitive resources or other containers. **Protection in Multitenant Environments** + In cloud or enterprise environments, multiple containers can be running on the same physical or virtual infrastructure. PoLP ensures that a compromised container doesn’t have the ability to access resources or data belonging to other tenants. This isolation is crucial for maintaining security in shared, multitenant environments, protecting against lateral movement between containers. **Mitigating Privilege Escalation** + Containers that run with high privileges can be used by attackers to escalate privileges within the system. PoLP mitigates this risk by restricting the container’s access to system resources, thereby preventing unauthorized actions or privilege escalations beyond the container’s environment. **Compliance and Auditing** + Many regulatory standards and security frameworks (e.g., PCI DSS, HIPAA, GDPR) require systems to adhere to PoLP to limit access to sensitive data. Running Windows containers with restricted privileges helps organizations comply with these regulations and ensures that applications are only granted access to the resources they specifically need. **Reducing the Risk of Misconfiguration** + When containers run with unnecessary privileges, even a minor misconfiguration can lead to severe security vulnerabilities. For example, if a container running as Administrator is accidentally exposed to the internet, an attacker could gain control of the system. PoLP helps prevent such risks by defaulting to limited privileges, making misconfigurations less dangerous. **Improved Container Security Posture** + By following PoLP, containers are better isolated from the underlying host system and from each other. This ensures that the containerized application is less likely to access or modify system files or processes outside its defined scope, preserving the integrity of the host operating system and other workloads. Dockerfile: ``` # Strongly recommended that when deploying a Windows server container to any multi-tenant environment that your application runs via the ContainerUser account USER ContainerUser ``` **Explanation:** In this section, the USER ContainerUser command specifies that the application inside the Windows container should run under the ContainerUser account instead of the default Administrator account. Here’s why this is important, especially in a multitenant environment: 1. **Principle of Least Privilege**: The ContainerUser account is a non-administrative user with limited privileges. Running the application under this account adheres to the principle of least privilege, which helps minimize the risk of exploitation. If an attacker were to compromise the application, they would have limited access to the system, reducing the potential damage. 1. **Enhanced Security**: In multitenant environments, containers can share the same underlying infrastructure. Running as ContainerUser ensures that even if one container is compromised, it won’t have administrative privileges to access or modify critical system files or other containers. This reduces the attack surface significantly. 1. **Avoiding Root Access**: By default, containers might run with elevated permissions (similar to root access in Linux containers), which can be dangerous if exploited. Using ContainerUser ensures that the application doesn’t run with unnecessary administrative rights, making it harder for attackers to escalate privileges. 1. **Best Practice for Multitenant Environments**: In environments where multiple users or organizations share the same infrastructure (such as in the cloud), security is critical. Running applications with restricted permissions prevents one tenant’s application from affecting others, protecting sensitive data and resources across the platform. The **USER ContainerUser** command ensures that the application runs with minimal privileges, enhancing security in multitenant environments by limiting the damage that could be done if the container is compromised. This is a best practice to prevent unauthorized access or privilege escalation in a containerized environment. The Principle of Least Privilege is essential for Windows containers because it limits the potential impact of security breaches, reduces the attack surface, and prevents unauthorized access to critical system components. By running containerized applications with only the necessary permissions, organizations can significantly enhance the security and stability of their container environments, especially in multitenant and shared infrastructures. ## Final Thoughts: Why Securing Your Windows Containers is a Must-Have in Today’s Threat Landscape In today’s fast-evolving digital world, where threats are becoming more sophisticated and abundant, securing your Windows containers is not just a recommendation, it’s an absolute necessity. Containers provide a lightweight, flexible way to package and deploy applications, but they are not immune to security vulnerabilities. As more businesses adopt containers to streamline their infrastructure, they also become a potential target for cyberattacks if not properly secured. The internet is flooded with various threats—​ranging from malicious actors targeting unpatched vulnerabilities to automated bots scanning for misconfigurations. Without the right security measures in place, containers can be exploited to expose sensitive data, escalate privileges, or serve as entry points for attacks that can compromise your broader infrastructure. This makes container security as critical as securing any other part of your environment. When using Windows containers, many traditional security best practices still apply. Implementing robust account policies, securing IIS configurations, enforcing HTTPS, using strict firewall rules, and applying least privilege access to critical files are all key measures that ensure the container remains resilient against attacks. Additionally, regular auditing and logging provide visibility into what’s happening inside the container, allowing you to catch suspicious activity before it turns into a full-blown incident. Securing Windows containers also aligns with regulatory requirements that mandate protecting sensitive data and ensuring application integrity. As cloud-native and containerized architectures become more prevalent, ensuring security at every layer, from the base image to the running container, will help safeguard your operations and maintain customer trust. In summary, the rise of containerized applications, coupled with the growing number of cyber threats, makes container security a nonnegotiable aspect of modern infrastructure management. By adhering to best practices and continuously monitoring for vulnerabilities, businesses can enjoy the agility and efficiency of Windows containers without compromising on security. In this threat-rich environment, securing your Windows containers is not just an option—​it’s a must-have.