Update the scheduler version of an AWS PCS cluster

Use these steps to update the scheduler version on your cluster. There are two options depending on whether you can tolerate job interruption. For more information about choosing between options, see Updating the scheduler version of a cluster in AWS PCS.

Option 1: Rolling update

The controller is updated while the fleet keeps running. Existing nodes continue using the previous Slurm version until they are drained and replaced. New nodes launched after the update use the target version. Running jobs are not interrupted.

When to use:

Step 0 — Check starting state

Your cluster is running controller version "A" (e.g. 24.11) and you want to migrate to version "B" (e.g. 25.11). Confirm all compute nodes in your fleet run the same major version, using this command from a cluster node:

scontrol show nodes | grep "Version="

# Example output:
#   NodeAddr=compute-1 NodeHostName=compute-1 Version=24.11.7
#   NodeAddr=compute-2 NodeHostName=compute-2 Version=24.11.7

Confirm the AWS PCS agent version on a compute node. Connect to the node with Systems Manager and check the bootstrap log:

grep "PCS Agent version" /var/log/amazon/pcs/bootstrap.log | tail -1

# Example output:
#   /opt/aws/pcs/bin/pcs_bootstrap_init.sh: INFO: Bootstrap starting with PCS Agent version: 1.3.2-1

Rolling updates require AWS PCS agent version 1.4.0 or later on all compute node AMIs. For more information, see AWS PCS agent versions.

Step 1 — Prepare and roll out the dual-version AMIs

Build or identify AMIs that include both Slurm version A and version B, and the latest AWS PCS agent.

Once the AMIs are ready:

Step 2 — Update the cluster controller

Call UpdateCluster with scheduler.version set to version B.

AWS Management Console

1. Open the AWS PCS console at https://console.aws.amazon.com/pcs/.

2. In the navigation pane, choose Clusters.

3. Select the cluster to update and choose Edit.

4. Under Cluster details, select the target scheduler version from the Scheduler dropdown.

5. Choose Update to submit the version update.

6. Monitor the cluster status. The cluster shows as UPDATING during the update and returns to ACTIVE when complete. The update typically completes in 5–15 minutes.

AWS CLI

aws pcs update-cluster \
  --cluster-identifier cluster-id \
  --scheduler version=25.11

Wait for the cluster to return to ACTIVE. The update typically completes in 5–15 minutes.

During this operation the controller is briefly unavailable:

After the update, the compute fleet is in a mixed state: nodes running before the update continue using Slurm version A's slurmd; new nodes use Slurm version B. This is expected.

If the cluster does not return to ACTIVE or UPDATE_FAILED within 30 minutes, contact AWS Support for assistance.

Step 3 — Drain nodes still running Slurm version A

Identify and drain nodes still on the previous version. From a node of the cluster, run:

scontrol show nodes | grep "Version="
scontrol update NodeName=node State=DRAIN Reason="Slurm version update"

Once drained nodes finish their current jobs they terminate and are replaced by nodes on Slurm version B.

Step 4 — Verify consistent fleet on Slurm version B

Confirm all nodes report version B. From a node of the cluster, run:

scontrol show nodes | grep "Version="

All nodes should now report Slurm version B. The update is complete.

Option 2: Full-fleet recycle

The entire fleet is terminated before the controller is updated, then scaled back up from a new AMI with the target Slurm version. This procedure is simpler, but requires all nodes and running jobs to be terminated.

When to use:

Step 0 — Check starting state

Your cluster is running controller version "A" (e.g. 24.11) and you want to migrate to version "B" (e.g. 25.11). Confirm all compute nodes in your fleet run the same major version, using this command from a cluster node:

scontrol show nodes | grep "Version="

# Example output:
#   NodeAddr=compute-1 NodeHostName=compute-1 Version=24.11.7
#   NodeAddr=compute-2 NodeHostName=compute-2 Version=24.11.7

Confirm the AWS PCS agent version on a compute node. Connect to the node with Systems Manager and check the bootstrap log:

grep "PCS Agent version" /var/log/amazon/pcs/bootstrap.log | tail -1

# Example output:
#   /opt/aws/pcs/bin/pcs_bootstrap_init.sh: INFO: Bootstrap starting with PCS Agent version: 1.3.2-1

Use the latest AWS PCS agent on your target AMIs. For more information, see AWS PCS agent versions.

Step 1 — Prepare the target AMIs

Build or identify AMIs that include Slurm version B and the latest AWS PCS agent.

Step 2 — Scale down the entire fleet

Record the current minNodeCount and maxNodeCount for each compute node group — you will restore these in Step 4.

for cng in $(aws pcs list-compute-node-groups --cluster-identifier cluster-id --query "computeNodeGroups[].id" --output text); do
    aws pcs get-compute-node-group \
      --cluster-identifier cluster-id \
      --compute-node-group-identifier "$cng" \
      --query "computeNodeGroup.{Id:id,AmiId:amiId,Min:scalingConfiguration.minInstanceCount,Max:scalingConfiguration.maxInstanceCount}" \
      --output table
  done

Set minNodeCount and maxNodeCount to 0 on every compute node group:

aws pcs update-compute-node-group \
  --cluster-identifier cluster-id \
  --compute-node-group-identifier cng-id \
  --scaling-configuration '{"minNodeCount": 0, "maxNodeCount": 0}'

Verify no instances tagged aws:pcs:cluster-id matching your cluster are running before continuing:

aws ec2 describe-instances \
    --filters "Name=tag:aws:pcs:cluster-id,Values=cluster-id" \
    --query "Reservations[].Instances[].[InstanceId,ImageId,State.Name]" \
    --output table

Step 3 — Update the cluster controller

AWS Management Console

1. Open the AWS PCS console at https://console.aws.amazon.com/pcs/.

2. In the navigation pane, choose Clusters.

3. Select the cluster to update and choose Edit.

4. Under Cluster details, select the target scheduler version from the Scheduler dropdown.

5. Choose Update to submit the version update.

6. Monitor the cluster status. The cluster shows as UPDATING during the update and returns to ACTIVE when complete. The update typically completes in 5–15 minutes.

AWS CLI

aws pcs update-cluster \
  --cluster-identifier cluster-id \
  --scheduler version=25.11

Wait for the cluster to return to ACTIVE. The update typically completes in 5–15 minutes.

If the cluster does not return to ACTIVE or UPDATE_FAILED within 30 minutes, contact AWS Support for assistance.

Step 4 — Update compute node groups and restore capacity

For each compute node group, set the new AMI and restore the original minimum and maximum capacity limits:

aws pcs update-compute-node-group \
  --cluster-identifier cluster-id \
  --compute-node-group-identifier cng-id \
  --ami-id new-ami-id \
  --scaling-configuration '{"minNodeCount": previous-min, "maxNodeCount": previous-max}'

The cluster scales back up. All new nodes run Slurm version B with the latest AWS PCS agent.

Example: Updating across multiple versions

If the target version is outside the compatibility window of your current version, you must move the controller through one or more intermediate versions, updating it one hop at a time. Each hop must target a supported version within the compatibility window of the current controller version.

Because Option 2: Full-fleet recycle scales the fleet to zero before updating the controller, no compute nodes are running while the controller moves between versions. As a result, your AMIs can use the final target version directly — only the controller update (Step 3) is repeated for each hop.

The following example updates a cluster from 23.11 to 25.11 using the Option 2 procedure. 23.11 is outside the compatibility window of 25.11, so the controller is updated in two hops (23.11 to 25.05, then 25.05 to 25.11). Follow the Option 2 steps, with Step 3 split into one update per hop: