View a markdown version of this page

Run custom scripts with node lifecycle actions in AWS PCS - AWS PCS
Lifecycle stagesHow execution works

Run custom scripts with node lifecycle actions in AWS PCS

Node lifecycle actions run custom scripts at defined points in a compute node's lifecycle. Use them to mount file systems, join directory services, install software, or set up monitoring. You define scripts in the compute node group configuration and reuse them across compute node groups and clusters.

A node lifecycle action is a script that AWS PCS runs automatically at a specific point in a compute node's lifecycle. You define scripts in your compute node group configuration: where each script lives (Amazon S3 or HTTPS), what arguments to pass, the stage at which it runs, when it re-runs on reboot, and what to do if it fails. AWS PCS retrieves the script, caches it on the instance, runs it as root, and writes its output to a dedicated log file.

Node lifecycle actions provide:

  • Reusability — Reference the same script from many compute node groups with different arguments.

  • Per-script error handling — Terminate the node, stop the sequence, or continue on a failure.

  • API-native configuration — Define and update lifecycle actions through UpdateComputeNodeGroup. Configuration changes drain and replace existing nodes. For more information, see Configure node lifecycle actions in AWS PCS.

Lifecycle stages

AWS PCS currently supports two stages. Within each stage, scripts run in the order you define them. By default, a script runs only on first boot; set its executionPolicy to EVERY_BOOT to also re-run it on reboot. For more information, see Configure node lifecycle actions in AWS PCS.

Stage

When it runs

Services available

Typical use

Node bootstrapped (nodeBootstrapped)

Runs after AWS PCS finishes node setup and before slurmd starts.

Network and Amazon S3 are up, and SSH, SSM, NTP (chrony), and cron are active for debugging. AWS PCS-managed setup is done (Slurm configured, Hyper-Threading disabled), so you can rely on it. Slurm client commands are not yet available.

Tasks that must complete before the node accepts jobs, such as mounting shared storage, configuring networking, or installing software packages.

Node ready (nodeReady)

Runs after slurmd starts and the node registers with the Slurm controller.

Everything in the nodeBootstrapped stage, plus Slurm client commands (sinfo, scontrol).

Tasks that require Slurm to be running.
nodeReady and the job-dispatch window

nodeReady scripts run after slurmd starts, so there is a brief window in which the scheduler might dispatch a job to the node before your nodeReady scripts finish. If a script must complete before any job runs:

  • Move it to the nodeBootstrapped stage, if it doesn't need Slurm commands.

  • Configure slurmd to start in a DRAIN state. Have the last script call scontrol update NodeName=$(hostname -s) State=RESUME.

  • Use a Slurm prolog.

How execution works

On first launch:

  1. The node boots and the AWS PCS agent calls RegisterComputeNodeGroupInstance.

  2. The agent caches the lifecycle configuration and downloads the scripts.

  3. The AWS PCS configuration phase runs (Slurm is configured, Hyper-Threading is disabled, and so on).

  4. Node bootstrapped scripts run in the order you defined them.

  5. slurmd starts and the node registers with the Slurm controller. At this point the node is available to accept jobs.

  6. Node ready scripts run in the order you defined them. These scripts run after the node is available to accept jobs, so the scheduler might dispatch a job before they finish (see Lifecycle stages).

On reboot, the same nodeBootstrapped → slurmd → nodeReady sequence runs. Only scripts marked EVERY_BOOT re-run. Scripts set to FIRST_BOOT_ONLY (the default) are skipped. The scriptCachingPolicy controls whether scripts are re-downloaded. For more information, see Configure node lifecycle actions in AWS PCS.

For limits on the number of scripts per stage, argument and script name lengths, and maximum script size, see UpdateComputeNodeGroup in the AWS PCS API Reference.

Topics