# Logging and monitoring for AWS PCS Logging and monitoring Monitoring is an important part of maintaining the reliability, availability, and performance of AWS PCS and your other AWS resources. AWS provides the following monitoring tools to watch AWS PCS, report when something is wrong, and take automatic actions when appropriate: + *Amazon CloudWatch* monitors your AWS resources and the applications you run on AWS in real time. You can collect and track metrics, create customized dashboards, and set alarms that notify you or take actions when a specified metric reaches a threshold that you specify. For example, you can have CloudWatch track CPU usage or other metrics of your Amazon EC2 instances and automatically launch new instances when needed. For more information, see the [Amazon CloudWatch User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/). + *Amazon CloudWatch Logs* enables you to monitor, store, and access your log files from Amazon EC2 instances, CloudTrail, and other sources. CloudWatch Logs can monitor information in the log files and notify you when certain thresholds are met. You can also archive your log data in highly durable storage. For more information, see the [Amazon CloudWatch Logs User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/). + *AWS CloudTrail* captures API calls and related events made by or on behalf of your AWS account and delivers the log files to an Amazon S3 bucket that you specify. You can identify which users and accounts called AWS, the source IP address from which the calls were made, and when the calls occurred. For more information, see the [AWS CloudTrail User Guide](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/). # Job completion logs in AWS PCS Job completion logs Job completion logs give you key details about your AWS Parallel Computing Service (AWS PCS) jobs when they complete, at no additional cost. You can use other AWS services to access and process your log data, such as Amazon CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), and Amazon Data Firehose; AWS PCS records metadata about your jobs, such as the following. + Job ID and name + User and group information + Job state (such as `COMPLETED`, `FAILED`, `CANCELLED`) + Partition used + Time limits + Start, end, submit, and eligible times + Node list and count + Processor count + Working directory + Resource usage (CPU, memory) + Exit codes + Node details (names, instance IDs, instance types) **Contents** + [ ## Prerequisites ](#monitoring_job-completion-logs_prereqs) + [ ## Set up job completion logs ](#monitoring_job-completion-logs_setup) + [ ## How to find job completion logs ](#monitoring_job-completion-logs_access) + [ ### CloudWatch Logs ](#monitoring_job-completion-logs_access_cloudwatch) + [ ### Amazon S3 ](#monitoring_job-completion-logs_access_s3) + [ ## Job completion log fields ](#monitoring_job-completion-logs_fields) + [ ## Example job completion logs ](#monitoring_job-completion-logs_example) ## Prerequisites The IAM principal that manages the AWS PCS cluster must allow the `pcs:AllowVendedLogDeliveryForResource` action. The following example IAM policy grants the required permissions. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "PcsAllowVendedLogsDelivery", "Effect": "Allow", "Action": ["pcs:AllowVendedLogDeliveryForResource"], "Resource": [ "arn:aws:pcs:*::cluster/*" ] } ] } ``` ------ ## Set up job completion logs You can set up job completion logs for your AWS PCS cluster with the AWS Management Console or AWS CLI. ------ #### [ AWS Management Console ] **To set up job completion logs with the console** 1. Open the [AWS PCS console](https://console.aws.amazon.com/pcs). 1. In the navigation pane, choose **Clusters**. 1. Choose the cluster where you want to add job completion logs. 1. On the cluster details page, choose the **Logs** tab. 1. Under **Job Completion Logs**, choose **Add** to add up to 3 log delivery destinations from among CloudWatch Logs, Amazon S3, and Firehose. 1. Choose **Update log deliveries**. ------ #### [ AWS CLI ] **To set up job completion logs with the AWS CLI** 1. Create a log delivery destination: ``` aws logs put-delivery-destination --region region \ --name pcs-logs-destination \ --delivery-destination-configuration \ destinationResourceArn=resource-arn ``` Replace: + *region* — The AWS Region where you want to create the destination, such as `us-east-1` + *pcs-logs-destination* — A name for the destination + *resource-arn* — The Amazon Resource Name (ARN) of a CloudWatch Logs log group, S3 bucket, or Firehose delivery stream. For more information, see [PutDeliveryDestination](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutDeliveryDestination.html) in the *Amazon CloudWatch Logs API Reference*. 1. Set the PCS cluster as a log delivery source: ``` aws logs put-delivery-source --region region \ --name cluster-logs-source-name \ --resource-arn cluster-arn \ --log-type PCS_JOBCOMP_LOGS ``` Replace: + *region* — The AWS Region of your cluster, such as `us-east-1` + *cluster-logs-source-name* — A name for the source + *cluster-arn* — the ARN of your AWS PCS cluster For more information, see [PutDeliverySource](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutDeliverySource.html) in the *Amazon CloudWatch Logs API Reference*. 1. Connect the delivery source to the delivery destination: ``` aws logs create-delivery --region region \ --delivery-source-name cluster-logs-source \ --delivery-destination-arn destination-arn ``` Replace: + *region* — The AWS Region, such as `us-east-1` + *cluster-logs-source* — The name of your delivery source + *destination-arn* — The ARN of your delivery destination For more information, see [CreateDelivery](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateDelivery.html) in the *Amazon CloudWatch Logs API Reference*. ------ ## How to find job completion logs You can configure log destinations in CloudWatch Logs and Amazon S3. AWS PCS uses the following structured path names and file names. ### CloudWatch Logs AWS PCS uses the following name format for the CloudWatch Logs stream: ``` AWSLogs/PCS/cluster-id/jobcomp.log ``` For example: `AWSLogs/PCS/pcs_abc123de45/jobcomp.log` ### Amazon S3 AWS PCS uses the following name format for the S3 path: ``` AWSLogs/account-id/PCS/region/cluster-id/jobcomp/year/month/day/hour/ ``` For example: `AWSLogs/111122223333/PCS/us-east-1/pcs_abc123de45/jobcomp/2025/06/19/11/` AWS PCS uses the following name format for the log files: ``` PCS_jobcomp_year-month-day-hour_cluster-id_random-id.log.gz ``` For example: `PCS_jobcomp_2025-06-19-11_pcs_abc123de45_04be080b.log.gz` ## Job completion log fields AWS PCS writes job completion log data as JSON objects. The JSON container `jobcomp` holds job details. The following table describes the fields inside the `jobcomp` container. Some fields are only present in specific circumstances, such as for array jobs or heterogeneous jobs. **Job completion log fields** | Name | Example value | Required | Notes | | --- | --- | --- | --- | | job\$1id | 11 | yes | Always present with value | | user | "root" | yes | Always present with value | | user\$1id | 0 | yes | Always present with value | | group | "root" | yes | Always present with value | | group\$1id | 0 | yes | Always present with value | | name | "wrap" | yes | Always present with value | | job\$1state | "COMPLETED" | yes | Always present with value | | partition | "Hydra-MpiQueue-abcdef01-7" | yes | Always present with value | | time\$1limit | "UNLIMITED" | yes | Always present, but might be "UNLIMITED" | | start\$1time | "2025-06-19T10:58:57" | yes | Always present, but might be "Unknown" | | end\$1time | "2025-06-19T10:58:57" | yes | Always present, but might be "Unknown" | | node\$1list | "Hydra-MpiNG-abcdef01-2345-1" | yes | Always present with value | | node\$1cnt | 1 | yes | Always present with value | | proc\$1cnt | 1 | yes | Always present with value | | work\$1dir | "/root" | yes | Always present, but might be "Unknown" | | reservation\$1name | "weekly\$1maintenance" | yes | Always present, but might be an empty string "" | | tres.cpu | 1 | yes | Always present with value | | tres.mem.val | 600 | yes | Always present with value | | tres.mem.unit | "M" | yes | Can be "M" or "bb" | | tres.node | 1 | yes | Always present with value | | tres.billing | 1 | yes | Always present with value | | account | "finance" | yes | Always present, but might be an empty string "" | | qos | "normal" | yes | Always present, but might be an empty string "" | | wc\$1key | "project\$11" | yes | Always present, but might be an empty string "" | | cluster | "unknown" | yes | Always present, but might be "unknown" | | submit\$1time | "2025-06-19T10:55:46" | yes | Always present, but might be "Unknown" | | eligible\$1time | "2025-06-19T10:55:46" | yes | Always present, but might be "Unknown" | | array\$1job\$1id | 12 | no | Only present if the job is an array job | | array\$1task\$1id | 1 | no | Only present if the job is an array job | | het\$1job\$1id | 10 | no | Only present if the job is a heterogeneous job | | het\$1job\$1offset | 0 | no | Only present if the job is a heterogeneous job | | derived\$1exit\$1code\$1status | 0 | yes | Always present with value | | derived\$1exit\$1code\$1signal | 0 | yes | Always present with value | | exit\$1code\$1status | 0 | yes | Always present with value | | exit\$1code\$1signal | 0 | yes | Always present with value | | node\$1details[0].name | "Hydra-MpiNG-abcdef01-2345-1" | no | Always present, but node\$1details might be "[]" | | node\$1details[0].instance\$1id | "i-0abcdef01234567a" | no | Always present, but node\$1details might be "[]" | | node\$1details[0].instance\$1type | "t4g.micro" | no | Always present, but node\$1details might be "[]" | ## Example job completion logs The following examples show job completion logs for various job types and states: ``` { "jobcomp": { "job_id": 1, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T16:32:57", "end_time": "2025-06-19T16:33:03", "node_list": "Hydra-MpiNG-abcdef01-2345-[1-2]", "node_cnt": 2, "proc_cnt": 2, "work_dir": "/usr/bin", "reservation_name": "", "tres": { "cpu": 2, "mem": { "val": 1944, "unit": "M" }, "node": 2, "billing": 2 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T16:29:40", "eligible_time": "2025-06-19T16:29:41", "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-1", "instance_id": "i-0abc123def45678", "instance_type": "t4g.micro" }, { "name": "Hydra-MpiNG-abcdef01-2345-2", "instance_id": "i-0def456abc78901", "instance_type": "t4g.micro" } ] } } { "jobcomp": { "job_id": 2, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T16:33:13", "end_time": "2025-06-19T16:33:14", "node_list": "Hydra-MpiNG-abcdef01-2345-[1-2]", "node_cnt": 2, "proc_cnt": 2, "work_dir": "/usr/bin", "reservation_name": "", "tres": { "cpu": 2, "mem": { "val": 1944, "unit": "M" }, "node": 2, "billing": 2 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T16:33:13", "eligible_time": "2025-06-19T16:33:13", "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-1", "instance_id": "i-0abc123def45678", "instance_type": "t4g.micro" }, { "name": "Hydra-MpiNG-abcdef01-2345-2", "instance_id": "i-0def456abc78901", "instance_type": "t4g.micro" } ] } } { "jobcomp": { "job_id": 3, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T22:58:57", "end_time": "2025-06-19T22:58:57", "node_list": "Hydra-MpiNG-abcdef01-2345-1", "node_cnt": 1, "proc_cnt": 1, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 972, "unit": "M" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T22:55:46", "eligible_time": "2025-06-19T22:55:46", "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-1", "instance_id": "i-0abc234def56789", "instance_type": "t4g.micro" } ] } } { "jobcomp": { "job_id": 4, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "525600", "start_time": "2025-06-19T23:04:27", "end_time": "2025-06-19T23:04:27", "node_list": "Hydra-MpiNG-abcdef01-2345-[1-2]", "node_cnt": 2, "proc_cnt": 2, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 2, "mem": { "val": 1944, "unit": "M" }, "node": 2, "billing": 2 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:01:38", "eligible_time": "2025-06-19T23:01:38", "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-1", "instance_id": "i-0abc234def56789", "instance_type": "t4g.micro" }, { "name": "Hydra-MpiNG-abcdef01-2345-2", "instance_id": "i-0def345abc67890", "instance_type": "t4g.micro" } ] } } { "jobcomp": { "job_id": 5, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "FAILED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T23:09:00", "end_time": "2025-06-19T23:09:00", "node_list": "(null)", "node_cnt": 0, "proc_cnt": 0, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 1, "unit": "G" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:09:00", "eligible_time": "2025-06-19T23:09:00", "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 1, "node_details": [] } } { "jobcomp": { "job_id": 6, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "CANCELLED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T23:09:36", "end_time": "2025-06-19T23:09:36", "node_list": "(null)", "node_cnt": 0, "proc_cnt": 0, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 400, "unit": "M" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:09:35", "eligible_time": "2025-06-19T23:09:36", "het_job_id": 6, "het_job_offset": 0, "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 1, "node_details": [] } } { "jobcomp": { "job_id": 7, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "CANCELLED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T23:10:03", "end_time": "2025-06-19T23:10:03", "node_list": "(null)", "node_cnt": 0, "proc_cnt": 0, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 400, "unit": "M" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:10:03", "eligible_time": "2025-06-19T23:10:03", "het_job_id": 7, "het_job_offset": 0, "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 1, "node_details": [] } } { "jobcomp": { "job_id": 8, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T23:11:24", "end_time": "2025-06-19T23:11:24", "node_list": "Hydra-MpiNG-abcdef01-2345-1", "node_cnt": 1, "proc_cnt": 1, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 400, "unit": "M" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:11:23", "eligible_time": "2025-06-19T23:11:23", "het_job_id": 8, "het_job_offset": 0, "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-1", "instance_id": "i-0abc234def56789", "instance_type": "t4g.micro" } ] } } { "jobcomp": { "job_id": 9, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T23:11:24", "end_time": "2025-06-19T23:11:24", "node_list": "Hydra-MpiNG-abcdef01-2345-2", "node_cnt": 1, "proc_cnt": 1, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 400, "unit": "M" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:11:23", "eligible_time": "2025-06-19T23:11:23", "het_job_id": 8, "het_job_offset": 1, "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-2", "instance_id": "i-0def345abc67890", "instance_type": "t4g.micro" } ] } } { "jobcomp": { "job_id": 10, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T23:12:24", "end_time": "2025-06-19T23:12:24", "node_list":"Hydra-MpiNG-abcdef01-2345-1", "node_cnt": 1, "proc_cnt": 1, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 400, "unit": "M" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:12:14", "eligible_time": "2025-06-19T23:12:14", "het_job_id": 10, "het_job_offset": 0, "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-1", "instance_id": "i-0abc234def56789", "instance_type": "t4g.micro" } ] } } { "jobcomp": { "job_id": 11, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T23:12:24", "end_time": "2025-06-19T23:12:24", "node_list":"Hydra-MpiNG-abcdef01-2345-2", "node_cnt": 1, "proc_cnt": 1, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 600, "unit": "M" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:12:14", "eligible_time": "2025-06-19T23:12:14", "het_job_id": 10, "het_job_offset": 1, "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-2", "instance_id": "i-0def345abc67890", "instance_type": "t4g.micro" } ] } } { "jobcomp": { "job_id": 13, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T23:47:57", "end_time": "2025-06-19T23:47:58", "node_list":"Hydra-MpiNG-abcdef01-2345-1", "node_cnt": 1, "proc_cnt": 1, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 972, "unit": "M" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:43:56", "eligible_time": "2025-06-19T23:43:56" , "array_job_id": 12, "array_task_id": 1, "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-1", "instance_id": "i-0abc345def67890", "instance_type": "t4g.micro" } ] } } { "jobcomp": { "job_id": 12, "user": "root", "user_id": 0, "group": "root", "group_id": 0, "name": "wrap", "job_state": "COMPLETED", "partition": "Hydra-MpiQueue-abcdef01-7", "time_limit": "UNLIMITED", "start_time": "2025-06-19T23:47:58", "end_time": "2025-06-19T23:47:58", "node_list":"Hydra-MpiNG-abcdef01-2345-1", "node_cnt": 1, "proc_cnt": 1, "work_dir": "/root", "reservation_name": "", "tres": { "cpu": 1, "mem": { "val": 972, "unit": "M" }, "node": 1, "billing": 1 }, "account": "", "qos": "", "wc_key": "", "cluster": "unknown", "submit_time": "2025-06-19T23:43:56", "eligible_time": "2025-06-19T23:43:56" , "array_job_id": 12, "array_task_id": 2, "derived_exit_code_status": 0, "derived_exit_code_signal": 0, "exit_code_status": 0, "exit_code_signal": 0, "node_details": [ { "name": "Hydra-MpiNG-abcdef01-2345-1", "instance_id": "i-0abc345def67890", "instance_type": "t4g.micro" } ] } } ``` # Scheduler logs in AWS PCS Scheduler logs You can configure AWS PCS to send detailed logging data from your cluster scheduler to Amazon CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), and Amazon Data Firehose. This can assist with monitoring and troubleshooting. **Contents** + [ ## Prerequisites ](#monitoring_scheduler-logs_prereqs) + [ ## Set up scheduler logs ](#monitoring_scheduler-logs_setup) + [ ## Scheduler log stream paths and names ](#monitoring_scheduler-logs_paths) + [ ## Example scheduler log record ](#monitoring_scheduler-logs_record) ## Prerequisites The IAM principal that manages the AWS PCS cluster must allow the `pcs:AllowVendedLogDeliveryForResource` action. The following example IAM policy grants the required permissions. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "PcsAllowVendedLogsDelivery", "Effect": "Allow", "Action": ["pcs:AllowVendedLogDeliveryForResource"], "Resource": [ "arn:aws:pcs:*::cluster/*" ] } ] } ``` ------ ## Set up scheduler logs You can set up scheduler logs for your AWS PCS cluster with the AWS Management Console or AWS CLI. ------ #### [ AWS Management Console ] **To set up scheduler logs with the console** 1. Open the [AWS PCS console](https://console.aws.amazon.com/pcs). 1. In the navigation pane, choose **Clusters**. 1. Choose the cluster where you want to add scheduler logs. 1. On the cluster details page, choose the **Logs** tab. 1. Under **Scheduler Logs**, choose **Add** to add up to 3 log delivery destinations from among CloudWatch Logs, Amazon S3, and Firehose. 1. Choose **Update log deliveries**. ------ #### [ AWS CLI ] **To set up scheduler logs with the AWS CLI** 1. Create a log delivery destination: ``` aws logs put-delivery-destination --region region \ --name pcs-logs-destination \ --delivery-destination-configuration \ destinationResourceArn=resource-arn ``` Replace: + *region* — The AWS Region where you want to create the destination, such as `us-east-1` + *pcs-logs-destination* — A name for the destination + *resource-arn* — The Amazon Resource Name (ARN) of a CloudWatch Logs log group, S3 bucket, or Firehose delivery stream. For more information, see [PutDeliveryDestination](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutDeliveryDestination.html) in the *Amazon CloudWatch Logs API Reference*. 1. Set the PCS cluster as a log delivery source: ``` aws logs put-delivery-source --region region \ --name cluster-logs-source-name \ --resource-arn cluster-arn \ --log-type PCS_SCHEDULER_LOGS ``` Replace: + *region* — The AWS Region of your cluster, such as `us-east-1` + *cluster-logs-source-name* — A name for the source + *cluster-arn* — the ARN of your AWS PCS cluster For more information, see [PutDeliverySource](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutDeliverySource.html) in the *Amazon CloudWatch Logs API Reference*. 1. Connect the delivery source to the delivery destination: ``` aws logs create-delivery --region region \ --delivery-source-name cluster-logs-source \ --delivery-destination-arn destination-arn ``` Replace: + *region* — The AWS Region, such as `us-east-1` + *cluster-logs-source* — The name of your delivery source + *destination-arn* — The ARN of your delivery destination For more information, see [CreateDelivery](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateDelivery.html) in the *Amazon CloudWatch Logs API Reference*. ------ ## Scheduler log stream paths and names The path and name for AWS PCS scheduler logs depend on the destination type. + **CloudWatch Logs** + A CloudWatch Logs stream follows this naming convention. ``` AWSLogs/PCS/${cluster_id}/${log_name}_${scheduler_major_version}.log ``` **Example** ``` AWSLogs/PCS/abcdef0123/slurmctld_24.05.log ``` + **S3 bucket** + An S3 bucket output path follows this naming convention: ``` AWSLogs/${account-id}/PCS/${region}/${cluster_id}/${log_name}/${scheduler_major_version}/yyyy/MM/dd/HH/ ``` **Example** ``` AWSLogs/111111111111/PCS/us-east-2/abcdef0123/slurmctld/24.05/2024/09/01/00. ``` + An S3 object name follows this convention: ``` PCS_${log_name}_${scheduler_major_version}_#{expr date 'event_timestamp', format: "yyyy-MM-dd-HH"}_${cluster_id}_${hash}.log ``` **Example** ``` PCS_slurmctld_24.05_2024-09-01-00_abcdef0123_0123abcdef.log ``` ## Example scheduler log record AWS PCS scheduler logs are structured. They include fields such as the cluster identifier, scheduler type, major and patch versions, in addition to the log message emitted from the Slurm controller process. Here is an example. ``` { "resource_id": "s3431v9rx2", "resource_type": "PCS_CLUSTER", "event_timestamp": 1721230979, "log_level": "info", "log_name": "slurmctld", "scheduler_type": "slurm", "scheduler_major_version": "25.05", "scheduler_patch_version": "3", "node_type": "controller_primary", "message": "[2024-07-17T15:42:58.614+00:00] Running as primary controller\n" } ``` # Monitoring AWS Parallel Computing Service with Amazon CloudWatch Monitoring with CloudWatch Amazon CloudWatch provides monitoring of your AWS Parallel Computing Service (AWS PCS) cluster health and performance by collecting metrics from the cluster at intervals. These metrics are retained, allowing you to access historical data and gain insights into your cluster's performance over time. CloudWatch also enables you to monitor the EC2 instances launched by AWS PCS to meet your scaling requirements. While you can inspect logs on running instances, CloudWatch metrics and logging data are typically deleted once instances are terminated. However, you can configure the CloudWatch agent on instances using an EC2 launch template to persist metrics and logs even after instance termination, enabling long-term monitoring and analysis. Explore the topics in this section to learn more about monitoring AWS PCS using CloudWatch. **Topics** + [ # Monitoring AWS PCS metrics using CloudWatch ](monitoring-cloudwatch_metrics.md) + [ # Monitoring AWS PCS instances using Amazon CloudWatch ](monitoring-cloudwatch_instances.md) # Monitoring AWS PCS metrics using CloudWatch Monitoring metrics You can monitor AWS PCS cluster health using Amazon CloudWatch, which collects data from your cluster and turns it into near real-time metrics. These statistics are retained for a period of 15 months, so that you can access historical information and gain a better perspective on how your cluster is performing. Cluster metrics are sent to CloudWatch at 1-minute periods. For more information about CloudWatch, see [What Is Amazon CloudWatch?](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) in the *Amazon CloudWatch User Guide*. AWS PCS publishes the following metrics into the **AWS/PCS** namespace in CloudWatch. They have a single dimension, `ClusterId`. | Name | Description | Units | | --- | --- | --- | | ActualCapacity | IdleCapacity \$1 UtilizedCapacity | Count | | CapacityUtilization | UtilizedCapacity / ActualCapacity | Count | | DesiredCapacity | ActualCapacity \$1 PendingCapacity | Count | | IdleCapacity | Count of instances that are running but not allocated to jobs | Count | | UtilizedCapacity | Count of instances that are running and allocated to jobs | Count | # Monitoring AWS PCS instances using Amazon CloudWatch Monitoring instances AWS PCS launches Amazon EC2 instances as needed to meet the scaling requirements defined in your PCS compute node groups. You can monitor these instances while they are running using Amazon CloudWatch. You can inspect the logs of running instances by logging into them and using interactive command line tools. However, by default, CloudWatch metrics data is only retained for a limited period once an instance is terminated, and instance logs are usually deleted along with the EBS volumes that back the instance. To retain metrics or logging data from the instances launched by PCS after they are terminated, you can configure the CloudWatch agent on your instances with an EC2 launch template. This topic provides an overview of monitoring running instances and provides examples of how to configure persistent instance metrics and logs. ## Monitoring running instances ### Finding AWS PCS instances To monitor instances launched by PCS, find the running instances associated with a cluster or compute node group. Then, in the EC2 console for a given instance, inspect the **Status and alarms** and **Monitoring** sections. If login access is configured for those instances, you can connect to them and inspect various log files on the instances. For more information on identifying which instances are managed by PCS, see [Finding compute node group instances in AWS PCS](working-with_compute-instances.md). ### Enabling detailed metrics By default, instance metrics are collected at 5-minute intervals. To collect metrics at one minute intervals, enable detailed CloudWatch monitoring in your compute node group launch template. For more information, see [Turn on detailed CloudWatch monitoring](working-with_launch-templates_parameters.md#working-with_launch-templates_parameters_cw). ## Configuring persistent instance metrics and logs You can retain the metrics and logs from your instances by installing and configuring the Amazon CloudWatch agent on them. This consists of three main steps: 1. Create a CloudWatch agent configuration. 1. Store the configuration where it can be retrieved by PCS instances. 1. Write an EC2 launch template that installs the CloudWatch agent software, fetches your configuration, and starts the CloudWatch agent using the configuration. For more information, see [Collect metrics, logs, and traces with the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html) in the *Amazon CloudWatch User Guide*, and [Using Amazon EC2 launch templates with AWS PCS](working-with_launch-templates.md). ### Create a CloudWatch Agent configuration Before deploying the CloudWatch agent on your instances, you must generate a JSON configuration file that specifies the metrics, logs, and traces to collect. Configuration files can be created using a wizard or manually, using a text editor. The configuration file will be created manually for this demonstration. On a computer where you have the AWS CLI installed, create a CloudWatch configuration file named **config.json** with the contents that follow. You can also use the following URL to download a copy of the file. ``` https://aws-hpc-recipes.s3.amazonaws.com/main/recipes/pcs/cloudwatch/assets/config.json ``` **Notes** + The log paths in the sample file are for Amazon Linux 2. If your instances will use a different base operating system, change the paths as appropriate. + To capture other logs, add additional entries under `collect_list`. + Values in `{brackets}` are templated variables. For the complete list of supported variables, see [Manually create or edit the CloudWatch agent configuration file](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html) in the *Amazon CloudWatch User Guide*. + You can choose to omit `logs` or `metrics` if you don't want to collect these information types. ``` { "agent": { "metrics_collection_interval": 60 }, "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "/var/log/cloud-init.log", "log_group_class": "STANDARD", "log_group_name": "/PCSLogs/instances", "log_stream_name": "{instance_id}.cloud-init.log", "retention_in_days": 30 }, { "file_path": "/var/log/cloud-init-output.log", "log_group_class": "STANDARD", "log_stream_name": "{instance_id}.cloud-init-output.log", "log_group_name": "/PCSLogs/instances", "retention_in_days": 30 }, { "file_path": "/var/log/amazon/pcs/bootstrap.log", "log_group_class": "STANDARD", "log_stream_name": "{instance_id}.bootstrap.log", "log_group_name": "/PCSLogs/instances", "retention_in_days": 30 }, { "file_path": "/var/log/slurmd.log", "log_group_class": "STANDARD", "log_stream_name": "{instance_id}.slurmd.log", "log_group_name": "/PCSLogs/instances", "retention_in_days": 30 }, { "file_path": "/var/log/messages", "log_group_class": "STANDARD", "log_stream_name": "{instance_id}.messages", "log_group_name": "/PCSLogs/instances", "retention_in_days": 30 }, { "file_path": "/var/log/secure", "log_group_class": "STANDARD", "log_stream_name": "{instance_id}.secure", "log_group_name": "/PCSLogs/instances", "retention_in_days": 30 } ] } } }, "metrics": { "aggregation_dimensions": [ [ "InstanceId" ] ], "append_dimensions": { "AutoScalingGroupName": "${aws:AutoScalingGroupName}", "ImageId": "${aws:ImageId}", "InstanceId": "${aws:InstanceId}", "InstanceType": "${aws:InstanceType}" }, "metrics_collected": { "cpu": { "measurement": [ "cpu_usage_idle", "cpu_usage_iowait", "cpu_usage_user", "cpu_usage_system" ], "metrics_collection_interval": 60, "resources": [ "*" ], "totalcpu": false }, "disk": { "measurement": [ "used_percent", "inodes_free" ], "metrics_collection_interval": 60, "resources": [ "*" ] }, "diskio": { "measurement": [ "io_time" ], "metrics_collection_interval": 60, "resources": [ "*" ] }, "mem": { "measurement": [ "mem_used_percent" ], "metrics_collection_interval": 60 }, "swap": { "measurement": [ "swap_used_percent" ], "metrics_collection_interval": 60 } } } } ``` This file instructs the CloudWatch agent to monitor several files that can be helpful in diagnosing errors in instance bootstrapping, authentication and login, and other troubleshooting domains. These include: + `/var/log/cloud-init.log` – Output from the initial stage of instance configuration + `/var/log/cloud-init-output.log` – Output from commands that run during instance configuration + `/var/log/amazon/pcs/bootstrap.log` – Output from PCS-specific operations that run during instance configuration + `/var/log/slurmd.log` – Output from the Slurm workload manager's daemon slurmd + `/var/log/messages` – System messages from the kernel, system services, and applications + `/var/log/secure` – Logs related to authentication attempts, such as SSH, sudo, and other security events The log files are sent to a CloudWatch log group named `/PCSLogs/instances`. The log streams are a combination of the instance ID and the base name of the log file. The log group has a retention time of 30 days. In addition, the file instructs CloudWatch agent to collect several common metrics, aggregating them by instance ID. ### Store the configuration The CloudWatch agent configuration file has to be stored where it can be accessed by PCS compute node instances. There are two common ways to do this. You can upload it to an Amazon S3 bucket that your compute node group instances will have access to via their instance profile, Alternatively, you can store it as an SSM parameter in Amazon Systems Manager Parameter Store. #### Upload to an S3 bucket To store your file in S3, use the AWS CLI commands that follow. Before running the command, make these replacements: + Replace *amzn-s3-demo-bucket* with your own S3 bucket name First, (this is optional if you have an existing bucket), create a bucket to hold your configuration file(s). ``` aws s3 mb s3://amzn-s3-demo-bucket ``` Next, upload the file to the bucket. ``` aws s3 cp ./config.json s3://amzn-s3-demo-bucket/ ``` #### Store as an SSM parameter To store your file as an SSM parameter, use the command that follows. Before running the command, make these replacements: + Replace *region-code* with the AWS Region where you are working with AWS PCS. + (Optional) Replace *AmazonCloudWatch-PCS* with your own name for the parameter. Note that if you change the prefix of the name from `AmazonCloudWatch-` you will need to specifically add read access to the SSM parameter in your node group instance profile. ``` aws ssm put-parameter \ --region region-code \ --name "AmazonCloudWatch-PCS" \ --type String \ --value file://config.json ``` ### Write an EC2 launch template The specific details for the launch template depend on whether your configuration file is stored in S3 or SSM. #### Use a configuration stored in S3 This script installs CloudWatch agent, imports a configuration file from an S3 bucket, and launches the CloudWatch agent with it. Replace the following values in this script with your own details: + *amzn-s3-demo-bucket* – The name of an S3 bucket your account can read from + */config.json* – Path relative to the S3 bucket root where the configuration is stored ``` MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="==MYBOUNDARY==" --==MYBOUNDARY== Content-Type: text/cloud-config; charset="us-ascii" packages: - amazon-cloudwatch-agent runcmd: - aws s3 cp s3://amzn-s3-demo-bucket/config.json /etc/s3-cw-config.json - /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file://etc/s3-cw-config.json --==MYBOUNDARY==-- ``` The IAM instance profile for the node group must have access to the bucket. Here is an example IAM policy for the bucket in the user data script above. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket", "arn:aws:s3:::amzn-s3-demo-bucket/*" ] } ] } ``` ------ Also note that the instances must allow outbound traffic to the S3 and CloudWatch endpoints. This can be accomplished using security groups or VPC endpoints, depending on your cluster architecture. #### Use a configuration stored in SSM This script installs CloudWatch agent, imports a configuration file from an SSM parameter, and launches the CloudWatch agent with it. Replace the following values in this script with your own details: + (Optional) Replace *AmazonCloudWatch-PCS* with your own name for the parameter. ``` MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="==MYBOUNDARY==" --==MYBOUNDARY== Content-Type: text/cloud-config; charset="us-ascii" packages: - amazon-cloudwatch-agent runcmd: - /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:AmazonCloudWatch-PCS --==MYBOUNDARY==-- ``` The IAM instance policy for the node group must have the **CloudWatchAgentServerPolicy** attached to it. If your parameter name does not start with `AmazonCloudWatch-` you will need to specifically add read access to the SSM parameter in your node group instance profile. Here is an example IAM policy that illustrates this for prefix *DOC-EXAMPLE-PREFIX*. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement" : [ { "Sid" : "CustomCwSsmMParamReadOnly", "Effect" : "Allow", "Action" : [ "ssm:GetParameter" ], "Resource" : "arn:aws:ssm:*:*:parameter/DOC-EXAMPLE-PREFIX*" } ] } ``` ------ Also note that the instances must allow outbound traffic to the SSM and CloudWatch endpoints. This can be accomplished using security groups or VPC endpoints, depending on your cluster architecture. # Logging AWS Parallel Computing Service API calls using AWS CloudTrail CloudTrail logs AWS PCS is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in AWS PCS. CloudTrail captures all API calls for AWS PCS as events. The calls captured include calls from the AWS PCS console and code calls to the AWS PCS API operations. If you create a trail, you can enable continuous delivery of CloudTrail events to an Amazon S3 bucket, including events for AWS PCS. If you don't configure a trail, you can still view the most recent events in the CloudTrail console in **Event history**. Using the information collected by CloudTrail, you can determine the request that was made to AWS PCS, the IP address from which the request was made, who made the request, when it was made, and additional details. To learn more about CloudTrail, see the [AWS CloudTrail User Guide](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html). ## AWS PCS information in CloudTrail CloudTrail is enabled on your AWS account when you create the account. When activity occurs in AWS PCS, that activity is recorded in a CloudTrail event along with other AWS service events in **Event history**. You can view, search, and download recent events in your AWS account. For more information, see [Viewing events with CloudTrail Event history](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html). For an ongoing record of events in your AWS account, including events for AWS PCS, create a trail. A *trail* enables CloudTrail to deliver log files to an Amazon S3 bucket. By default, when you create a trail in the console, the trail applies to all AWS Regions. The trail logs events from all Regions in the AWS partition and delivers the log files to the Amazon S3 bucket that you specify. Additionally, you can configure other AWS services to further analyze and act upon the event data collected in CloudTrail logs. For more information, see the following: + [Overview for creating a trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) + [CloudTrail supported services and integrations](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-aws-service-specific-topics.html) + [Configuring Amazon SNS notifications for CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/configure-sns-notifications-for-cloudtrail.html) + [Receiving CloudTrail log files from multiple regions](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/receive-cloudtrail-log-files-from-multiple-regions.html) and [Receiving CloudTrail log files from multiple accounts](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-receive-logs-from-multiple-accounts.html) All AWS PCS actions are logged by CloudTrail and are documented in the [AWS Parallel Computing Service API Reference](https://docs.aws.amazon.com/pcs/latest/APIReference/). For example, calls to the `CreateComputeNodeGroup`, `UpdateQueue`, and `DeleteCluster` actions generate entries in the CloudTrail log files. Every event or log entry contains information about who generated the request. The identity information helps you determine the following: + Whether the request was made with root or AWS Identity and Access Management (IAM) user credentials. + Whether the request was made with temporary security credentials for a role or federated user. + Whether the request was made by another AWS service. For more information, see the [CloudTrail userIdentity element](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html). ## Understanding CloudTrail log file entries from AWS PCS A trail is a configuration that enables delivery of events as log files to an S3 bucket that you specify. CloudTrail log files contain one or more log entries. An event represents a single request from any source and includes information about the requested action, the date and time of the action, request parameters, and so on. CloudTrail log files aren't an ordered stack trace of the public API calls, so they don't appear in any specific order. The following example shows a CloudTrail log entry for a `CreateQueue` action. ``` { "eventVersion": "1.09", "userIdentity": { "type": "AssumedRole", "principalId": "AIDACKCEVSQ6C2EXAMPLE:admin", "arn": "arn:aws:sts::012345678910:assumed-role/Admin/admin", "accountId": "012345678910", "accessKeyId": "ASIAY36PTPIEXAMPLE", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "AROAY36PTPIEEXAMPLE", "arn": "arn:aws:iam::012345678910:role/Admin", "accountId": "012345678910", "userName": "Admin" }, "attributes": { "creationDate": "2024-07-16T17:05:51Z", "mfaAuthenticated": "false" } } }, "eventTime": "2024-07-16T17:13:09Z", "eventSource": "pcs.amazonaws.com", "eventName": "CreateQueue", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36", "requestParameters": { "clientToken": "c13b7baf-2894-42e8-acec-example", "clusterIdentifier": "abcdef0123", "computeNodeGroupConfigurations": [ { "computeNodeGroupId": "abcdef0123" } ], "queueName": "all" }, "responseElements": { "queue": { "arn": "arn:aws:pcs:us-east-1:609783872011:cluster/abcdef0123/queue/abcdef0123", "clusterId": "abcdef0123", "computeNodeGroupConfigurations": [ { "computeNodeGroupId": "abcdef0123" } ], "createdAt": "2024-07-16T17:13:09.276069393Z", "id": "abcdef0123", "modifiedAt": "2024-07-16T17:13:09.276069393Z", "name": "all", "status": "CREATING" } }, "requestID": "a9df46d7-3f6d-43a0-9e3f-example", "eventID": "7ab18f88-0040-47f5-8388-example", "readOnly": false, "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "012345678910", "eventCategory": "Management", "tlsDetails": { "tlsVersion": "TLSv1.3", "cipherSuite": "TLS_AES_128_GCM_SHA256", "clientProvidedHostHeader": "pcs.us-east-1.amazonaws.com" }, "sessionCredentialFromConsole": "true" } ```