Decoupled Serverless Scheduler, Part 1
Publication date: February 18, 2021 (Diagram history)
This architecture shows how to deploy a decoupled serverless scheduler to run any HPC application at scale. You submit jobs to Amazon Simple Queue Service using AWS Systems Manager Run Command, and AWS Step Functions orchestrates the job execution on Amazon Elastic Compute Cloud instances.
Decoupled Serverless Scheduler, Part 1
The following steps describe the architecture:
-
Jobs submitted to Amazon SQS use AWS Systems Manager Run Command such as bash or Windows PowerShell.
-
You launch an Amazon EC2 instance or cluster of Amazon EC2 instances with the tag key
scheduler-queue. -
On Amazon EC2 launch, an Amazon CloudWatch Event triggers an AWS Lambda function.
-
The Lambda function looks for the tag key
scheduler-queueand triggers a new Step Functions state machine, passing the instance ID and tag value (SQS job queue name). -
The workflow polls Amazon SQS for a new job, and continues to poll until there are no more jobs.
-
The workflow runs the job on previously launched Amazon EC2 instances or cluster of Amazon EC2 instances.
-
The workflow continuously writes job status to a Amazon DynamoDB table.
-
You monitor job status through the AWS Management Console or AWS Command Line Interface (AWS CLI).
Further reading
For additional information, refer to the following resources:
Diagram history
To be notified about updates to this reference architecture diagram, subscribe to the RSS feed.
| Change | Description | Date |
|---|---|---|
Initial publication | Reference architecture diagram first published. | February 18, 2021 |
Reference architecture diagram first published. | February 18, 2021 |
Note
To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.
