View a markdown version of this page

Spark jobs optimization - AWS Prescriptive Guidance
Actions

Spark jobs optimization

Apache Spark jobs on EKS have unique characteristics, they require large burst capacity, run for variable durations, and can waste resources if drivers outlive executors or if executor pools are over-provisioned.

Actions

Implement Spark-specific node management and scheduling optimizations. Refer to this GitHub repository for sample commands to create dedicated nodepool for Spark workloads.

Spark configuration for cost optimization

For sample commands to create dynamic allocation scales executors based on pending tasks,see this GitHub repository

Spark job lifecycle optimization

Optimization

For the complete scripts and manifests, see the 09-spark-jobs-optimization folder in the code repository.

Configuration

Savings

Dynamic allocation

spark.dynamicAllocation.enabled=true

30–50% fewer executor-hours

Executor idle timeout

spark.dynamicAllocation.executorIdleTimeout=60s

Rapid scale-down

Spot instances for executors

Node selector for spot capacity type

60–90% off on-demand pricing

On-demand for driver

Prevent job failure from Spot interruption

N/A (reliability)

Node decommissioning

spark.decommission.enabled=true

Graceful Spot handling

Recommendations:

  • Use Spot instances for Spark executors (they are fault-tolerant by design)

  • Keep Spark drivers on On-Demand instances to prevent job failures

  • Enable dynamic allocation to scale executors based on pending tasks

  • Set aggressive executorIdleTimeout (30–60s) to release resources quickly

  • Use Karpenter's consolidationPolicy: WhenEmpty for Spark NodePools to avoid disrupting running executors

  • Schedule large batch Spark jobs during off-peak hours for better Spot availability and lower contention