Spark Driver, Executors, and Cluster Manager

Architecture

Driver: Coordinates the application, creates RDDs/DAGs. Executors: Run tasks on worker nodes, store cached data. Cluster Manager: Allocates resources (YARN, Mesos, Kubernetes).

Job Execution Flow

Driver creates DAG -> DAG Scheduler divides into stages -> Task Scheduler assigns tasks to executors -> Executors run tasks on data partitions -> Results returned to Driver.

Key Metrics

Monitor: executor memory usage, GC time, shuffle read/write, task duration, serialization time. Poor performance usually means too many shuffles or insufficient memory.

Code Example

from pyspark.sql import SparkSession

spark = SparkSession.builder
.appName("ETL Pipeline")
.config("spark.executor.memory", "8g")
.config("spark.sql.shuffle.partitions", "200")
.getOrCreate()

Read with partitioning

df = spark.read.parquet("s3://data/orders/")

Transform (avoid shuffles)

result = df
.filter(df["amount"] > 100)
.groupBy("region")
.agg({"amount": "sum", "*": "count"})

Write output

result.write.mode("overwrite").parquet("s3://data/output/")

Best Practices

• Cache intermediate results when reused
• Avoid shuffles by designing data layouts
• Use broadcast joins for small tables
• Partition by common filter columns

Interview Tips

• Explain driver-executor architecture
• Discuss shuffle optimization
• When to use broadcast joins?