[Spark_2_보충] Dag, Stages and Task

Workflow: DAG(Directed Acyclic Graph), Stages and Task

Overview (link)

Steps

1. Action되는 순간 DAG 생성 및 DAG scheduler로 전송
2. DAG Scheduler는 Operations(Map, reduce 등등)를 stages로 나눔 (link)
   
   • 2.1) stage는 transformation에 근거해 만들어짐
   • 2.2) 각 stage는 Task로 구성됨(stage: a collection of tasks)
     
   • 2.3) Task는 partion에서 행해지는 작업의 단위(Task: a unit of work on a partition)
   • 2.4) Task의 수는 $\sum(stage \times \text{Partitions in the stage} )$, 즉 task갯수는 partion갯수/per stage eg. following stage1 consists of 4 set of tasks(=4 partions)
     
     • 모든 RDD는 defined number of partitions를 가짐
     • partitions의 갯수는 input format에 따라 쪼개진 수량 (partitions: subset of RDD, rdd2=rdd1.repartition(1000) # resulting in rdd2 having 1000 partitions) (link)
       
   • 2.5) DAG Scheduler의 result는 a set of stages
3. a set of stages를 Task Scheduler에게 전송
4. Task Scheduler는 Cluster Manager에게 task전송
5. Worker Node의 Executor가 task 실행
6. 결과 Driver Program에 전달

---

reference

• https://stackoverflow.com/questions/37528047/how-are-stages-split-into-tasks-in-spark
• https://stackoverflow.com/questions/25836316/how-dag-works-under-the-covers-in-rdd
• https://data-flair.training/blogs/how-apache-spark-works/