前些天发现了一个巨牛的人工智能学习网站,通俗易懂,风趣幽默,忍不住给大家分享一下。点击跳转到网站:https://www.captainai.net/dongkelun
前言
最近在进行Spark任务调度的源码学习,最开始对CoarseGrainedExecutorBackend的启动流程不是很清楚,所以带着这个疑问继续深入学习,终于弄清楚了
CoarseGrainedExecutorBackend是如何启动的,并且对Spark任务调度源码的了解更深入了一点。本篇主要是带着这个疑问以standalone模式总结一下
CoarseGrainedExecutorBackend启动的主要的流程,并不对每一部分的源码进行深入详细的总结。
SparkContext 初始化
首先从SparkContext的初始化开始,一般我们写Spark代码也是首先创建SparkContext。其中会执行:
1 | // Create and start the scheduler 这个master是在sparkSubmit.Main方法得到 |
createTaskScheduler 根据master模式匹配判断返回
1 | /** |
其中TaskScheduler目前只有TaskSchedulerImpl这个实现类
然后会执行
1 | // start TaskScheduler after taskScheduler sets DAGScheduler reference in DAGScheduler's |
new StandaloneSchedulerBackend()
在执行val backend = new StandaloneSchedulerBackend(scheduler, sc, masterUrls),会先在父类CoarseGrainedSchedulerBackend中执行
1 | val driverEndpoint = rpcEnv.setupEndpoint(ENDPOINT_NAME, createDriverEndpoint()) |
其中DriverEndpoint的onstart方法
1 | override def onStart(): Unit = { |
receive方法
1 | override def receive: PartialFunction[Any, Unit] = { |
即不停的调用makeOffers
1 | // Make fake resource offers on all executors |
TaskSchedulerImpl.start()
1 | override def start(): Unit = { |
backend.start()
1 | override def start(): Unit = { |
上面的start方法的关键点是:
1 | val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend", |
client.start()
1 | def start(): Unit = { |
ClientEndpoint的 onstart()
1 | override def onStart(): Unit = { |
registerWithMaster()
1 |
|
tryRegisterAllMasters()
1 |
|
这里的Master.ENDPOINT_NAME为
1 | val ENDPOINT_NAME = "Master" |
Master
1 | private[deploy] class Master( |
可以看出来Master也是RpcEndpoint
Master.receive()
1 | override def receive: PartialFunction[Any, Unit] = { |
Master.schedule()
1 |
|
Master.startExecutorsOnWorkers()
1 |
|
Master.allocateWorkerResourceToExecutors()
1 |
|
launchExecutor()
1 | private def launchExecutor(worker: WorkerInfo, exec: ExecutorDesc): Unit = { |
Worker.receive()
1 | override def receive: PartialFunction[Any, Unit] = synchronized { |
manager.start() (ExecutorRunner)
1 | private[worker] def start(): Unit = { |
ExecutorRunner.fetchAndRunExecutor()
前面提到过appDesc.command包含类名org.apache.spark.executor.CoarseGrainedExecutorBackend,builder.start()会执行Linux命令,
启动CoarseGrainedExecutorBackend
1 | /** |
CoarseGrainedExecutorBackend.main()
最后启动CoarseGrainedExecutorBackend执行main
1 | def main(args: Array[String]): Unit = { |
1 | def parseArguments(args: Array[String], classNameForEntry: String): Arguments = { |
1 | def run( |
参考
https://blog.csdn.net/luyllyl/category_7506344.html