前些天发现了一个巨牛的人工智能学习网站,通俗易懂,风趣幽默,忍不住给大家分享一下。点击跳转到网站:https://www.captainai.net/dongkelun
前言
由于工作原因,之前查询Hudi主要是用Hive来查询的,所以对Hive查询Hudi的逻辑比较了解,但是对于Spark查询Hudi的逻辑不太了解。所以现在想要学习一下Spark查询Hudi的大概逻辑,搞清楚它是如何从Spark的源码跳转到Hudi源码执行Hudi查询的逻辑, 这样既能搞清楚Spark查询表的逻辑,也能搞清楚Spark查询Hudi的逻辑,也便于在后面使用Kyuubi Spark SQL 时出现问题能更好的定位解决。
版本
Spark 2.4.4
Hudi master 0.12.0-SNAPSHOT 最新代码
(可以借助Spark3 planChangeLog 打印日志信息查看哪些规则生效)
示例代码
先用上篇文章写Hudi数据,再进行查询
1 | import spark.implicits._ |
spark.read
返回DataFrameReader,后面的format、load方法都是在这个类中
1 | def read: DataFrameReader = new DataFrameReader(self) |
format
source=”hudi”,后面load时会用到
1 | def format(source: String): DataFrameReader = { |
load
save方法首先添加path参数,然后判断source是否等于hive,我们这里source等于hudi,所以不满足,接下来通过DataSource.lookupDataSource查找hudi对应的dataSouce类,然后判断它是不是DataSourceV2的子类,再执行后面的逻辑,我们在上篇文章讲过了DataSource.lookupDataSource返回的是Spark2DefaultSource且不是DataSourceV2的子类,所以接下来执行loadV1Source方法
1 | def load(path: String): DataFrame = { |
loadV1Source
loadV1Source方法的核心是后面的sparkSession.baseRelationToDataFrame,先看他的参数DataSource.resolveRelation
1 | private def loadV1Source(paths: String*) = { |
DataSource.resolveRelation
同样地这里的providingClass.newInstance()为DataSource.lookupDataSource返回的为Spark2DefaultSource,它的父类DefaultSource既实现了SchemaRelationProvider也实现了RelationProvider,但是这里的userSpecifiedSchema为None,所以会匹配到第二个,所以relation = dataSource.createRelation(sparkSession.sqlContext, caseInsensitiveOptions)
1 | /** |
dataSource.createRelation
这里的createRelation是在Spark2DefaultSource的父类DefaultSource中实现的,首先根据传入的路径参数,获取表路径,然后创建HoodieTableMetaClient,获取表类型等表配置,再根据表的配置信息判断返回什么类型的Relation,这里返回resolveBaseFileOnlyRelation
(可见createRelation是实现Hudi自己的查询逻辑的入口)
1 | override def createRelation(sqlContext: SQLContext, |
resolveBaseFileOnlyRelation
1 | private def resolveBaseFileOnlyRelation(sqlContext: SQLContext, |
hasSchemaOnRead是在BaseFileOnlyRelation的父类HoodieBaseRelation中定义的
1 | def hasSchemaOnRead: Boolean = internalSchemaOpt.isDefined |
因为SCHEMA_EVOLUTION_ENABLED的默认值为false,所以internalSchemaOpt返回None,hasSchemaOnRead返回false,所以在resolveBaseFileOnlyRelation返回调用baseRelation.toHadoopFsRelation
1 | val SCHEMA_EVOLUTION_ENABLED: ConfigProperty[Boolean] = HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE |
baseRelation.toHadoopFsRelation
这个方法返回HadoopFsRelation,其中location为HoodieFileIndex,是在其父类HoodieBaseRelation中定义的,fileFomart为Spark24HoodieParquetFileFormat
1 | override protected val shouldExtractPartitionValuesFromPartitionPath: Boolean = |
1 | protected lazy val fileIndex: HoodieFileIndex = |
HoodieFileIndex继承了SparkHoodieTableFileIndex,实现了Spark SQL源码里的FileIndex的listFiles等方法,这在查询Hudi表文件时有用,也正是实现查询Hudi表逻辑的地方,接下来让我们往下看它的listFiles方法是如何被调用以及如何返回查询结果的
1 | case class HoodieFileIndex(spark: SparkSession, |
1 | lazy val (fileFormat: FileFormat, fileFormatClassName: String) = |
baseRelationToDataFrame
通过上面的分析我们知道了,resolveRelation返回的是HadoopFsRelation(HoodieFileIndex,partitionSchema,dataSchema,None,Spark24HoodieParquetFileFormat,optParams)(sparkSession),接下来看一下baseRelationToDataFrame
1 | def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame = { |
Dataset.ofRows在前面的几篇文章中已经分析过多次了,所以我们挑重点进行分析,其中参数logicalPlan为LogicalRelation(HadoopFsRelation(HoodieFileIndex,_...),relation.schema.toAttributes, None, false)
1 | def ofRows(sparkSession: SparkSession, logicalPlan: LogicalPlan): DataFrame = { |
Dataset.show
我们知道对于查询上面的Dataset.ofRows只会通过qe.assertAnalyzed()触发analysis阶段,通过分析日志我们发现并没有生效的规则,所以我们重点看一下后面的show,它最后会调用到showString,而showString的核心逻辑在getRows
1 | def show(): Unit = show(20) |
getRows
getRows的核心是在newDf.select(castCols: _*).take(numRows + 1),它会触发planning
1 | private[sql] def getRows( |
df.select
先看它的select,其中select中的logicalPlan为LogicalRelation(HadoopFsRelation(HoodieFileIndex,_...),所以withPlan的logicalPlan为Project(cols.map(_.named),LogicalRelation(HadoopFsRelation(HoodieFileIndex,_...)))
1 | def select(cols: Column*): DataFrame = withPlan { |
take
接下来到了take
1 | def take(n: Int): Array[T] = head(n) |
limit
先看参数limit
1 | def limit(n: Int): Dataset[T] = withTypedPlan { |
其中Literal(n)的apply方法返回Literal(21, IntegerType)
1 | object Limit { |
Limit的apply返回GlobalLimit(Literal(21, IntegerType), LocalLimit(Literal(21, IntegerType), Project(cols.map(_.named),LogicalRelation(HadoopFsRelation(HoodieFileIndex,_...)))),所以limit返回Dataset(sparkSession, GlobalLimit(Literal(21, IntegerType), LocalLimit(Literal(21, IntegerType), Project(cols.map(.named),LogicalRelation(HadoopFsRelation(HoodieFileIndex,…)))))
1 | private def withTypedPlan[U : Encoder](logicalPlan: LogicalPlan): Dataset[U] = { |
withAction
withAction在前面的几篇文章也讲过了,其中executedPlan都会触发一遍完整的Spark SQL的parsing、analysis、optimization 、planning,并且最终会在withNewExecutionId调用方法action,具体为上面的collectFromPlan,让我们先看看有哪些重要的规则或者策略生效了
1 | private def withAction[U](name: String, qe: QueryExecution)(action: SparkPlan => U) = { |
plan 入口
这里提一下plan入口,之前没有注意到,这里是把optimizedPlan用ReturnAnswer包装了一下(因为后面分析会用到)
1 | lazy val sparkPlan: SparkPlan = { |
根据ReturnAnswer注释可知,对于take或collect等操作,仅会在逻辑查询计划的顶部进行规则转化
1 | /** |
strategies
先看一下有哪些策略
1 | 0 = {SparkStrategies$PythonEvals$@13376} |
planner.plan
再看一下plan方法,之前对它的理解仅限于他会遍历策略调用aplly,现在对它的理解稍微多了一点,它首先遍历一遍策略返回candidates,再遍历candidates,调用collectPlaceholders拆开被PlanLater包装的子计划返回placeholders,如果placeholders非空,则再递归调用plan方法应用其子节点
1 | def plan(plan: LogicalPlan): Iterator[PhysicalPlan] = { |
SpecialLimits
第一遍遍历因为有ReturnAnswer包装所以在SpecialLimits匹配成功
1 | /** |
匹配Limit时,会调用它的unapply,它会匹配到第三个,返回CollectLimitExec(21,PlanLater(Project(cols.map(_.named),LogicalRelation(HadoopFsRelation(HoodieFileIndex,_...)))
1 | def unapply(p: GlobalLimit): Option[(Expression, LogicalPlan)] = { |
其中PlanLater只是把plan包装了一下,在planner.plan方法中会通过collectPlaceholders拆开
1 | override protected def collectPlaceholders(plan: SparkPlan): Seq[(SparkPlan, LogicalPlan)] = { |
FileSourceStrategy
拆开PlanLater,为Project(cols.map(_.named),LogicalRelation(HadoopFsRelation(HoodieFileIndex,_...)),在FileSourceStrategy匹配时调用PhysicalOperation的unapply方法,所以这里会在第一个匹配成功,返回ProjectExec(projectList,FileSourceScanExec(HadoopFsRelation(HoodieFileIndex,_...))),因为这里为子节点,所以加上父节点,最后返回的是CollectLimitExec(21,ProjectExec(projectList,FileSourceScanExec(HadoopFsRelation(HoodieFileIndex,_...))))
1 | object PhysicalOperation extends PredicateHelper { |
1 | def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match { |
prepareForExecution
上面返回的sparkPlan为CollectLimitExec(21,ProjectExec(projectList,FileSourceScanExec(HadoopFsRelation(HoodieFileIndex,_...))))
1 | lazy val executedPlan: SparkPlan = prepareForExecution(sparkPlan) |
CollapseCodegenStages
1 | def apply(plan: SparkPlan): SparkPlan = { |
首先CollectLimitExec匹配到other,所以递归调用children,ProjectExec实现了CodegenSupport,返回WholeStageCodegenExec(ProjectExec(projectList,FileSourceScanExec(HadoopFsRelation(HoodieFileIndex,_...)))),因为ProjectExec是子节点,所以最终返回CollectLimitExec(21,WholeStageCodegenExec(ProjectExec(projectList,FileSourceScanExec(HadoopFsRelation(HoodieFileIndex,_...)))))
1 | private def insertWholeStageCodegen(plan: SparkPlan): SparkPlan = plan match { |
1 | case class ProjectExec(projectList: Seq[NamedExpression], child: SparkPlan) |
collectFromPlan
1 | private def collectFromPlan(plan: SparkPlan): Array[T] = { |
executeCollect
这里的plan为CollectLimitExec,调用child.executeTake,这里child为WholeStageCodegenExec,它没有重写executeTake,所以调用父类SparkPlan的executeTake
1 | case class CollectLimitExec(limit: Int, child: SparkPlan) extends UnaryExecNode { |
executeTake
核心getByteArrayRdd
1 | /** |
getByteArrayRdd
1 | private def getByteArrayRdd(n: Int = -1): RDD[(Long, Array[Byte])] = { |
doExecute
WholeStageCodegenExec.doExecute,核心:child.asInstanceOf[CodegenSupport].inputRDDs(),这里的child为ProjectExec
1 | override def doExecute(): RDD[InternalRow] = { |
inputRDDs
ProjectExec.inputRDDs, child为FileSourceScanExec
1 | override def inputRDDs(): Seq[RDD[InternalRow]] = { |
FileSourceScanExec.inputRDD
- relation: HadoopFsRelation
- relation.fileFormat: Spark24HoodieParquetFileFormat
- relation.bucketSpec: None
1 | private lazy val inputRDD: RDD[InternalRow] = { |
createNonBucketedReadRDD
1 | /** |
selectedPartitions
- relation.location: HoodieFileIndex
1 | private lazy val selectedPartitions: Array[PartitionDirectory] = { |
HoodieFileIndex.listFiles
涉及方法:lookupCandidateFilesInMetadataTable、refresh、doRefresh、loadPartitionPathFiles等,其中在初始化new时会调用refresh 、doRefresh
1 | /** |
在doRefresh初始化的cachedAllInputFileSlices实现了获取Hudi文件列表的逻辑
1 | cachedAllInputFileSlices = partitionFiles.keySet().stream() |
Spark24HoodieParquetFileFormat
PR:[HUDI-3204] Fixing partition-values being derived from partition-path instead of source columns 大家可以自己研究一下
总结
这样基本上理清了spark.read.format(“hudi”).load 从Spark源码到Hudi源码的逻辑,限于能力和精力有些地方还没有完全搞清楚,不过只要搞懂整体流程,细节上在有需要时再慢慢研究。