当前位置:   article > 正文

Spark源码编译与本地调试环境搭建

spark源码编译


最近开始读《深入理解Spark·核心思想与源码分析》,书是16年出版的,基于 Spark1.2 版本进行讲解。刚开始感觉Spark版本有点老,读了三章后发现作者思路清晰,可学习的地方很多。
想要本地方便地阅读和调试源码,需要在本地IDE顺利build通过。下面是跟着书中的讲解做的Spark源码编译和本地调试环境的搭建。作者选用的IDE是Eclipse,构建工具是SBT;我选用的IDE是IntelliJ IDEA,构建工具是Maven。

前置准备工作

以下各前置工作相对简单,不再详细描述。我本地环境是 JDK-1.8,Scala-2.12.4,Maven-3.3.9,Git-2.17.1。

  • JDK 安装和环境变量的配置
  • Scala 安装和环境变量配置
  • Maven 安装和环境变量配置
  • Git 客户端安装和 SSH 配置

Spark 源码编译

Spark-1.2.3 源码编译

首先需要clone源码到本地仓库,这个过程可能需要等待较长的时间。

git clone git@github.com:apache/spark.git -b branch-1.2

clone完成后,开始进行源码编译。编译时可以设置 Maven 选项 -T 1C,使用 Maven 提供的并行编译能力,表示每个 CPU 核心跑一个线程,加快编译速度;-e 选项表示遇到异常打印异常详细信息。

mvn -T 1C -DskipTests -e clean install

第一次编译过程中遇到如下报错信息:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.824 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 13.824 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 10.192 s]
[INFO] Spark Project Core ................................. SUCCESS [02:38 min]
[INFO] Spark Project Bagel ................................ SUCCESS [ 27.409 s]
[INFO] Spark Project GraphX ............................... SUCCESS [01:06 min]
[INFO] Spark Project Streaming ............................ SUCCESS [01:40 min]
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 18.871 s]
[INFO] Spark Project External Flume ....................... FAILURE [ 10.060 s]
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on project spark-streaming-flume_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile failed. CompileFailed -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on project spark-streaming-flume_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile failed.
...
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-streaming-flume_2.10
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33

经过排查,报错原因是项目中缺少了一个依赖:

<dependency>
    <groupId>net.alchim31.maven</groupId>
    <artifactId>scala-maven-plugin</artifactId>
    <version>3.2.0</version>
</dependency>
  • 1
  • 2
  • 3
  • 4
  • 5

在项目的主 pom.xml 中引入上述依赖后,再次执行编译命令进行编译。不过,在这里踩了一个坑:根据 Maven 报错信息最后一行提示,解决问题后可执行如下命令从当前状态恢复build过程:

mvn <goals> -rf :spark-streaming-flume_2.10

于是我按照提示,重新键入编译命令如下:

mvn -T 1C -DskipTests -e clean install -rf :spark-streaming-flume_2.10

结果,编译过程中又得到如下报错信息:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project External Flume ....................... SUCCESS [ 25.568 s]
[INFO] Spark Project External MQTT ........................ SUCCESS [03:45 min]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 48.209 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [01:49 min]
[INFO] Spark Project Examples ............................. FAILURE [06:12 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:58 min (Wall Clock)
···
[ERROR] Failed to execute goal on project spark-examples_2.10: Could not resolve dependencies for project org.apache.spark:spark-examples_2.10:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-mllib_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-hive_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-streaming-twitter_2.10:jar:1.2.3-SNAPSHOT: Could not find artifact org.apache.spark:spark-mllib_2.10:jar:1.2.3-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) -> [Help 1]
···
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15

从输出信息可以看出:使用上述命令恢复 build 后,Maven 只是简单的从 failure 的模块开始往下执行,但并没有将之前跳过的依赖模块重新build,如 Spark Project ML Library 等,见下图:
在这里插入图片描述
所以,到这里我只能clean后重新build:

mvn -T 1C -DskipTests -e clean install

经过上述折腾之后,重新build比较顺利,总计耗时14分钟:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  3.703 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 16.933 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 11.269 s]
[INFO] Spark Project Core ................................. SUCCESS [02:44 min]
[INFO] Spark Project Bagel ................................ SUCCESS [ 28.633 s]
[INFO] Spark Project GraphX ............................... SUCCESS [01:12 min]
[INFO] Spark Project Streaming ............................ SUCCESS [01:41 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [02:09 min]
[INFO] Spark Project SQL .................................. SUCCESS [01:38 min]
[INFO] Spark Project ML Library ........................... SUCCESS [05:52 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 32.681 s]
[INFO] Spark Project Hive ................................. SUCCESS [05:33 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 46.894 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 27.147 s]
[INFO] Spark Project External Twitter ..................... SUCCESS [ 57.706 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 20.473 s]
[INFO] Spark Project External Flume ....................... SUCCESS [01:09 min]
[INFO] Spark Project External MQTT ........................ SUCCESS [ 57.697 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 58.706 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [01:12 min]
[INFO] Spark Project Examples ............................. SUCCESS [01:08 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:10 min (Wall Clock)
···
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29

Spark-2.2.4 源码编译

由于1.2 版本的源码相对较老,后续学习工作中还是选用较新的 2.2 版本,跟着书本对比阅读。这里直接切换到 2.2 分支进行build:

git checkout branch-2.2

切换成功后,使用同样的命令开始编译:

mvn -T 1C -DskipTests -e clean install

源码编译过程中,CPU的利用率如下图:
在这里插入图片描述
整个源码编译过程大约耗时20分钟,没有出现异常,最后顺利完成,输出的构建信息如下:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 23.283 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 12.210 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 21.720 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 44.170 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 16.677 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 37.268 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 50.450 s]
[INFO] Spark Project Core ................................. SUCCESS [05:35 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [01:00 min]
[INFO] Spark Project GraphX ............................... SUCCESS [01:00 min]
[INFO] Spark Project Streaming ............................ SUCCESS [01:53 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [03:52 min]
[INFO] Spark Project SQL .................................. SUCCESS [04:55 min]
[INFO] Spark Project ML Library ........................... SUCCESS [03:36 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 13.582 s]
[INFO] Spark Project Hive ................................. SUCCESS [02:49 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 23.556 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  3.623 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 44.048 s]
[INFO] Spark Project External Flume ....................... SUCCESS [ 42.474 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [  2.933 s]
[INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [01:09 min]
[INFO] Kafka 0.10 Source for Structured Streaming ......... SUCCESS [ 50.854 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 55.074 s]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [  3.453 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 59.934 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  8.501 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20:32 min (Wall Clock)
···
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35

源码阅读环境搭建

源码编译完成,进行简单测试。首先找到 examples 模块中的JavaWordCount.java,看到部分代码如下:

if (args.length < 1) {
	System.err.println("Usage: JavaWordCount <file>");
	System.exit(1);
}

SparkSession spark = SparkSession
  .builder()
  .appName("JavaWordCount")
  .getOrCreate();

JavaRDD<String> lines = spark.read().textFile(args[0]).javaRDD();

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12

这里,需要做的配置有两点:

  1. 根据源码可以看出,这个例子接收一个外部参数 args[0],即需要统计的文件名。这里,我直接修改应用的运行配置,添加一个参数“E:\WorkSpace\IntelliJ IDEA\apache-spark\spark\README.md”,如下图
    在这里插入图片描述
  2. 需要修改 Spark 的运行模式为 local 模式:
SparkSession spark = SparkSession
  .builder()
  .appName("JavaWordCount").master("local")
  .getOrCreate();
  • 1
  • 2
  • 3
  • 4

配置完成后,运行 JavaWordCount,输出信息如下:

MLlib: 1
["Building: 1
contributing: 1
shell:: 2
instance:: 1
Scala,: 1
and: 9
command,: 2
package.): 1
./dev/run-tests: 1
sample: 1
19/07/22 14:15:20 INFO AbstractConnector: Stopped Spark@61f05988{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12

这样,看到第一个 example 就跑通了,接下来就可以愉快地阅读和调试源码了。

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/我家自动化/article/detail/757006
推荐阅读
相关标签
  

闽ICP备14008679号