CDH6.3.2 Spark-submit 提交作业到 yarn_cdh spark-submit

一、在idea中 写一个 wordcount 案例：

这里一定不能设置 .setMaster(“local[*]”)

这里一定不能设置 .setMaster(“local[*]”)

这里一定不能设置 .setMaster(“local[*]”)

重要的事情要说三遍！！！！否则 在启动yarn 模式的时候会报错！！！
val conf = new SparkConf().setMaster(“local[*]”).setAppName(“WordCount”)

package submit_test
import org.apache.spark.{SparkConf, SparkContext, SparkFiles}
object WordCount {
  def main(args: Array[String]): Unit = {
    //scala代码实现spark入口 (这里没有指明是什么模式,在提交时指明)
    //yarn这里一定不能设置.setMaster("local[*]")
    //val conf = new SparkConf().setMaster("local[*]").setAppName("WordCount")
    val conf = new SparkConf().setAppName("WordCount")
   
    val sc = new SparkContext(conf)
    //加载hdfs文件到SparkRDD
    val lines = sc.textFile(args(0))
    //WordCount实现
    val result = lines.flatMap(x=>x.split(" ")).map((_,1)).reduceByKey(_+_)
    //结果保存在hdfs
    result.saveAsTextFile(args(1))
    sc.stop()
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

二、使用idea打包：

2.1 默认都点ok (有的依赖不需要 ，可以自选)



2.2build 和 rebuild 都可以：

2.3 将spark-submit.jar 上传到 虚拟机上：

三、虚拟机上的操作

[root@cdh01 sparktest]# pwd
/opt/sparktest
[root@cdh01 sparktest]# ll
总用量 178308
-rw-r--r-- 1 root root 182583469 3月  23 20:51 spark-submit.jar
[root@cdh01 sparktest]# zip -d spark-submit.jar META-INF/*.RSA META-INF/*.DSA META-INF/*.SF
	
	zip warning: name not matched: META-INF/*.RSA
deleting: META-INF/DUMMY.SF
deleting: META-INF/DUMMY.DSA

[root@cdh01 sparktest]# 
1
2
3
4
5
6
7
8
9
10
11
12

四、在hdfs 上创建一个 文件夹 sparktest

	在sparktest文件夹下创建一个 word.txt文件
	
	文件内容如下：
1
2
3

hello world
hello print
1
2

五、提交任务 ：

[root@cdh01 sparktest]# spark-submit --master yarn --deploy-mode cluster --driver-memory 500m --executor-memory 500m --executor-cores 2 --class submit_test.WordCount /opt/sparktest/spark-submit.jar hdfs://192.168.2.112:8020/sparktest/word.txt hdfs://192.168.2.112:8020/sparktest/wcOut


1
2
3

其中 这个输入、输出路径 的ip 是高可用活跃的那天机器的ip
hdfs://192.168.2.112:8020/sparktest/word.txt
hdfs://192.168.2.112:8020/sparktest/wcOut

输出的文件夹如下：

wordcount 的结果：

备注：

我用这种打包方式 spark-submit 也可以：

[root@cdh01 sparktest]# ll
总用量 366372
-rw-r--r-- 1 root root 194196468 3月  23 21:13 spark-submit-1.0-SNAPSHOT.jar
-rw-r--r-- 1 root root 180963015 3月  23 20:53 spark-submit.jar

[root@cdh01 sparktest]# spark-submit --master yarn --deploy-mode cluster --driver-memory 500m --executor-memory 500m --executor-cores 2 --class submit_test.WordCount /opt/sparktest/spark-submit-1.0-SNAPSHOT.jar hdfs://192.168.2.112:8020/sparktest/word.txt hdfs://192.168.2.112:8020/sparktest/wcOut333
1
2
3
4
5
6

pom.xml如下：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.atguidian.sparksubmit</groupId>
    <artifactId>spark-submit</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <spark.version>2.4.0</spark.version>
        <scala.version>2.11</scala.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
    </dependencies>
    <build>
        <pluginManagement>
            <plugins>
                <!--  scala-maven-plugin：编译scala程序的Maven插件 -->
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.2</version>
                </plugin>
                <!--  maven-compiler-plugin：编译java程序的Maven插件 -->
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.6.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <!--  编译scala程序的Maven插件的一些配置参数 -->
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <!--  编译java程序的Maven插件的一些配置参数 -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <!--  maven-shade-plugin：打jar包用的Mavne插件 -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124