赞
踩
一、环境:
-- Hadoop集群
1. Hadoop集群(dm46、dm47、dm48),开启了Kerberos安全,集群所有组件基于K8S管理,运行在docker pod中
2. yarn中的python版本是3.6,java版本是1.8
-- 客户端:dm45
1. 开源spark版本:3.3.1
2. conda有两个环境:base的python是3.6,pyspark_env的版本是3.8
二、目标:
在dm45结点,通过spark(3.3.1)客户端,基于yarn模式,向Hadoop集群提交spark作业。
三、步骤(在dm45操作):
1. 下载spark客户端,并解压
spark-3.3.1-bin-hadoop3.tgz
spark home路径:/home/xxxxx/kdh/spark
2. kinit yarn
3. spark配置
a. 将hdfs-site.xml、core-site.xml、yarn-site.xml拷贝到/home/xxxxx/kdh/spark/conf
b. 修改spark-env.sh
cd /home/xxxxx/kdh/spark/conf
cp spark-env.sh.template spark-env.sh
vi /home/xxxxx/kdh/spark/conf/spark-env.sh
-------------------------------------------------------------------------------------------
HADOOP_CONF_DIR=/home/xxxxx/soft/TDH-Client/conf/hadoop
YARN_CONF_DIR=/home/xxxxx/soft/TDH-Client/conf/hadoop
YARN_HOME=/home/xxxxx/soft/TDH-Client/hadoop/hadoop-yarn
HADOOP_YARN_HOME=/home/xxxxx/soft/TDH-Client/hadoop/hadoop-yarn
HADOOP_HOME=/home/xxxxx/soft/TDH-Client/hadoop/hadoop
HADOOP_LIBEXEC_DIR=/home/xxxxx/soft/TDH-Client/hadoop/hadoop/libexec/
HADOOP_HDFS_HOME=/home/xxxxx/soft/TDH-Client/hadoop/hadoop-hdfs
HADOOP_COMMON_HOME=/home/xxxxx/soft/TDH-Client/hadoop/hadoop
HADOOP_MAPRED_HOME=/home/xxxxx/soft/TDH-Client/hadoop/hadoop-mapreduce
SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://dm47:8020//tmp/zzdb/sparklog/ -Dspark.history.fs.cleaner.enabled=true"
c. 修改spark-defaults.conf
cd /home/xxxxx/kdh/spark/conf
cp spark-defaults.conf.template spark-defaults.conf
vi /home/xxxxx/kdh/spark/conf/spark-defaults.conf
-------------------------------------------------------------------------------------------
spark.eventLog.enabled true
spark.eventLog.dir hdfs:///tmp/zzdb/sparklog
spark.eventLog.compress true
spark.yarn.historyServer dm48:19888
spark.yarn.jars hdfs:///tmp/zzdb/spark/jars/*.jar
d. 修改日志级别
cd /home/xxxxx/kdh/spark/conf
cp log4j2.properties.template log4j2.properties
e. 依赖包(TDH-Client里寻找)
cp guardian-common-guardian-3.1.0.jar /home/xxxxx/kdh/spark/jars/
cp yarn-plugin-transwarp-6.2.0.jar /home/xxxxx/kdh/spark/jars/
hadoop fs -mkdir -p /tmp/zzdb/spark/jars
cd /home/xxxxx/kdh/spark/jars
hadoop fs -put * /tmp/zzdb/spark/jars/
f. 其它
hadoop fs -mkdir -p /tmp/zzdb/sparklog
4. 启动history server
cd /home/xxxxx/kdh/spark/sbin
./start-history-server.sh
5. 测试
a. pi任务
-- base环境是python3.6
conda activate base
./spark-submit \
--master yarn \
--principal yarn@TDH \
--keytab /home/xxxxx/soft/yarn.keytab \
/home/xxxxx/kdh/spark/examples/src/main/python/pi.py 30
b. spark-shell
sh ./spark-shell --master yarn --deploy-mode client --driver-memory 4g --executor-memory 4g --num-executors 2 --executor-cores 2 --principal yarn@TDH --keytab /home/xxxxx/soft/yarn.keytab
c. pi任务-找茬:指定了高版本的python,任务在yarn上显示SUCCEEDED,但实际没看到pi的输出,可能是眼神问题
conda activate pyspark_env
cd /home/xxxxx/kdh/spark/bin
./spark-submit \
--master yarn \
--conf 'spark.pyspark.driver.python=/software/anaconda3/envs/pyspark_env/bin/python' \
--conf 'spark.pyspark.python=/software/anaconda3/envs/pyspark_env/bin/python' \
--principal yarn@TDH \
--keytab /home/xxxxx/soft/yarn.keytab \
/home/xxxxx/kdh/spark/examples/src/main/python/pi.py 30
d.提交wordcount - cluster模式
conda activate base
cd /home/kangwenqi/kdh/spark/bin
./spark-submit \
--master yarn \
--deploy-mode cluster \
--principal yarn@TDH \
--keytab /home/kangwenqi/soft/yarn.keytab \
/home/kangwenqi/workspace/pyspark_learn/02_pyspark_core/main/02_Wordcount_hdfs_yarn_cluster.py
四、地址查看:
五、幕后花絮
1. 更换yarn pod的java版本,原版本是1.7,换为1.8
[root@dm46 ~]# docker images | grep yarn
dm46:5000/transwarp/yarn transwarp-6.2.1-final cb9ccbe898b6 3 years ago 2.22GB
transwarp/yarn transwarp-6.2.1-final cb9ccbe898b6 3 years ago 2.22GB
[root@dm46 ~]#
docker run -id dm46:5000/transwarp/yarn:transwarp-6.2.1-final bash
docker ps -a | grep yarn | grep bash
docker exec -it d0f513cd0780 bash
mv /usr/java/jdk1.7.0_71 /usr/java/jdk1.7.0_71-bak
mv /usr/java/jdk1.8.0_25 /usr/java/jdk1.7.0_71
docker tag dm46:5000/transwarp/yarn:transwarp-6.2.1-final dm46:5000/transwarp/yarn:transwarp-6.2.1-final-jdk17bak
docker commit d0f513cd0780 dm46:5000/transwarp/yarn:transwarp-6.2.1-final
docker push dm46:5000/transwarp/yarn:transwarp-6.2.1-final
2. pod中找不到dm45(找到就见鬼了)
tdh pod在宿主机上映射的hosts:/etc/transwarp/conf/hosts
3. yarn一直报:Operation category READ is not supported in state standby
但是感觉也没有太大影响,加上了以下语句,感觉没什么卵用
export SPARK_MASTER_HOST=dm47
暂时和解了吧
4. 失误,检查晚了:版本不对,一切白费!!!
之所以放到最后,是因为这是我走过的弯路,不是一开始,都是正确的。
(base) [root@dm45 bin]# hadoop version
Hadoop 2.7.2-transwarp-6.2.0
Subversion http://xxx:10080/hadoop/hadoop-2.7.2-transwarp.git -r f31230971c2a36e77e4886e0f621366826cec3a3
Compiled by jenkins on 2019-07-27T11:33Z
Compiled with protoc 2.5.0
From source with checksum 42cb923f1631e3c548d6b7e572aa6962
This command was run using /home/xxxxx/soft/TDH-Client/hadoop/hadoop/hadoop-common-2.7.2-transwarp-6.2.0.jar
正解:https://dlcdn.apache.org/spark/spark-3.2.3/spark-3.2.3-bin-hadoop2.7.tgz
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。