DataSophon组件之Spark升级_datasophon custom.spark.defaults.conf

1.解压新旧组件安装包

安装包:

Index of /apache

cd  /opt/datasophon/DDP/packages

tar  zxvf  spark-3.4.1.tar.gz  -C  /opt/datasophon/DDP/packages

tar  zxvf  spark-3.1.3.tar.gz  -C  /opt/datasophon/DDP/packages

2.修改安装包中文件和目录

cd  /opt/datasophon/DDP/packages

chown  -R root:root  spark-3.4.1

cd  /opt/datasophon/DDP/packages/spark-3.4.1

2.1.在hadoop家目录中新建日志和进程目录

mkdir  logs  pid  work

2.2.拷贝组件包

scp -r  /opt/datasophon/DDP/packages/spark-3.1.3/jmx                /opt/datasophon/DDP/packages/spark-3.4.1

scp -r  /opt/datasophon/DDP/packages/spark-3.1.3/carbonlib                /opt/datasophon/DDP/packages/spark-3.4.1

2.3拷贝配置文件

mv /opt/datasophon/DDP/packages/spark-3.4.1/sbin/spark-daemon.sh  /opt/datasophon/DDP/packages/spark-3.4.1/sbin/spark-daemon.sh.bak

scp     /opt/datasophon/DDP/packages/spark-3.1.3/conf/ranger-spark-security.xml  /opt/datasophon/DDP/packages/spark-3.4.1/conf

scp      /opt/datasophon/DDP/packages/spark-3.1.3/conf/spark-defaults.conf  /opt/datasophon/DDP/packages/spark-3.4.1/conf

scp      /opt/datasophon/DDP/packages/spark-3.1.3/conf/carbon.properties  /opt/datasophon/DDP/packages/spark-3.4.1/conf

scp      /opt/datasophon/DDP/packages/spark-3.1.3/conf/spark-env.sh  /opt/datasophon/DDP/packages/spark-3.4.1/conf

scp     /opt/datasophon/DDP/packages/spark-3.1.3/conf/ranger-spark-audit.xml  /opt/datasophon/DDP/packages/spark-3.4.1/conf

scp      /opt/datasophon/DDP/packages/spark-3.1.3/jars/submarine-spark-security-0.6.0.jar  /opt/datasophon/DDP/packages/spark-3.4.1/jars

2.4创建软连接

cd   /opt/datasophon/DDP/packages/spark-3.4.1/conf/

ln -s   /opt/datasophon/hadoop-3.3.6/etc/hadoop/core-site.xml    /opt/datasophon/DDP/packages/spark-3.4.1/conf/core-site.xml  

ln -s  /opt/datasophon/hadoop-3.3.6/etc/hadoop/hdfs-site.xml     /opt/datasophon/DDP/packages/spark-3.4.1/conf/hdfs-site.xml

ln -s  /opt/datasophon/hive-3.1.3/conf/hive-site.xml            /opt/datasophon/DDP/packages/spark-3.4.1/conf/hive-site.xml

ln -s  /opt/datasophon/hadoop-3.3.6/etc/hadoop/yarn-site.xml     /opt/datasophon/DDP/packages/spark-3.4.1/conf/yarn-site.xml  

2.5修改安装配置参数(若改请忽略)

vi  /opt/datasophon/DDP/packages/spark-3.4.1/conf/spark-env.sh

3.重新生成安装包

3.1重新打包

cd  /opt/datasophon/DDP/packages

mv spark-3.4.1.tar.gz  spark-3.4.1.tar.gz.bak   -- 备份

tar czf  spark-3.4.1.tar.gz   spark-3.4.1      -- 重新打包

3.2生成加密码

md5sum spark-3.4.1.tar.gz

3.3生成md5加密文件

echo '8d9ee35827dd3fbb758bf592c27bdbd4' > spark-3.4.1.tar.gz.md5

4.删除已装的组件包Spark(ALL)

rm -rf  /opt/datasophon/spark*

5.修改service_ddl.json

cd   /opt/datasophon/DDP/packages/datasophon-manager-1.1.3/conf/meta/DDP-1.1.3/

vi   /opt/datasophon/DDP/packages/datasophon-manager-1.1.3/conf/meta/DDP-1.1.3/SPARK3/service_ddl.json

*datasophon-manager-1.1.3 为当前的datasohon安装版本,可能多处地方需要修改 

6.修改env环境变量(ALL)

6.1修改环境变量配置参数(ALL)

vi   /etc/profile.d/datasophon-env.sh

grep -r "spark-3.1.3" /opt/datasophon/DDP/packages/

grep -r "spark-3.1.3" /etc/profile.d/

echo  ${SPARK_HOME}

6.2生效环境变量

  source  /etc/profile

  echo  ${SPARK_HOME}

6.3重启服务器

reboot

echo ${SPARK_HOME}

*不重启SPARK_HOME的环境变量安装时还是不生效

6.4重启manager服务

  rm -rf   /opt/datasophon/DDP/packages/datasophon-manager-1.1.3/logs/*

  /opt/datasophon/DDP/packages/datasophon-manager-1.1.3/bin/datasophon-api.sh restart api

cd /opt/datasophon/DDP/packages/datasophon-manager-1.1.3/logs

查看相应的表配置数据

   select * from t_ddh_frame_service;

   select * from t_ddh_frame_service_role;

7.重新安装服务

8.测试验证

spark-submit --master yarn --deploy-mode client --queue default --num-executors 2 --driver-memory 1g --class org.apache.spark.examples.SparkPi  /opt/datasophon/spark-3.4.1/examples/jars/spark-examples_2.13-3.4.1.jar  1

spark-submit --master yarn --deploy-mode cluster --queue default --num-executors 2 --driver-memory 1g --class org.apache.spark.examples.SparkPi  /opt/datasophon/spark-3.4.1/examples/jars/spark-examples_2.13-3.4.1.jar  1

集群提交，日志需要在yarn上查看