赞
踩
- CDH集群执行spark sql,出现下面错误:
-
- ERROR ApplicationMaster:94 - User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
- java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
一、在节点查看linux下
/opt/cloudera/parcels/CDH/lib/hive/bin hive能否正常执行。如不正常,处理hive问题。
二、上述正常,
处理步骤:
1.查看 /opt/cloudera/parcels/CDH/lib/spark/jars 下是否有 spark-hive_2.11-2.4.0-cdh6.3.1.jar,及权限。
2.查看/opt/cloudera/parcels/CDH/lib/spark/conf 下是否有 yarn-conf 配置文件,及权限。
3.查看/etc/spark 是否配置conf 、conf.cloudera.spark_on_yarn
4. yarn-conf 下包括
core-site.xml
hadoop-env.sh
hbase-env.sh
hbase-site.xml
hdfs-site.xml
jaas.conf
mapred-site.xml
ssl-client.xml
topology.map
topology.py
yarn-site.xml
5.是否创建软连接 :ln -s /etc/spark/conf.cloudera.spark_on_yarn spark-conf。
6.查看 /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/etc/spark/conf.dist 下
是否配置
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:$HADOOP_CONF_DIR"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-hdfs/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-hdfs/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-mapreduce/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-mapreduce/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-yarn/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-yarn/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hive/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/flume-ng/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/parquet/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/avro/lib/*"
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。