当前位置:   article > 正文

Unable to instantiate SparkSession with Hive support because Hive classes are not found.

unable to instantiate sparksession with hive support because hive classes ar
  1. CDH集群执行spark sql,出现下面错误:
  2. ERROR ApplicationMaster:94 - User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
  3. java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.

一、在节点查看linux下

/opt/cloudera/parcels/CDH/lib/hive/bin  hive能否正常执行。如不正常,处理hive问题。

二、上述正常,

处理步骤:

1.查看 /opt/cloudera/parcels/CDH/lib/spark/jars 下是否有 spark-hive_2.11-2.4.0-cdh6.3.1.jar,及权限。

2.查看/opt/cloudera/parcels/CDH/lib/spark/conf 下是否有 yarn-conf 配置文件,及权限。

3.查看/etc/spark 是否配置conf 、conf.cloudera.spark_on_yarn

4. yarn-conf 下包括

core-site.xml
hadoop-env.sh
hbase-env.sh
hbase-site.xml
hdfs-site.xml
jaas.conf
mapred-site.xml
ssl-client.xml
topology.map
topology.py
yarn-site.xml

5.是否创建软连接 :ln -s /etc/spark/conf.cloudera.spark_on_yarn  spark-conf。

6.查看 /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/etc/spark/conf.dist 下

是否配置

SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:$HADOOP_CONF_DIR"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-hdfs/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-hdfs/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-mapreduce/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-mapreduce/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-yarn/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hadoop-yarn/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/hive/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/flume-ng/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/parquet/lib/*"
SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/usr/lib/avro/lib/*"

 

 

 

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/代码探险家/article/detail/869703
推荐阅读
相关标签
  

闽ICP备14008679号