spark-on-hive环境,hive-on-spark环境配置_spark包中指定hive_home

不管是on-hive还是on-spark，都需要配置hive得metastore服务

①配置hive

<property>
    <name>hive.server2.thrift.port</name>
     <value>10000</value>
</property>
 
    	<property>
       		<name>hive.server2.thrift.bind.host</name>
       		<value>localhost</value>
     	</property>
 
 <property>
        　　　　<name>hive.metastore.uris</name>
        　　　　<value>thrift://localhost:9083</value>
        　　</property>

启动：

nohup  hive --service metastore &
nohup  hive --service hiveserver2 &

spark-on-hive:

spark中如果不配置hive环境是无法读取hive表数据的，如下配置：

①如果你的hive元数据存在mysql，把mysql-connector-java-8.0.18.jar复制到spark的jars中

②在spark的spark-env.sh中添加：

export HIVE_HOME=/root/bigdata/hive-2.3.6
export HIVE_CONF_DIR=${HIVE_HOME}/conf
export SPARK_CLASSPATH=$HIVE_HOME/lib:$SPARK_CLASSPATH

③将hive中conf目录中的hive-site.xml复制到spark的conf目录中

hive-on-spark

①配置hive得hive-site.xml

<property>
  <name>hive.execution.engine</name>
  <value>spark</value>
</property>
<property>
  <name>hive.enable.spark.execution.engine</name>
  <value>true</value>
</property>
 
<property>
  <name>spark.home</name>
  <value>/home/work/spark-2.3.0</value>
</property>
<property>
  <name>spark.master</name>
  <value>spark://vm10-38-248-149.ksc.com:7077</value>
</property>
<property>
  <name>spark.serializer</name>
  <value>org.apache.spark.serializer.KryoSerializer</value>
</property>
 
 
<property>
  <name>spark.executor.memeory</name>
  <value>5g</value>
</property>
<property>
  <name>spark.driver.memeory</name>
  <value>4g</value>
</property>
 
<property>
  <name>spark.executor.cores</name>
  <value>6</value>
</property>
<property>
  <name>spark.executor.instances</name>
  <value>4</value>
</property>
<property>
  <name>spark.sql.shuffle.partitions</name>
  <value>200</value>
</property>
 
 
 
<property>
  <name>spark.executor.extraJavaOptions</name>
  <value>-XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"</value>
</property>
<property>
  <name>hive.server2.enable.doAs</name>
  <value>false</value>
</property>
<property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>
</configuration>