cdh6.3.2中2种spark命令行读取hive表数据方法_spark操作cdh版本hive

前提

将hive配置文件hive-site.xml放到spark的conf目录下

cp /etc/hive/conf/hive-site.xml /etc/spark/conf
1

在页面重启spark

1.sparksession

spark-shell

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://agent01:4040
Spark context available as 'sc' (master = yarn, app id = application_1659692413492_0026).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0-cdh6.3.2
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.sql("show databases").show();
+------------+
|databaseName|
+------------+
|     default|
|  kylin_hive|
|      testdb|
+------------+


scala>

scala> spark.sql("use testdb").show();
++
||
++
++


scala> spark.sql("show tables").show();
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
|  testdb|     test|      false|
|  testdb|    test2|      false|
+--------+---------+-----------+


scala> spark.sql("select * from testdb.test").show();
+---+----+
| id|name|
+---+----+
|  1|张三|
+---+----+


scala>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

2.hivecontext

spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/08/09 14:39:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark context Web UI available at http://agent02:4041
Spark context available as 'sc' (master = yarn, app id = application_1659692413492_0027).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0-cdh6.3.2
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.

scala>
scala> import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.HiveContext

scala> val hiveContext = new HiveContext(sc)
warning: there was one deprecation warning; re-run with -deprecation for details
hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@6aa27145

scala> hiveContext.sql("show databases").show()
+------------+
|databaseName|
+------------+
|     default|
|  kylin_hive|
|      testdb|
+------------+

scala> hiveContext.sql("use testdb").show()
++
||
++
++

scala> hiveContext.sql("show tables").show()
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
|  testdb|     test|      false|
|  testdb|    test2|      false|
+--------+---------+-----------+

scala> hiveContext.sql("select * from testdb.test").show()
+---+----+
| id|name|
+---+----+
|  1|张三|
+---+----+

scala>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58