devbox
运维做开发
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

spark sql 访问hive数据时找不mysql的解决方法_spark-sql 查询hive 报错unable to fetch table

作者：运维做开发 | 2024-06-30 19:16:49

赞

踩

spark-sql 查询hive 报错unable to fetch table

1现象：

错误如下：

  1. org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table SOGOUQ1
  2.         at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:958)
  3.         at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:924)
  4.         at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:54)
  5.         at org.apache.spark.sql.hive.HiveContext$anon$1.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$super$lookupRelation(HiveContext.scala:143)
  6.         at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$anonfun$lookupRelation$3.apply(Catalog.scala:96)
  7.         at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$anonfun$lookupRelation$3.apply(Catalog.scala:96)
  8.         at scala.Option.getOrElse(Option.scala:120)
  9.         at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:96)
  10.         at org.apache.spark.sql.hive.HiveContext$anon$1.lookupRelation(HiveContext.scala:149)
  11.         at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$anonfun$apply$2.applyOrElse(Analyzer.scala:83)
  12.         at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$anonfun$apply$2.applyOrElse(Analyzer.scala:81)
  13.         at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:165)
  14.         at org.apache.spark.sql.catalyst.trees.TreeNode$anonfun$4.apply(TreeNode.scala:183)
  15.         at scala.collection.Iterator$anon$11.next(Iterator.scala:328)
  16.         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
  17.         at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
  18.         at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
  19.         at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
  20.         at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
  21.         at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
  22.         at scala.collection.AbstractIterator.to(Iterator.scala:1157)
  23.         at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
  24.         at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
  25.         at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
  26.         at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
  27.         at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:212)
  28.         at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:168)
  29.         at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:156)
  30.         at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:81)
  31.         at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:80)
  32.         at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1$anonfun$apply$2.apply(RuleExecutor.scala:62)
  33.         at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1$anonfun$apply$2.apply(RuleExecutor.scala:60)
  34.         at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
  35.         at scala.collection.immutable.List.foldLeft(List.scala:84)
  36.         at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1.apply(RuleExecutor.scala:60)
  37.         at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1.apply(RuleExecutor.scala:52)
  38.         at scala.collection.immutable.List.foreach(List.scala:318)
  39.         at org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:52)
  40.         at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:313)
  41.         at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:313)
  42.         at org.apache.spark.sql.hive.HiveContext$QueryExecution.optimizedPlan$lzycompute(HiveContext.scala:248)
  43.         at org.apache.spark.sql.hive.HiveContext$QueryExecution.optimizedPlan(HiveContext.scala:247)
  44.         at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:316)
  45.         at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:316)
  46.         at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:319)
  47.         at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:319)
  48.         at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:405)
  49.         at $iwC$iwC$iwC$iwC.<init>(<console>:15)
  50.         at $iwC$iwC$iwC.<init>(<console>:20)
  51.         at $iwC$iwC.<init>(<console>:22)
  52.         at $iwC.<init>(<console>:24)
  53.         at <init>(<console>:26)
  54.         at .<init>(<console>:30)
  55.         at .<clinit>(<console>)
  56.         at .<init>(<console>:7)
  57.         at .<clinit>(<console>)
  58.         at $print(<console>)
  59.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  60.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  61.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  62.         at java.lang.reflect.Method.invoke(Method.java:606)
  63.         at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
  64.         at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
  65.         at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
  66.         at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
  67.         at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
  68.         at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
  69.         at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
  70.         at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
  71.         at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
  72.         at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
  73.         at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
  74.         at org.apache.spark.repl.SparkILoop$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
  75.         at org.apache.spark.repl.SparkILoop$anonfun$process$1.apply(SparkILoop.scala:884)
  76.         at org.apache.spark.repl.SparkILoop$anonfun$process$1.apply(SparkILoop.scala:884)
  77.         at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
  78.         at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
  79.         at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
  80.         at org.apache.spark.repl.Main$.main(Main.scala:31)
  81.         at org.apache.spark.repl.Main.main(Main.scala)
  82.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  83.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  84.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  85.         at java.lang.reflect.Method.invoke(Method.java:606)
  86.         at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
  87.         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
  88.         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  89. Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
  90.         at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1212)
  91.         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
  92.         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
  93.         at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
  94.         at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
  95.         at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:950)
  96.         ... 86 more
  97. Caused by: java.lang.reflect.InvocationTargetException
  98.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  99.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  100.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  101.         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  102.         at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
  103.         ... 91 more
  104. Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
  105. NestedThrowables:
  106. java.lang.reflect.InvocationTargetException
  107.         at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
  108.         at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:781)
  109.         at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
  110.         at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
  111.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  112.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  113.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  114.         at java.lang.reflect.Method.invoke(Method.java:606)
  115.         at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
  116.         at java.security.AccessController.doPrivileged(Native Method)
  117.         at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
  118.         at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
  119.         at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
  120.         at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
  121.         at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:275)
  122.         at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:304)
  123.         at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:234)
  124.         at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:209)
  125.         at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
  126.         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
  127.         at org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:64)
  128.         at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:73)
  129.         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:415)
  130.         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402)
  131.         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
  132.         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
  133.         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286)
  134.         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
  135.         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
  136.         at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
  137.         at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
  138.         ... 96 more
  139. Caused by: java.lang.reflect.InvocationTargetException
  140.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  141.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  142.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  143.         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  144.         at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
  145.         at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)
  146.         at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:281)
  147.         at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:239)
  148.         at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:292)
  149.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  150.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  151.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  152.         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  153.         at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
  154.         at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
  155.         at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1069)
  156.         at org.datanucleus.NucleusContext.initialise(NucleusContext.java:359)
  157.         at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:768)
  158.         ... 125 more
  159. Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BoneCP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
  160.         at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:237)
  161.         at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:110)
  162.         at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
  163.         ... 143 more
  164. Caused by: org.datanucleus.store.rdbms.datasource.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
  165.         at org.datanucleus.store.rdbms.datasource.AbstractDataSourceFactory.loadDriver(AbstractDataSourceFactory.java:58)
  166.         at org.datanucleus.store.rdbms.datasource.BoneCPDataSourceFactory.makePooledDataSource(BoneCPDataSourceFactory.java:61)
  167.         at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:217)
  168.         ... 145 more

我尝试着在classpath中加n入mysql的驱动仍不行


解决方法：在启动的时候加入参数--driver-class中加入mysql 驱动

  1. [hadoop@master spark-1.0.1-bin-hadoop2]$ bin/spark-shell --driver-class-path lib/mysql-connector-java-5.1.30-bin.jar 
  2. Spark assembly has been built with Hive, including Datanucleus jars on classpath
  3. 14/08/14 16:07:14 INFO spark.SecurityManager: Changing view acls to: hadoop
  4. 14/08/14 16:07:14 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop)
  5. 14/08/14 16:07:15 INFO spark.HttpServer: Starting HTTP Server
  6. 14/08/14 16:07:15 INFO server.Server: jetty-8.y.z-SNAPSHOT
  7. 14/08/14 16:07:15 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:43007
  8. Welcome to
  9.       ____              __
  10.      / __/__  ___ _____/ /__
  11.     _ / _ / _ `/ __/  '_/
  12.    /___/ .__/_,_/_/ /_/_   version 1.0.1
  13.       /_/
  15. Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_65)
  16. Type in expressions to have them evaluated.
  17. Type :help for more information.
  18. 14/08/14 16:07:25 INFO spark.SecurityManager: Changing view acls to: hadoop
  19. 14/08/14 16:07:25 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop)
  20. 14/08/14 16:07:26 INFO slf4j.Slf4jLogger: Slf4jLogger started
  21. 14/08/14 16:07:26 INFO Remoting: Starting remoting
  22. 14/08/14 16:07:27 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@master:60674]
  23. 14/08/14 16:07:27 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@master:60674]
  24. 14/08/14 16:07:27 INFO spark.SparkEnv: Registering MapOutputTracker
  25. 14/08/14 16:07:27 INFO spark.SparkEnv: Registering BlockManagerMaster
  26. 14/08/14 16:07:27 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140814160727-d7c2
  27. 14/08/14 16:07:27 INFO storage.MemoryStore: MemoryStore started with capacity 297.0 MB.
  28. 14/08/14 16:07:27 INFO network.ConnectionManager: Bound socket to port 57661 with id = ConnectionManagerId(master,57661)
  29. 14/08/14 16:07:27 INFO storage.BlockManagerMaster: Trying to register BlockManager
  30. 14/08/14 16:07:27 INFO storage.BlockManagerInfo: Registering block manager master:57661 with 297.0 MB RAM
  31. 14/08/14 16:07:27 INFO storage.BlockManagerMaster: Registered BlockManager
  32. 14/08/14 16:07:27 INFO spark.HttpServer: Starting HTTP Server
  33. 14/08/14 16:07:27 INFO server.Server: jetty-8.y.z-SNAPSHOT
  34. 14/08/14 16:07:27 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:41277
  35. 14/08/14 16:07:27 INFO broadcast.HttpBroadcast: Broadcast server started at http://192.168.200.182:41277
  36. 14/08/14 16:07:27 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-b5c432fe-e4fb-45d0-8514-ba50fea348c9
  37. 14/08/14 16:07:27 INFO spark.HttpServer: Starting HTTP Server
  38. 14/08/14 16:07:27 INFO server.Server: jetty-8.y.z-SNAPSHOT
  39. 14/08/14 16:07:27 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:46317
  40. 14/08/14 16:07:38 INFO server.Server: jetty-8.y.z-SNAPSHOT
  41. 14/08/14 16:07:38 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
  42. 14/08/14 16:07:38 INFO ui.SparkUI: Started SparkUI at http://master:4040
  43. 14/08/14 16:07:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  44. 14/08/14 16:07:39 INFO executor.Executor: Using REPL class URI: http://192.168.200.182:43007
  45. 14/08/14 16:07:40 INFO repl.SparkILoop: Created spark context..
  46. Spark context available as sc.
  48. scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
  49. 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
  50. 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
  51. 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
  52. 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
  53. 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
  54. 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
  55. 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
  56. hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@30667a81
  58. scala> 
  60. scala> hiveContext.hql("Select count(*) from SOGOUQ1").collect().foreach(println)
  61. 14/08/14 16:08:15 INFO parse.ParseDriver: Parsing command: Select count(*) from SOGOUQ1
  62. 14/08/14 16:08:15 INFO parse.ParseDriver: Parse Completed
  63. 14/08/14 16:08:15 INFO analysis.Analyzer: Max iterations (2) reached for batch MultiInstanceRelations
  64. 14/08/14 16:08:15 INFO analysis.Analyzer: Max iterations (2) reached for batch CaseInsensitiveAttributeReferences
  65. 14/08/14 16:08:16 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
  66. 14/08/14 16:08:16 INFO metastore.ObjectStore: ObjectStore, initialize called
  67. 14/08/14 16:08:16 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
  68. 14/08/14 16:08:16 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
  69. 14/08/14 16:08:17 WARN bonecp.BoneCPConfig: Max Connections < 1. Setting to 20
  70. 14/08/14 16:08:28 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
  71. 14/08/14 16:08:28 INFO metastore.ObjectStore: Initialized ObjectStore
  72. 14/08/14 16:08:30 WARN bonecp.BoneCPConfig: Max Connections < 1. Setting to 20
  73. 14/08/14 16:08:31 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=SOGOUQ1
  74. 14/08/14 16:08:31 INFO HiveMetaStore.audit: ugi=hadoop        ip=unknown-ip-addr        cmd=get_table : db=default tbl=SOGOUQ1        
  75. 14/08/14 16:08:31 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
  76. 14/08/14 16:08:31 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
  77. 14/08/14 16:08:32 INFO analysis.Analyzer: Max iterations (2) reached for batch Check Analysis
  78. 14/08/14 16:08:32 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  79. 14/08/14 16:08:32 INFO storage.MemoryStore: ensureFreeSpace(391062) called with curMem=0, maxMem=311387750
  80. 14/08/14 16:08:32 INFO storage.MemoryStore: Block broadcast_0 stored as values to memory (estimated size 381.9 KB, free 296.6 MB)
  81. 14/08/14 16:08:32 INFO sql.SQLContext$anon$1: Max iterations (2) reached for batch Add exchange
  82. 14/08/14 16:08:32 INFO sql.SQLContext$anon$1: Max iterations (2) reached for batch Prepare Expressions
  83. 14/08/14 16:08:32 INFO spark.SparkContext: Starting job: collect at SparkPlan.scala:52
  84. 14/08/14 16:08:33 INFO mapred.FileInputFormat: Total input paths to process : 1
  85. 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Registering RDD 6 (mapPartitions at Exchange.scala:69)
  86. 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Got job 0 (collect at SparkPlan.scala:52) with 1 output partitions (allowLocal=false)
  87. 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Final stage: Stage 0(collect at SparkPlan.scala:52)
  88. 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 1)
  89. 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Missing parents: List(Stage 1)
  90. 14/08/14 16:08:34 INFO scheduler.DAGScheduler: Submitting Stage 1 (MapPartitionsRDD[6] at mapPartitions at Exchange.scala:69), which has no missing parents
  91. 14/08/14 16:08:34 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from Stage 1 (MapPartitionsRDD[6] at mapPartitions at Exchange.scala:69)
  92. 14/08/14 16:08:34 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
  93. 14/08/14 16:08:34 INFO scheduler.TaskSetManager: Starting task 1.0:0 as TID 0 on executor localhost: localhost (PROCESS_LOCAL)
  94. 14/08/14 16:08:34 INFO scheduler.TaskSetManager: Serialized task 1.0:0 as 4567 bytes in 12 ms
  95. 14/08/14 16:08:34 INFO executor.Executor: Running task ID 0
  96. 14/08/14 16:08:34 INFO storage.BlockManager: Found block broadcast_0 locally
  97. 14/08/14 16:08:34 INFO rdd.HadoopRDD: Input split: hdfs://master:9000/sougouq1/SogouQ1.txt:0+54375287
  98. 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
  99. 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
  100. 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
  101. 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
  102. 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
  103. 14/08/14 16:08:36 INFO executor.Executor: Serialized size of result for 0 is 776
  104. 14/08/14 16:08:36 INFO executor.Executor: Sending result for 0 directly to driver
  105. 14/08/14 16:08:36 INFO executor.Executor: Finished task ID 0
  106. 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Starting task 1.0:1 as TID 1 on executor localhost: localhost (PROCESS_LOCAL)
  107. 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Serialized task 1.0:1 as 4567 bytes in 2 ms
  108. 14/08/14 16:08:36 INFO executor.Executor: Running task ID 1
  109. 14/08/14 16:08:36 INFO storage.BlockManager: Found block broadcast_0 locally
  110. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Completed ShuffleMapTask(1, 0)
  111. 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Finished TID 0 in 1812 ms on localhost (progress: 1/2)
  112. 14/08/14 16:08:36 INFO rdd.HadoopRDD: Input split: hdfs://master:9000/sougouq1/SogouQ1.txt:54375287+54375287
  113. 14/08/14 16:08:36 INFO executor.Executor: Serialized size of result for 1 is 776
  114. 14/08/14 16:08:36 INFO executor.Executor: Sending result for 1 directly to driver
  115. 14/08/14 16:08:36 INFO executor.Executor: Finished task ID 1
  116. 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Finished TID 1 in 682 ms on localhost (progress: 2/2)
  117. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Completed ShuffleMapTask(1, 1)
  118. 14/08/14 16:08:36 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
  119. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Stage 1 (mapPartitions at Exchange.scala:69) finished in 2.492 s
  120. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: looking for newly runnable stages
  121. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: running: Set()
  122. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: waiting: Set(Stage 0)
  123. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: failed: Set()
  124. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Missing parents for Stage 0: List()
  125. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Submitting Stage 0 (MappedRDD[10] at map at SparkPlan.scala:52), which is now runnable
  126. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 0 (MappedRDD[10] at map at SparkPlan.scala:52)
  127. 14/08/14 16:08:36 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
  128. 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 2 on executor localhost: localhost (PROCESS_LOCAL)
  129. 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 4766 bytes in 0 ms
  130. 14/08/14 16:08:36 INFO executor.Executor: Running task ID 2
  131. 14/08/14 16:08:36 INFO storage.BlockManager: Found block broadcast_0 locally
  132. 14/08/14 16:08:36 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, targetRequestSize: 10066329
  133. 14/08/14 16:08:36 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
  134. 14/08/14 16:08:36 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote fetches in 10 ms
  135. 14/08/14 16:08:36 INFO executor.Executor: Serialized size of result for 2 is 1075
  136. 14/08/14 16:08:36 INFO executor.Executor: Sending result for 2 directly to driver
  137. 14/08/14 16:08:36 INFO executor.Executor: Finished task ID 2
  138. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Completed ResultTask(0, 0)
  139. 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Stage 0 (collect at SparkPlan.scala:52) finished in 0.163 s
  140. 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Finished TID 2 in 171 ms on localhost (progress: 1/1)
  141. 14/08/14 16:08:36 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
  142. 14/08/14 16:08:36 INFO spark.SparkContext: Job finished: collect at SparkPlan.scala:52, took 4.498212701 s
  143. [1000000]
然后执行成功，

总结：

1.spark的版本必须编译的时候加上了hive 1.0.0预编译版没有加入hive  1.0.1是含有hive的
2.需要将hive-site.xml这个配置文件放到spark conf下面
3.需要将mysql驱动加载到classpath中

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/运维做开发/article/detail/773691
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号