java.lang.VerifyError: class org.apache.spark.sql.hive.HiveExternalCatalog overrides final method al_lorg/apache/spark/sql/catalyst/catalog/externalcat

作者：2023面试高手 | 2024-02-23 07:11:09

踩

lorg/apache/spark/sql/catalyst/catalog/externalcatalog;

java.lang.VerifyError: class org.apache.spark.sql.hive.HiveExternalCatalog overrides final method alterDatabase.(Lorg/apache/spark/sql/catalyst/catalog/CatalogDatabase;)V
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:53)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
at org.apache.spark.sql.hive.HiveSessionStateBuilder

a n o n $ 1. < i n i t > (H i v e S e s s i o n S t a t e B u i l d e r . s c a l a : 69) a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e S e s s i o n S t a t e B u i l d e r . a n a l y z e r (H i v e S e s s i o n S t a t e B u i l d e r . s c a l a : 69) a t o r g . a p a c h e . s p a r k . s q l . i n t e r n a l . B a s e S e s s i o n S t a t e B u i l d e r

$anon$1.<init>(HiveSessionStateBuilder.scala:69) at org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:69) at org.apache.spark.sql.internal.BaseSessionStateBuilder$ anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
at org.apache.spark.sql.internal.BaseSessionStateBuilder

a n o n f u n $ b u i l d $ 2. a p p l y (B a s e S e s s i o n S t a t e B u i l d e r . s c a l a : 293) a t o r g . a p a c h e . s p a r k . s q l . i n t e r n a l . S e s s i o n S t a t e . a n a l y z e r $ l z y c o m p u t e (S e s s i o n S t a t e . s c a l a : 79) a t o r g . a p a c h e . s p a r k . s q l . i n t e r n a l . S e s s i o n S t a t e . a n a l y z e r (S e s s i o n S t a t e . s c a l a : 79) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . Q u e r y E x e c u t i o n . a n a l y z e d $ l z y c o m p u t e (Q u e r y E x e c u t i o n . s c a l a : 57) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . Q u e r y E x e c u t i o n . a n a l y z e d (Q u e r y E x e c u t i o n . s c a l a : 55) a t o r g . a p a c h e . s p a r k . s q l . e x e c u t i o n . Q u e r y E x e c u t i o n . a s s e r t A n a l y z e d (Q u e r y E x e c u t i o n . s c a l a : 47) a t o r g . a p a c h e . s p a r k . s q l . D a t a s e t $ . o f R o w s (D a t a s e t . s c a l a : 74) a t o r g . a p a c h e . s p a r k . s q l . S p a r k S e s s i o n . b a s e R e l a t i o n T o D a t a F r a m e (S p a r k S e s s i o n . s c a l a : 428) a t o r g . a p a c h e . s p a r k . s q l . D a t a F r a m e R e a d e r . l o a d V 1 S o u r c e (D a t a F r a m e R e a d e r . s c a l a : 233) a t o r g . a p a c h e . s p a r k . s q l . D a t a F r a m e R e a d e r . l o a d (D a t a F r a m e R e a d e r . s c a l a : 227) a t o r g . a p a c h e . s p a r k . s q l . D a t a F r a m e R e a d e r . l o a d (D a t a F r a m e R e a d e r . s c a l a : 174) a t c o m . d a t a b r i c k s . s p a r k . a v r o . p a c k a g e $ A v r o D a t a F r a m e R e a d e r

$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293) at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:79) at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:79) at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57) at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55) at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74) at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:428) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:233) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:174) at com.databricks.spark.avro.package$AvroDataFrameReader$ anonfun$avro$2.apply(package.scala:34)
at com.databricks.spark.avro.package$AvroDataFrameReader$$anonfun$avro$2.apply(package.scala:34)

报错是因为倒入错误的pom文件，spark-hive 和spark-sql下面都有SparkSession类，因此类加载器验证错误。pom中删除spark-sql就可以了，使用spark-hive 就可以对hive进行读写

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/2023面试高手/article/detail/134601