pyspark设置了环境变量,调用python函数就报错,求指点(已解决)_新版spark运行会报pythonfunction错误

作者：Cpp五条 | 2024-03-23 04:34:58

踩

新版spark运行会报pythonfunction错误

弱弱问下,我设置了环境变量,但是运行就报错,是哪里错了啊? 求指点~~拜谢

问题解决了,是python版本问题,版本高了,我降了版本就正常了谢谢大家

代码如下

from pyspark import SparkConf, SparkContext
import os
os.environ['PYSPARK_PYTHON'] = "E:/Python_code/pythonProject/.venv/Scripts/python.exe"
# 创建SparkConf类对象
conf = SparkConf().setMaster("local[*]").setAppName("test_spark_app")
# 基于SparkConf类对象创建SparkContext对象
sc = SparkContext(conf=conf)
# 打印spark版本信息
# print(sc.version)
rdd = sc.parallelize([1, 2, 3, 4])
print(rdd.collect())
print(rdd.map(lambda x: x * 10).collect())
# 停止SparkContext对象的运行
sc.stop()

报错信息:

24/02/22 09:36:38 ERROR Executor: Exception in task 5.0 in stage 1.0 (TID 25)
org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator

a n o n f u n $ 1. a p p l y O r E l s e (P y t h o n R u n n e r . s c a l a : 612) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . B a s e P y t h o n R u n n e r $ R e a d e r I t e r a t o r

$anonfun$1.applyOrElse(PythonRunner.scala:612)    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$ anonfun$1.applyOrElse(PythonRunner.scala:594)
   at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
   at org.apache.spark.api.python.PythonRunner

a n o n $ 3. r e a d (P y t h o n R u n n e r . s c a l a : 789) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . P y t h o n R u n n e r

$anon$3.read(PythonRunner.scala:789)    at org.apache.spark.api.python.PythonRunner$ anon$3.read(PythonRunner.scala:766)
   at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525)
   at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
   at scala.collection.Iterator.foreach(Iterator.scala:943)
   at scala.collection.Iterator.foreach$(Iterator.scala:943)
   at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
   at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
   at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
   at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
   at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
   at scala.collection.TraversableOnce.to(TraversableOnce.scala:366)
   at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364)
   at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
   at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
   at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
   at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
   at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
   at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
   at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
   at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1046)
   at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438)
   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
   at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
   at org.apache.spark.scheduler.Task.run(Task.scala:141)
   at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
   at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
   at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
   at java.io.DataInputStream.readInt(DataInputStream.java:392)
   at org.apache.spark.api.python.PythonRunner

a n o n $ 3. r e a d (P y t h o n R u n n e r . s c a l a : 774) . . . 32 m o r e 24 / 02 / 2209 : 36 : 38 W A R N T a s k S e t M a n a g e r : L o s t t a s k 5.0 i n s t a g e 1.0 (T I D 25) (G Y G - L Z C e x e c u t o r d r i v e r) : o r g . a p a c h e . s p a r k . S p a r k E x c e p t i o n : P y t h o n w o r k e r e x i t e d u n e x p e c t e d l y (c r a s h e d) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . B a s e P y t h o n R u n n e r $ R e a d e r I t e r a t o r

$anon$3.read(PythonRunner.scala:774) ... 32 more 24/02/22 09:36:38 WARN TaskSetManager: Lost task 5.0 in stage 1.0 (TID 25) (GYG-LZC executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$ anonfun$1.applyOrElse(PythonRunner.scala:612)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator

a n o n f u n $ 1. a p p l y O r E l s e (P y t h o n R u n n e r . s c a l a : 594) a t s c a l a . r u n t i m e . A b s t r a c t P a r t i a l F u n c t i o n . a p p l y (A b s t r a c t P a r t i a l F u n c t i o n . s c a l a : 38) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . P y t h o n R u n n e r

$anonfun$1.applyOrElse(PythonRunner.scala:594) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38) at org.apache.spark.api.python.PythonRunner$ anon$3.read(PythonRunner.scala:789)
at org.apache.spark.api.python.PythonRunner

a n o n $ 3. r e a d (P y t h o n R u n n e r . s c a l a : 766) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . B a s e P y t h o n R u n n e r $ R e a d e r I t e r a t o r . h a s N e x t (P y t h o n R u n n e r . s c a l a : 525) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . h a s N e x t (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 37) a t s c a l a . c o l l e c t i o n . I t e r a t o r . f o r e a c h (I t e r a t o r . s c a l a : 943) a t s c a l a . c o l l e c t i o n . I t e r a t o r . f o r e a c h $ (I t e r a t o r . s c a l a : 943) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . f o r e a c h (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 28) a t s c a l a . c o l l e c t i o n . g e n e r i c . G r o w a b l e . $ p l u s $ p l u s $ e q (G r o w a b l e . s c a l a : 62) a t s c a l a . c o l l e c t i o n . g e n e r i c . G r o w a b l e . $ p l u s $ p l u s $ e q $ (G r o w a b l e . s c a l a : 53) a t s c a l a . c o l l e c t i o n . m u t a b l e . A r r a y B u f f e r . $ p l u s $ p l u s $ e q (A r r a y B u f f e r . s c a l a : 105) a t s c a l a . c o l l e c t i o n . m u t a b l e . A r r a y B u f f e r . $ p l u s $ p l u s $ e q (A r r a y B u f f e r . s c a l a : 49) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o (T r a v e r s a b l e O n c e . s c a l a : 366) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o $ (T r a v e r s a b l e O n c e . s c a l a : 364) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . t o (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 28) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o B u f f e r (T r a v e r s a b l e O n c e . s c a l a : 358) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o B u f f e r $ (T r a v e r s a b l e O n c e . s c a l a : 358) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . t o B u f f e r (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 28) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o A r r a y (T r a v e r s a b l e O n c e . s c a l a : 345) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o A r r a y $ (T r a v e r s a b l e O n c e . s c a l a : 339) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . t o A r r a y (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 28) a t o r g . a p a c h e . s p a r k . r d d . R D D . $ a n o n f u n $ c o l l e c t $ 2 (R D D . s c a l a : 1046) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . $ a n o n f u n $ r u n J o b $ 5 (S p a r k C o n t e x t . s c a l a : 2438) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . R e s u l t T a s k . r u n T a s k (R e s u l t T a s k . s c a l a : 93) a t o r g . a p a c h e . s p a r k . T a s k C o n t e x t . r u n T a s k W i t h L i s t e n e r s (T a s k C o n t e x t . s c a l a : 161) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . T a s k . r u n (T a s k . s c a l a : 141) a t o r g . a p a c h e . s p a r k . e x e c u t o r . E x e c u t o r $ T a s k R u n n e r . $ a n o n f u n $ r u n $ 4 (E x e c u t o r . s c a l a : 620) a t o r g . a p a c h e . s p a r k . u t i l . S p a r k E r r o r U t i l s . t r y W i t h S a f e F i n a l l y (S p a r k E r r o r U t i l s . s c a l a : 64) a t o r g . a p a c h e . s p a r k . u t i l . S p a r k E r r o r U t i l s . t r y W i t h S a f e F i n a l l y $ (S p a r k E r r o r U t i l s . s c a l a : 61) a t o r g . a p a c h e . s p a r k . u t i l . U t i l s $ . t r y W i t h S a f e F i n a l l y (U t i l s . s c a l a : 94) a t o r g . a p a c h e . s p a r k . e x e c u t o r . E x e c u t o r $ T a s k R u n n e r . r u n (E x e c u t o r . s c a l a : 623) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r . r u n W o r k e r (T h r e a d P o o l E x e c u t o r . j a v a : 1149) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r $ W o r k e r . r u n (T h r e a d P o o l E x e c u t o r . j a v a : 624) a t j a v a . l a n g . T h r e a d . r u n (T h r e a d . j a v a : 748) C a u s e d b y : j a v a . i o . E O F E x c e p t i o n a t j a v a . i o . D a t a I n p u t S t r e a m . r e a d I n t (D a t a I n p u t S t r e a m . j a v a : 392) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . P y t h o n R u n n e r

$anon$3.read(PythonRunner.scala:766) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1046) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.spark.api.python.PythonRunner$ anon$3.read(PythonRunner.scala:774)
... 32 more

24/02/22 09:36:38 ERROR TaskSetManager: Task 5 in stage 1.0 failed 1 times; aborting job
Traceback (most recent call last):
File "E:\Python_code\pythonProject\spark_test.py", line 14, in <module>
print(rdd.map(lambda x: x * 10).collect())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Python_code\pythonProject\.venv\Lib\site-packages\pyspark\rdd.py", line 1833, in collect
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Python_code\pythonProject\.venv\Lib\site-packages\py4j\java_gateway.py", line 1322, in __call__
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File "E:\Python_code\pythonProject\.venv\Lib\site-packages\py4j\protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 1.0 failed 1 times, most recent failure: Lost task 5.0 in stage 1.0 (TID 25) (GYG-LZC executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator

a n o n f u n $ 1. a p p l y O r E l s e (P y t h o n R u n n e r . s c a l a : 612) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . B a s e P y t h o n R u n n e r $ R e a d e r I t e r a t o r

a n o n $ 3. r e a d (P y t h o n R u n n e r . s c a l a : 789) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . P y t h o n R u n n e r

Driver stacktrace:
   at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2844)
   at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2780)
   at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2779)
   at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
   at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
   at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2779)
   at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1242)
   at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1242)
   at scala.Option.foreach(Option.scala:407)
   at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1242)
   at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3048)
   at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2982)
   at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2971)
   at org.apache.spark.util.EventLoop

a n o n $ 1. r u n (E v e n t L o o p . s c a l a : 49) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r . r u n J o b (D A G S c h e d u l e r . s c a l a : 984) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . r u n J o b (S p a r k C o n t e x t . s c a l a : 2398) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . r u n J o b (S p a r k C o n t e x t . s c a l a : 2419) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . r u n J o b (S p a r k C o n t e x t . s c a l a : 2438) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . r u n J o b (S p a r k C o n t e x t . s c a l a : 2463) a t o r g . a p a c h e . s p a r k . r d d . R D D . $ a n o n f u n $ c o l l e c t $ 1 (R D D . s c a l a : 1046) a t o r g . a p a c h e . s p a r k . r d d . R D D O p e r a t i o n S c o p e $ . w i t h S c o p e (R D D O p e r a t i o n S c o p e . s c a l a : 151) a t o r g . a p a c h e . s p a r k . r d d . R D D O p e r a t i o n S c o p e $ . w i t h S c o p e (R D D O p e r a t i o n S c o p e . s c a l a : 112) a t o r g . a p a c h e . s p a r k . r d d . R D D . w i t h S c o p e (R D D . s c a l a : 407) a t o r g . a p a c h e . s p a r k . r d d . R D D . c o l l e c t (R D D . s c a l a : 1045) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . P y t h o n R D D $ . c o l l e c t A n d S e r v e (P y t h o n R D D . s c a l a : 195) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . P y t h o n R D D . c o l l e c t A n d S e r v e (P y t h o n R D D . s c a l a) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e 0 (N a t i v e M e t h o d) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e (N a t i v e M e t h o d A c c e s s o r I m p l . j a v a : 62) a t s u n . r e f l e c t . D e l e g a t i n g M e t h o d A c c e s s o r I m p l . i n v o k e (D e l e g a t i n g M e t h o d A c c e s s o r I m p l . j a v a : 43) a t j a v a . l a n g . r e f l e c t . M e t h o d . i n v o k e (M e t h o d . j a v a : 498) a t p y 4 j . r e f l e c t i o n . M e t h o d I n v o k e r . i n v o k e (M e t h o d I n v o k e r . j a v a : 244) a t p y 4 j . r e f l e c t i o n . R e f l e c t i o n E n g i n e . i n v o k e (R e f l e c t i o n E n g i n e . j a v a : 374) a t p y 4 j . G a t e w a y . i n v o k e (G a t e w a y . j a v a : 282) a t p y 4 j . c o m m a n d s . A b s t r a c t C o m m a n d . i n v o k e M e t h o d (A b s t r a c t C o m m a n d . j a v a : 132) a t p y 4 j . c o m m a n d s . C a l l C o m m a n d . e x e c u t e (C a l l C o m m a n d . j a v a : 79) a t p y 4 j . C l i e n t S e r v e r C o n n e c t i o n . w a i t F o r C o m m a n d s (C l i e n t S e r v e r C o n n e c t i o n . j a v a : 182) a t p y 4 j . C l i e n t S e r v e r C o n n e c t i o n . r u n (C l i e n t S e r v e r C o n n e c t i o n . j a v a : 106) a t j a v a . l a n g . T h r e a d . r u n (T h r e a d . j a v a : 748) C a u s e d b y : o r g . a p a c h e . s p a r k . S p a r k E x c e p t i o n : P y t h o n w o r k e r e x i t e d u n e x p e c t e d l y (c r a s h e d) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . B a s e P y t h o n R u n n e r $ R e a d e r I t e r a t o r

$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:984) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463) at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1046) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:407) at org.apache.spark.rdd.RDD.collect(RDD.scala:1045) at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:195) at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.spark.SparkException: Python worker exited unexpectedly (crashed) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$ anonfun$1.applyOrElse(PythonRunner.scala:612)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator

a n o n f u n $ 1. a p p l y O r E l s e (P y t h o n R u n n e r . s c a l a : 594) a t s c a l a . r u n t i m e . A b s t r a c t P a r t i a l F u n c t i o n . a p p l y (A b s t r a c t P a r t i a l F u n c t i o n . s c a l a : 38) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . P y t h o n R u n n e r

a n o n $ 3. r e a d (P y t h o n R u n n e r . s c a l a : 766) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . B a s e P y t h o n R u n n e r $ R e a d e r I t e r a t o r . h a s N e x t (P y t h o n R u n n e r . s c a l a : 525) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . h a s N e x t (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 37) a t s c a l a . c o l l e c t i o n . I t e r a t o r . f o r e a c h (I t e r a t o r . s c a l a : 943) a t s c a l a . c o l l e c t i o n . I t e r a t o r . f o r e a c h $ (I t e r a t o r . s c a l a : 943) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . f o r e a c h (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 28) a t s c a l a . c o l l e c t i o n . g e n e r i c . G r o w a b l e . $ p l u s $ p l u s $ e q (G r o w a b l e . s c a l a : 62) a t s c a l a . c o l l e c t i o n . g e n e r i c . G r o w a b l e . $ p l u s $ p l u s $ e q $ (G r o w a b l e . s c a l a : 53) a t s c a l a . c o l l e c t i o n . m u t a b l e . A r r a y B u f f e r . $ p l u s $ p l u s $ e q (A r r a y B u f f e r . s c a l a : 105) a t s c a l a . c o l l e c t i o n . m u t a b l e . A r r a y B u f f e r . $ p l u s $ p l u s $ e q (A r r a y B u f f e r . s c a l a : 49) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o (T r a v e r s a b l e O n c e . s c a l a : 366) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o $ (T r a v e r s a b l e O n c e . s c a l a : 364) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . t o (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 28) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o B u f f e r (T r a v e r s a b l e O n c e . s c a l a : 358) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o B u f f e r $ (T r a v e r s a b l e O n c e . s c a l a : 358) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . t o B u f f e r (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 28) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o A r r a y (T r a v e r s a b l e O n c e . s c a l a : 345) a t s c a l a . c o l l e c t i o n . T r a v e r s a b l e O n c e . t o A r r a y $ (T r a v e r s a b l e O n c e . s c a l a : 339) a t o r g . a p a c h e . s p a r k . I n t e r r u p t i b l e I t e r a t o r . t o A r r a y (I n t e r r u p t i b l e I t e r a t o r . s c a l a : 28) a t o r g . a p a c h e . s p a r k . r d d . R D D . $ a n o n f u n $ c o l l e c t $ 2 (R D D . s c a l a : 1046) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . $ a n o n f u n $ r u n J o b $ 5 (S p a r k C o n t e x t . s c a l a : 2438) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . R e s u l t T a s k . r u n T a s k (R e s u l t T a s k . s c a l a : 93) a t o r g . a p a c h e . s p a r k . T a s k C o n t e x t . r u n T a s k W i t h L i s t e n e r s (T a s k C o n t e x t . s c a l a : 161) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . T a s k . r u n (T a s k . s c a l a : 141) a t o r g . a p a c h e . s p a r k . e x e c u t o r . E x e c u t o r $ T a s k R u n n e r . $ a n o n f u n $ r u n $ 4 (E x e c u t o r . s c a l a : 620) a t o r g . a p a c h e . s p a r k . u t i l . S p a r k E r r o r U t i l s . t r y W i t h S a f e F i n a l l y (S p a r k E r r o r U t i l s . s c a l a : 64) a t o r g . a p a c h e . s p a r k . u t i l . S p a r k E r r o r U t i l s . t r y W i t h S a f e F i n a l l y $ (S p a r k E r r o r U t i l s . s c a l a : 61) a t o r g . a p a c h e . s p a r k . u t i l . U t i l s $ . t r y W i t h S a f e F i n a l l y (U t i l s . s c a l a : 94) a t o r g . a p a c h e . s p a r k . e x e c u t o r . E x e c u t o r $ T a s k R u n n e r . r u n (E x e c u t o r . s c a l a : 623) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r . r u n W o r k e r (T h r e a d P o o l E x e c u t o r . j a v a : 1149) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r $ W o r k e r . r u n (T h r e a d P o o l E x e c u t o r . j a v a : 624) . . . 1 m o r e C a u s e d b y : j a v a . i o . E O F E x c e p t i o n a t j a v a . i o . D a t a I n p u t S t r e a m . r e a d I n t (D a t a I n p u t S t r e a m . j a v a : 392) a t o r g . a p a c h e . s p a r k . a p i . p y t h o n . P y t h o n R u n n e r

$anon$3.read(PythonRunner.scala:766) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1046) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.spark.api.python.PythonRunner$ anon$3.read(PythonRunner.scala:774)
... 32 more

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/Cpp五条/article/detail/293162