Python调用pyspark报错整理

前言

Pycharm配置了SSH服务器和Anaconda的python解释器，如果没有配置可参考 大数据单机学习环境搭建(8)Linux单节点Anaconda安装和Pycharm连接

Pycharm执行的脚本

执行如下 pyspark_model.py 的python脚本，构建SparkSession来执行sparksql

"""
    脚本名称：Pycharm使用pyspark测试
    功能：Pycharm远程执行sparksql
"""
from pyspark.sql import SparkSession
import os

os.environ['SPARK_HOME'] = '/opt/spark'
os.environ['JAVA_HOME'] = '/opt/jdk1.8'

spark = SparkSession.builder \
    .appName('pyspark_conda') \
    .master("yarn") \
    .config("spark.sql.warehouse.dir", "hdfs://bigdata01:8020/user/hive/warehouse") \
    .config("hive.metastore.uris", "thrift://bigdata01:9083") \
    .enableHiveSupport() \
    .getOrCreate()

spark.sql('select * from hostnames limit 10;').show()

spark.stop()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

报错一：pyspark版本不匹配

例如我当前集群环境Spark3.0.0，python的pyspark3.5.0，没有指定版本默认下载了最新的

报错信息 [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number., 具体如下：

ssh://slash@bigdata01:22/opt/python3/bin/python3 -u /home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py
JAVA_HOME is not set
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 7, in <module>
    spark = SparkSession.builder \
  File "/opt/python3/lib/python3.8/site-packages/pyspark/sql/session.py", line 497, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 515, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 201, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 436, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/java_gateway.py", line 107, in launch_gateway
    raise PySparkRuntimeError(
pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

如果坚持不更换python的pyspark版本，即使像报错2已经指定了JAVA_HOME 依然会有其他报错。例如下方报错 Py4JError ，所以最彻底的方法是替换pyspark版本与spark版本一致

Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 7, in <module>
    spark = SparkSession.builder \
  File "/opt/python3/lib/python3.8/site-packages/pyspark/sql/session.py", line 497, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 515, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 203, in __init__
    self._do_init(
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 316, in _do_init
    self._jvm.PythonUtils.getPythonAuthSocketTimeout(self._jsc)
  File "/opt/python3/lib/python3.8/site-packages/py4j/java_gateway.py", line 1549, in __getattr__
    raise Py4JError(
py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM
1
2
3
4
5
6
7
8
9
10
11
12
13
14

报错二：JAVA_HOME指定不成功

python的pyspark已经重装3.0.0版本（下载时指定版本 pip install pyspark==3.0.0），报错信息 Java gateway process exited before sending its port number., JAVA_HOME is not set 具体如下：

ssh://slash@bigdata01:22/opt/python3/bin/python3 -u /home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py
JAVA_HOME is not set
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 7, in <module>
    spark = SparkSession.builder \
  File "/opt/python3/lib/python3.8/site-packages/pyspark/sql/session.py", line 186, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 371, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 128, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/context.py", line 320, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/python3/lib/python3.8/site-packages/pyspark/java_gateway.py", line 105, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

指定内容如下：

# pyspark3.5.0指定了 SPARK_HOME JAVA_HOME还是会报错
# pyspark3.0.0指定后成功运行
os.environ['SPARK_HOME'] = '/opt/spark'
os.environ['JAVA_HOME'] = '/opt/jdk1.8'
1
2
3
4

报错三：python版本问题

最开始安装的最新版的anaconda环境，其中python3.11，安装pyspark3.0.0也会报错 TypeError: code() argument 13 must be str, not int，具体内容如下：

ssh://slash@bigdata01:22/opt/anaconda3/bin/python3.11 -u /home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py
Traceback (most recent call last):
  File "/home/slash/etl/dwtool/pyspark/pyspark_script/pyspark_model.py", line 1, in <module>
    from pyspark.sql import SparkSession
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/context.py", line 30, in <module>
    from pyspark import accumulators
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/serializers.py", line 71, in <module>
    from pyspark import cloudpickle
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/cloudpickle.py", line 209, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/site-packages/pyspark/cloudpickle.py", line 172, in _make_cell_set_template_code
    return types.CodeType(
           ^^^^^^^^^^^^^^^
TypeError: code() argument 13 must be str, not int
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

删除 /opt/anaconda3的文件夹后，重新安装了 Anaconda3-2021.05-Linux-x86_64.sh 版本的anaconda，其中python3.8，利用pyspark3.0.0第三方库操作spark3.0.0的计算引擎构建SparkSession，执行sparksql成功。

---

声明：本文所载信息不保证准确性和完整性。文中所述内容和意见仅供参考，不构成实际商业建议，如有雷同纯属巧合。