Warning: file_get_contents(/data/phpspider/zhask/data//catemap/8/python-3.x/17.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
&引用;。。。构造函数org.apache.spark.api.python.pythonacumeratorv2(…)不存在;创建新Sparksession时出错,我做错了什么?_Python_Python 3.x_Apache Spark_Pyspark - Fatal编程技术网

&引用;。。。构造函数org.apache.spark.api.python.pythonacumeratorv2(…)不存在;创建新Sparksession时出错,我做错了什么?

&引用;。。。构造函数org.apache.spark.api.python.pythonacumeratorv2(…)不存在;创建新Sparksession时出错,我做错了什么?,python,python-3.x,apache-spark,pyspark,Python,Python 3.x,Apache Spark,Pyspark,所以我对Spark的每一件事都非常熟悉,我正在努力找出我为什么会犯这个错误。这就是我到目前为止所做的: 我安装了最新的Java 安装Python;我第一次安装了3.8,但是当我去控制台写“pip install jupyter lab”时,我得到了一个错误;所以我不得不降级到最近的3.7.x。。。没有错误,那么 通过cmd控制台,我编写了“pip install pyspark==2.3.2”(注意:我已经尝试过更简单的“pip install pyspark”,但得到了下面相同的错误) 我去

所以我对Spark的每一件事都非常熟悉,我正在努力找出我为什么会犯这个错误。这就是我到目前为止所做的:

  • 我安装了最新的Java
  • 安装Python;我第一次安装了3.8,但是当我去控制台写“pip install jupyter lab”时,我得到了一个错误;所以我不得不降级到最近的3.7.x。。。没有错误,那么
  • 通过cmd控制台,我编写了“pip install pyspark==2.3.2”(注意:我已经尝试过更简单的“pip install pyspark”,但得到了下面相同的错误)
  • 我去jupyter实验室写了我的前3行:
从pyspark.sql.functions导入lit、col、count、sum、when

从pyspark.sql导入SparkSession

从pyspark导入SparkContext

没有错误

  • 但后来我写道:
spark=SparkSession.builder.appName(“app1”).getOrCreate()

我犯了一个巨大的错误,我想不出为什么会这样。我只是想说清楚;我的第一次安装是使用Anaconda Navigator,我得到了相同的错误。。。我们将非常感谢您对此提供的任何帮助:

Py4JError                                 Traceback (most recent call last)
<ipython-input-7-b5258aee5a77> in <module>
----> 1 spark = SparkSession.builder.appName("app1").getOrCreate()

c:\users\hp\appdata\local\programs\python\python37\lib\site-packages\pyspark\sql\session.py in getOrCreate(self)
    171                     for key, value in self._options.items():
    172                         sparkConf.set(key, value)
--> 173                     sc = SparkContext.getOrCreate(sparkConf)
    174                     # This SparkContext may be an existing one.
    175                     for key, value in self._options.items():

c:\users\hp\appdata\local\programs\python\python37\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
    349         with SparkContext._lock:
    350             if SparkContext._active_spark_context is None:
--> 351                 SparkContext(conf=conf or SparkConf())
    352             return SparkContext._active_spark_context
    353 

c:\users\hp\appdata\local\programs\python\python37\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    116         try:
    117             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
--> 118                           conf, jsc, profiler_cls)
    119         except:
    120             # If an error occurs, clean up in order to allow future SparkContext creation:

c:\users\hp\appdata\local\programs\python\python37\lib\site-packages\pyspark\context.py in _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, jsc, profiler_cls)
    187         self._accumulatorServer = accumulators._start_update_server(auth_token)
    188         (host, port) = self._accumulatorServer.server_address
--> 189         self._javaAccumulator = self._jvm.PythonAccumulatorV2(host, port, auth_token)
    190         self._jsc.sc().register(self._javaAccumulator)
    191 

c:\users\hp\appdata\local\programs\python\python37\lib\site-packages\py4j\java_gateway.py in __call__(self, *args)
   1523         answer = self._gateway_client.send_command(command)
   1524         return_value = get_return_value(
-> 1525             answer, self._gateway_client, None, self._fqn)
   1526 
   1527         for temp_arg in temp_args:

c:\users\hp\appdata\local\programs\python\python37\lib\site-packages\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
    330                 raise Py4JError(
    331                     "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
--> 332                     format(target_id, ".", name, value))
    333         else:
    334             raise Py4JError(

Py4JError: An error occurred while calling None.org.apache.spark.api.python.PythonAccumulatorV2. Trace:
py4j.Py4JException: Constructor org.apache.spark.api.python.PythonAccumulatorV2([class java.lang.String, class java.lang.Integer, class java.lang.String]) does not exist
    at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:179)
    at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:196)
    at py4j.Gateway.invoke(Gateway.java:237)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)```

Py4JError回溯(最近一次调用)
在里面
---->1 spark=SparkSession.builder.appName(“app1”).getOrCreate()
getOrCreate(self)中的c:\users\hp\appdata\local\programs\python\python37\lib\site packages\pyspark\sql\session.py
171对于键,值在self.\u options.items()中:
172 sparkConf.集合(键、值)
-->173 sc=SparkContext.getOrCreate(sparkConf)
174#此SparkContext可能是现有的。
175对于键,值在self.\u options.items()中:
c:\users\hp\appdata\local\programs\python37\lib\site packages\pyspark\context.py在getOrCreate(cls,conf)中
带SparkContext的349。\u锁:
350如果SparkContext.\u active\u spark\u上下文为无:
-->351 SparkContext(conf=conf或SparkConf())
352返回SparkContext.\u活动\u spark\u上下文
353
c:\users\hp\appdata\local\programs\python37\lib\site packages\pyspark\context.py in\uuuuu init\uuuu(self、master、appName、sparkHome、pyFiles、environment、batchSize、serializer、conf、gateway、jsc、profiler\u cls)
116尝试:
117 self.\u do\u init(主程序、appName、sparkHome、pyFiles、环境、batchSize、序列化程序、,
-->118形态、jsc、探查器(cls)
119除:
120#如果发生错误,请清理以允许将来创建SparkContext:
c:\users\hp\appdata\local\programs\python37\lib\site packages\pyspark\context.py in\u do\u init(self、master、appName、sparkHome、pyFiles、environment、batchSize、serializer、conf、jsc、profiler\u cls)
187 self.\u累加器服务器=累加器.\u启动\u更新\u服务器(身份验证令牌)
188(主机、端口)=自身。\u累加器server.server\u地址
-->189 self.\u javaAccumerator=self.\u jvm.pythonacumeratorv2(主机、端口、身份验证令牌)
190 self._jsc.sc()寄存器(self._java累加器)
191
c:\users\hp\appdata\local\programs\python37\lib\site packages\py4j\java\u gateway.py in\uuuuu调用(self,*args)
1523 answer=self.\u gateway\u client.send\u命令(command)
1524返回值=获取返回值(
->1525回答,自。\网关\客户端,无,自。\ fqn)
1526
1527对于临时参数中的临时参数:
c:\users\hp\appdata\local\programs\python37\lib\site packages\py4j\protocol.py在get\u return\u值中(应答、网关\u客户端、目标\u id、名称)
330升起Py4JError(
331“调用{0}{1}{2}时出错。跟踪:\n{3}\n”。
-->332格式(目标_id,“.”,名称,值))
333其他:
334升起Py4JError(
Py4JError:调用None.org.apache.spark.api.python.pythonacumeratorv2时出错。跟踪:
py4j.Py4JException:Constructor org.apache.spark.api.python.PythonAccumulatorV2([class java.lang.String,class java.lang.Integer,class java.lang.String])不存在
位于py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:179)
位于py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:196)
在py4j.Gateway.invoke处(Gateway.java:237)
位于py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
在py4j.commands.ConstructorCommand.execute处(ConstructorCommand.java:69)
在py4j.GatewayConnection.run处(GatewayConnection.java:238)
运行(Thread.java:748)```