Apache spark 如何在Spark 2.0程序(实际上是pyspark 2.0)中编写正确的入口点?

Apache spark 如何在Spark 2.0程序(实际上是pyspark 2.0)中编写正确的入口点?,apache-spark,pyspark,Apache Spark,Pyspark,今天,我想尝试Spark2.0的一些新功能,以下是我的程序: #coding:utf-8 from pyspark.conf import SparkConf from pyspark.sql import SparkSession spark = SparkSession.builder.master("local").appName('test 2.0').config(conf=SparkConf()).getOrCreate() df = spark.read.json("/Users

今天,我想尝试Spark2.0的一些新功能,以下是我的程序:

#coding:utf-8
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession

spark = SparkSession.builder.master("local").appName('test 2.0').config(conf=SparkConf()).getOrCreate()
df = spark.read.json("/Users/lyj/Programs/Apache/Spark2/examples/src/main/resources/people.json")
df.show()
但它的错误如下:

Traceback (most recent call last):
  File "/Users/lyj/Programs/kiseliugit/MyPysparkCodes/test/spark2.0.py", line 5, in <module>
spark = SparkSession.builder.master("local").appName('test 2.0').config(conf=SparkConf()).getOrCreate()
  File "/Users/lyj/Programs/Apache/Spark2/python/pyspark/conf.py", line 104, in __init__
SparkContext._ensure_initialized()
  File "/Users/lyj/Programs/Apache/Spark2/python/pyspark/context.py", line 243, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway()
  File "/Users/lyj/Programs/Apache/Spark2/python/pyspark/java_gateway.py", line 116, in launch_gateway
java_import(gateway.jvm, "org.apache.spark.SparkConf")
  File "/Library/Python/2.7/site-packages/py4j/java_gateway.py", line 90, in java_import
return_value = get_return_value(answer, gateway_client, None, None)
  File "/Library/Python/2.7/site-packages/py4j/protocol.py", line 306, in get_return_value
value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
KeyError: u'y'
回溯(最近一次呼叫最后一次):
文件“/Users/lyj/Programs/kiseliugit/mypysparkcode/test/spark2.0.py”,第5行,在
spark=SparkSession.builder.master(“本地”).appName(“测试2.0”).config(conf=SparkConf()).getOrCreate()
文件“/Users/lyj/Programs/Apache/Spark2/python/pyspark/conf.py”,第104行,在__
SparkContext.\u确保\u已初始化()
文件“/Users/lyj/Programs/Apache/Spark2/python/pyspark/context.py”,第243行,已初始化
SparkContext.\u gateway=网关或启动\u gateway()
文件“/Users/lyj/Programs/Apache/Spark2/python/pyspark/java_gateway.py”,第116行,在launch_gateway中
java_导入(gateway.jvm,“org.apache.spark.SparkConf”)
java_导入中的文件“/Library/Python/2.7/site packages/py4j/java_gateway.py”,第90行
返回值=获取返回值(应答,网关客户端,无,无)
文件“/Library/Python/2.7/site packages/py4j/protocol.py”,第306行,在get_return_值中
值=输出\转换器[类型](回答[2:],网关\客户端)
关键错误:你

这几行代码怎么了?它有java环境的问题吗?另外,我使用IDE PyCharm进行开发。

尝试升级py4j,pip安装py4j——升级

这对我有用