使用pycharm在本地运行pyspark

我用Pycharm IDE编写了以下非常简单的python脚本

from pyspark.sql import SparkSession
from pyspark.sql import Row
from pyspark.sql import SQLContext
from pyspark.sql.types import LongType, FloatType,IntegerType,StringType,DoubleType
from pyspark.sql.functions import udf
from pyspark.sql import functions as F
from pyspark.sql.window import Window
from  pyspark.sql.functions import abs
from pyspark.sql import HiveContext

spark = SparkSession.builder.config("requiredconfig").appName("SparkSran").enableHiveSupport().getOrCreate()
sc = spark.sparkContext

当我在IDE上单击run或运行以下命令时

spark-submit --py-files   /home/user/PycharmProjects/helloSparkWorld/test.py
I get

/usr/lib/spark/bin/spark-class: line 71: /usr/local/java/jdk10.0.1/bin/java: No such file or directory

我的JAVA_主页和SPARK_主页设置如下 echo

$SPARK\u HOME

给出

/usr/lib/SPARK

及 echo

$JAVA_HOME

提供

/usr/local/java/jdk10.0.1

---

您只需在您的环境中执行

pip安装pyspark

，您正在使用pycharm安装来运行python程序。如果您在本地运行pyspark

.py

文件，则可以将其作为python filename.py本身运行

基本上只需为您的

pip

或python解释器提供pyspark-pip包，您就可以使用相同的解释器通过pycharm运行它