获得;无效语法";使用pyspark从文本文件读取数据时出错

获得;无效语法";使用pyspark从文本文件读取数据时出错,pyspark,syntax-error,datareader,Pyspark,Syntax Error,Datareader,我正在尝试使用pyspark读取文本文件。文件中的数据以逗号分隔 我已经尝试过使用sqlcontext读取数据 from pyspark import SparkContext from pyspark.sql import SQLContext from pyspark.sql.types import * from pyspark.sql.functions import * sc = SparkContext._active_spark_context filePath = './da

我正在尝试使用pyspark读取文本文件。文件中的数据以逗号分隔

我已经尝试过使用sqlcontext读取数据

from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql.functions import *

sc = SparkContext._active_spark_context

filePath = './data_files/data.txt'

sqlContext = SQLContext(sc)

print(fileData)
schema = StructType([StructField('ID', IntegerType(), False),
                     StructField('Name', StringType(), False),
                     StructField('Project', StringType(), False),
                     StructField('Location', StringType(), False)])
print(schema)

fileRdd = sc.textFile(fileData).map(_.split(",")).map{x => org.apache.spark.sql.Row(x:_*)}
sqlDf = sqlContext.createDataFrame(fileRdd,schema)
sqlDf.show()
我有以下错误

文件“”,第1行 fileRdd=sc.textFile(fileData).map(.split(“,”).map{x=>org.apache.spark.sql.Row(x:*)} ^SyntaxError:无效语法


我试过使用下面的代码,效果很好

from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql.functions import *


sc = SparkContext._active_spark_context
sc = SparkContext("local", "first app")
sqlContext = SQLContext(sc)

filePath = "./data_files/data.txt"

# Load a text file and convert each line to a Row.
lines = sc.textFile(filePath)
parts = lines.map(lambda l: l.split(","))
# Each line is converted to a tuple.
people = parts.map(lambda p: (p[0].strip(), p[1], p[2], p[3]))

# The schema is encoded in a string.
schemaString = "ID Name Project Location"

fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split()]
schema = StructType(fields)

schemaPeople = sqlContext.createDataFrame(people, schema)
schemaPeople.show()

是Python吗?您不能在Python中使用Scala语法。您能建议一种替代解决方案吗?请检查示例: