Python 转换拼花错误中的CSV_Python_Csv_Apache Spark_Pyspark

Python 转换拼花错误中的CSV

python csv apache-spark pyspark

Python 转换拼花错误中的CSV,python,csv,apache-spark,pyspark,Python,Csv,Apache Spark,Pyspark,我的python脚本有问题。我的脚本应该将我的csv转换为拼花文件。但是，当我执行脚本时，出现以下错误：什么是o59.csv？这不是我当前的文件这是我的剧本 from pyspark import SparkContext from pyspark.sql import SQLContext from pyspark.sql import SparkSession from pyspark.sql.types import * spark = SparkSession.builder \

我的python脚本有问题。我的脚本应该将我的csv转换为拼花文件。但是，当我执行脚本时，出现以下错误：

什么是o59.csv？这不是我当前的文件

这是我的剧本

from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import SparkSession
from pyspark.sql.types import *

spark = SparkSession.builder \
    .appName ("Convert into Parquet") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()


schema = StructType([
    StructField("date", DateType(),True),
    StructField("semaine",IntegerType(),True),
    StructField("annee",IntegerType(),True),
    StructField("mois",IntegerType(),True)])

# read csv
 df = spark.read.csv('//home/sshuser3/calendrier.csv', header=True)

# Displays the content of the DataFrame to stdout
 df  = sqlContext.createDataFrame(rdd,schema)


df.write.parquet('//home/sshuser3/outputParquet/calendrier.parquet')

你对我有什么建议吗

整个信息是：

spark-submit Convert.py
SPARK_MAJOR_VERSION is set to 2, using Spark2
17/08/11 15:01:58 INFO SparkContext: Running Spark version 2.1.0.2.6.0.10-29
17/08/11 15:01:59 INFO SecurityManager: Changing view acls to: sshuser3
17/08/11 15:01:59 INFO SecurityManager: Changing modify acls to: sshuser3
17/08/11 15:01:59 INFO SecurityManager: Changing view acls groups to: 
17/08/11 15:01:59 INFO SecurityManager: Changing modify acls groups to: 
17/08/11 15:01:59 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sshuser3); groups with view permissions: Set(); users  with modify permissions: Set(sshuser3); groups with modify permissions: Set()
17/08/11 15:01:59 INFO Utils: Successfully started service 'sparkDriver' on port 44872.
17/08/11 15:01:59 INFO SparkEnv: Registering MapOutputTracker
17/08/11 15:01:59 INFO SparkEnv: Registering BlockManagerMaster
17/08/11 15:01:59 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
17/08/11 15:01:59 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
17/08/11 15:01:59 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-cf215ce8-b19e-4270-9c62-57a37dea703b
17/08/11 15:01:59 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
17/08/11 15:01:59 INFO SparkEnv: Registering OutputCommitCoordinator
17/08/11 15:01:59 INFO log: Logging initialized @2107ms
17/08/11 15:01:59 INFO Server: jetty-9.2.z-SNAPSHOT
17/08/11 15:01:59 INFO Server: Started @2176ms
17/08/11 15:01:59 INFO ServerConnector: Started ServerConnector@617e9f14{HTTP/1.1}{0.0.0.0:4040}
17/08/11 15:01:59 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@31c59a{/jobs,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@498e3b10{/jobs/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7836dfef{/jobs/job,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2d7e5980{/jobs/job/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6c63d029{/stages,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@732e5b44{/stages/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2bfe44d0{/stages/stage,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6a3c33dc{/stages/stage/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4707f607{/stages/pool,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@16843d33{/stages/pool/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6e341ff2{/storage,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2ec88f8e{/storage/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2d5cca61{/storage/rdd,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4c3ab7d9{/storage/rdd/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@289b1fcb{/environment,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@30201b75{/environment/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5d07431a{/executors,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2b18c427{/executors/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@72eda80f{/executors/threadDump,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1891056d{/executors/threadDump/json,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3bbbcd13{/static,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@71f8537e{/,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@72382d83{/api,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2f2da5a4{/jobs/job/kill,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@27e6b3da{/stages/stage/kill,null,AVAILABLE,@Spark}
17/08/11 15:01:59 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.0.0.18:4040
17/08/11 15:02:00 INFO RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
17/08/11 15:02:00 INFO RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
17/08/11 15:02:00 INFO Client: Requesting a new application from cluster with 2 NodeManagers
17/08/11 15:02:00 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (25600 MB per container)
17/08/11 15:02:00 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/08/11 15:02:00 INFO Client: Setting up container launch context for our AM
17/08/11 15:02:00 INFO Client: Setting up the launch environment for our AM container
17/08/11 15:02:00 INFO Client: Preparing resources for our AM container
17/08/11 15:02:02 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/pyspark.zip -> adl://home/user/sshuser3/.sparkStaging/application_1501619838490_0047/pyspark.zip
17/08/11 15:02:03 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip -> adl://home/user/sshuser3/.sparkStaging/application_1501619838490_0047/py4j-0.10.4-src.zip
17/08/11 15:02:03 INFO Client: Uploading resource file:/tmp/spark-813e8548-f7f4-4ff7-80f4-236e0e3b5ee2/__spark_conf__630655753844577263.zip -> adl://home/user/sshuser3/.sparkStaging/application_1501619838490_0047/__spark_conf__.zip
17/08/11 15:02:03 INFO SecurityManager: Changing view acls to: sshuser3
17/08/11 15:02:03 INFO SecurityManager: Changing modify acls to: sshuser3
17/08/11 15:02:03 INFO SecurityManager: Changing view acls groups to: 
17/08/11 15:02:03 INFO SecurityManager: Changing modify acls groups to: 
17/08/11 15:02:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sshuser3); groups with view permissions: Set(); users  with modify permissions: Set(sshuser3); groups with modify permissions: Set()
17/08/11 15:02:03 INFO Client: Submitting application application_1501619838490_0047 to ResourceManager
17/08/11 15:02:03 INFO YarnClientImpl: Submitted application application_1501619838490_0047
17/08/11 15:02:03 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1501619838490_0047 and attemptId None
17/08/11 15:02:04 INFO Client: Application report for application_1501619838490_0047 (state: ACCEPTED)
17/08/11 15:02:04 INFO Client: 
     client token: N/A
     diagnostics: AM container is launched, waiting for AM container to Register with RM
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1502463723834
     final status: UNDEFINED
     tracking URL: http://hn1-pzhdla.stikrkashgqefoqg3xnv2chu0d.fx.internal.cloudapp.net:8088/proxy/application_1501619838490_0047/
     user: sshuser3
17/08/11 15:02:05 INFO Client: Application report for application_1501619838490_0047 (state: ACCEPTED)
17/08/11 15:02:06 INFO Client: Application report for application_1501619838490_0047 (state: ACCEPTED)
17/08/11 15:02:07 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
17/08/11 15:02:07 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> hn0-pzhdla.stikrkashgqefoqg3xnv2chu0d.fx.internal.cloudapp.net,hn1-pzhdla.stikrkashgqefoqg3xnv2chu0d.fx.internal.cloudapp.net, PROXY_URI_BASES -> http://hn0-pzhdla.stikrkashgqefoqg3xnv2chu0d.fx.internal.cloudapp.net:8088/proxy/application_1501619838490_0047,http://hn1-pzhdla.stikrkashgqefoqg3xnv2chu0d.fx.internal.cloudapp.net:8088/proxy/application_1501619838490_0047), /proxy/application_1501619838490_0047
17/08/11 15:02:07 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/08/11 15:02:07 INFO Client: Application report for application_1501619838490_0047 (state: RUNNING)
17/08/11 15:02:07 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 10.0.0.7
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1502463723834
     final status: UNDEFINED
     tracking URL: http://hn1-pzhdla.stikrkashgqefoqg3xnv2chu0d.fx.internal.cloudapp.net:8088/proxy/application_1501619838490_0047/
     user: sshuser3
17/08/11 15:02:07 INFO YarnClientSchedulerBackend: Application application_1501619838490_0047 has started running.
17/08/11 15:02:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37070.
17/08/11 15:02:07 INFO NettyBlockTransferService: Server created on 10.0.0.18:37070
17/08/11 15:02:07 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
17/08/11 15:02:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.0.18, 37070, None)
17/08/11 15:02:07 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.18:37070 with 366.3 MB RAM, BlockManagerId(driver, 10.0.0.18, 37070, None)
17/08/11 15:02:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.0.18, 37070, None)
17/08/11 15:02:07 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.0.0.18, 37070, None)
17/08/11 15:02:08 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1651e3d3{/metrics/json,null,AVAILABLE,@Spark}
17/08/11 15:02:08 INFO EventLoggingListener: Logging events to adl:///hdp/spark2-events/application_1501619838490_0047
17/08/11 15:02:10 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.0.7:51484) with ID 1
17/08/11 15:02:10 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.7:40909 with 3.0 GB RAM, BlockManagerId(1, 10.0.0.7, 40909, None)
17/08/11 15:02:12 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.0.4:42318) with ID 2
17/08/11 15:02:12 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.0.4:46697 with 3.0 GB RAM, BlockManagerId(2, 10.0.0.4, 46697, None)
17/08/11 15:02:12 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/08/11 15:02:12 INFO SharedState: Warehouse path is 'file:/home/sshuser3/spark-warehouse/'.
17/08/11 15:02:12 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@66612a38{/SQL,null,AVAILABLE,@Spark}
17/08/11 15:02:12 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5b8829c6{/SQL/json,null,AVAILABLE,@Spark}
17/08/11 15:02:12 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@367f0e38{/SQL/execution,null,AVAILABLE,@Spark}
17/08/11 15:02:12 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4f95bd8d{/SQL/execution/json,null,AVAILABLE,@Spark}
17/08/11 15:02:12 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@468f4cc5{/static/sql,null,AVAILABLE,@Spark}
17/08/11 15:02:12 WARN DataSource: Error while looking for metadata directory.
Traceback (most recent call last):
  File "/home/sshuser3/Convert.py", line 22, in <module>
    df = spark.read.csv('//home/sshuser3/calendrier.csv',header = True)
  File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 380, in csv
  File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o59.csv.
: java.io.IOException: No FileSystem for scheme: null
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2793)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:372)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.immutable.List.flatMap(List.scala:344)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:370)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
    at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:415)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:280)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:748)

我建议再次检查您的代码片段

# Displays the content of the DataFrame to stdout
 df  = sqlContext.createDataFrame(rdd,schema)

看起来您应该首先将数据帧转换为RDD，然后使用

rdd = df.rdd

我做了一个小实验

//read csv file
df = spark.read.csv('/<path_to_csv>', header=True)

//casting types for specific columns because loaded data is string at it has unicode prefix
df = df.select(df.<column_name>.cast('timestamp'),df.<column_name>.cast('int'),df.<column_name>.cast('int'),df.<column_name>.cast('int'))

//creating dataframe using schema
dt = spark.createDataFrame(df.rdd,schema)

//write  as parquet
dt.write.parquet('/path_to_parquet_file')

//读取csv文件
df=spark.read.csv（'/'，header=True）
//强制转换特定列的类型，因为加载的数据是字符串，并且具有unicode前缀
df=df.select（df..cast（'timestamp'）、df..cast（'int'）、df..cast（'int'）、df..cast（'int'））
//使用模式创建数据帧
dt=spark.createDataFrame（df.rdd，模式）
//写拼花地板
dt.write.parquet（'/path\u to\u parquet\u file'）

经过一些研究，我发现了我的问题。我的路径未正确写入。我必须使用file:/my path，而不是//my path

所以这个帖子可以关闭。感谢各位的回答。

Spark集群中的某些节点是否可能存在文件系统问题（在其本地文件系统或HDFS层上）？也许o59.csv是Caldenrier.csv的一个分区/块，它被分发到了某个节点，而该节点上运行的进程遇到了I/O错误？我必须承认，我在这里有点自作主张。但我会从更新Java开始，并确保它是运行脚本的正确Python版本（2.7 vs 3.x）。如果Python版本与您当前的Java版本兼容。只是一些想法…试试

df.write.parquet（“file:///home/sshuser....“”

如果是您的本地FS路径或

hdfs:///home/...

如果打开HDFS@philantrovert我试过了，但问题还是一样。@Zeth我目前的Java版本是：openjdk版本“1.8.0131”，python版本是2.7.12。这两个是兼容的？这是有效的吗？”因为就像我说的，问题似乎来自我的函数df=spark.read.csv。在那之后我什么也做不了。我试着打印一些东西以查看它出现在日志中，但它不工作…是的。这是工作繁琐的部分铸造列所需的类型。好吧，我试图这样做，但它不适合我。。。我不知道我的剧本哪里出了问题。。。。

rdd = df.rdd

//read csv file
df = spark.read.csv('/<path_to_csv>', header=True)

//casting types for specific columns because loaded data is string at it has unicode prefix
df = df.select(df.<column_name>.cast('timestamp'),df.<column_name>.cast('int'),df.<column_name>.cast('int'),df.<column_name>.cast('int'))

//creating dataframe using schema
dt = spark.createDataFrame(df.rdd,schema)

//write  as parquet
dt.write.parquet('/path_to_parquet_file')