Apache spark apachespark&x2B；不考虑配置的拼花地板使用“；分区的”；提交人_Apache Spark_Hadoop_Amazon S3_Parquet

Apache spark apachespark&x2B；不考虑配置的拼花地板使用“；分区的”；提交人

apache-spark hadoop amazon-s3

Apache spark apachespark&x2B；不考虑配置的拼花地板使用“；分区的”；提交人,apache-spark,hadoop,amazon-s3,parquet,Apache Spark,Hadoop,Amazon S3,Parquet,我在本地机器上使用ApacheSpark（3.0）将分区数据（拼花文件）写入AWS S3，而没有在机器上安装Hadoop。当我有很多文件要写到大约50个分区（partitionBy=date）时，我在写入S3时得到了FileNotFoundException 然后我遇到了新的问题，所以我尝试配置“分区”提交程序。但我仍然可以看到，当文件格式为“parquet”时，Spark使用ParquetOutputCommitter而不是PartitionedStagingCommitter。当我有很多数据

我在本地机器上使用ApacheSpark（3.0）将分区数据（拼花文件）写入AWS S3，而没有在机器上安装Hadoop。当我有很多文件要写到大约50个分区（partitionBy=date）时，我在写入S3时得到了FileNotFoundException

然后我遇到了新的问题，所以我尝试配置“分区”提交程序。但我仍然可以看到，当文件格式为“parquet”时，Spark使用ParquetOutputCommitter而不是PartitionedStagingCommitter。当我有很多数据要写时，我仍然会得到FileNotFoundException

我的配置：

        sparkSession.conf().set("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", 2);
        sparkSession.conf().set("spark.hadoop.fs.s3a.committer.name", "partitioned");
        sparkSession.conf().set("spark.hadoop.fs.s3a.committer.magic.enabled ", false);
        sparkSession.conf().set("spark.hadoop.fs.s3a.committer.staging.conflict-mode", "append");
        sparkSession.conf().set("spark.hadoop.fs.s3a.committer.staging.unique-filenames", true);
        sparkSession.conf().set("spark.hadoop.fs.s3a.committer.staging.abort.pending.uploads", true);
        sparkSession.conf().set("spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a", "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory");
        sparkSession.conf().set("spark.sql.sources.commitProtocolClass", "org.apache.spark.internal.io.cloud.PathOutputCommitProtocol");
        sparkSession.conf().set("spark.sql.parquet.output.committer.class", "org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter");
        sparkSession.conf().set("spark.hadoop.fs.s3a.committer.staging.tmp.path", "tmp/staging");

我在做什么？有人能帮忙吗

注意：我已经在Spark中创建了一个JIRA，但至今没有帮助：

==============================================================

我尝试了（@Rajadayalan）的答案。但它仍然使用FileOutputFormatter。我尝试将spark版本降级到2.4.5，但没有任何运气

20/04/06 12:44:52 INFO ParquetFileFormat: Using user defined output committer for Parquet: org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
20/04/06 12:44:52 WARN AbstractS3ACommitterFactory: **Using standard FileOutputCommitter to commit work**. This is slow and potentially unsafe.
20/04/06 12:44:52 INFO FileOutputCommitter: File Output Committer Algorithm version is 2
20/04/06 12:44:52 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
20/04/06 12:44:52 INFO AbstractS3ACommitterFactory: Using Commmitter FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_20200406124452_0000}; taskId=attempt_20200406124452_0000_m_000000_0, status=''}; org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter@61deb03f}; outputPath=s3a://******/observation, workPath=s3a://******/observation/_temporary/0/_temporary/attempt_20200406124452_0000_m_000000_0, algorithmVersion=2, skipCleanup=false, ignoreCleanupFailures=false} for s3a://********/observation
20/04/06 12:44:53 INFO HashAggregateExec: spark.sql.codegen.aggregate.map.twolevel.enabled is set to true, but current version of codegened fast hashmap does not support this aggregate.
20/04/06 12:44:54 INFO CodeGenerator: Code generated in 81.077046 ms
20/04/06 12:44:54 INFO HashAggregateExec: spark.sql.codegen.aggregate.map.twolevel.enabled is set to true, but current version of codegened fast hashmap does not support this aggregate.
20/04/06 12:44:54 INFO CodeGenerator: Code generated in 31.993775 ms
20/04/06 12:44:54 INFO CodeGenerator: Code generated in 9.967359 ms

注意：我的本地计算机中没有安装Spark。因此将spark-hadoop-cloud_2.11作为编译时依赖项我的build.gradle如下所示：

    compile group: 'org.apache.spark', name: 'spark-hadoop-cloud_2.11', version: '2.4.2.3.1.3.0-79'
    compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.4.5'
    // https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind
    compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.10.0'
    // https://mvnrepository.com/artifact/org.apache.parquet/parquet-column
    compile group: 'org.apache.parquet', name: 'parquet-column', version: '1.10.1'
    // https://mvnrepository.com/artifact/org.apache.parquet/parquet-hadoop
    compile group: 'org.apache.parquet', name: 'parquet-hadoop', version: '1.10.1'
    compile group: 'org.apache.parquet', name: 'parquet-avro', version: '1.10.1'
    // https://mvnrepository.com/artifact/org.apache.spark/spark-sketch
    compile group: 'org.apache.spark', name: 'spark-sketch_2.11', version: '2.4.5'
    // https://mvnrepository.com/artifact/org.apache.spark/spark-core
    compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.4.5'
    // https://mvnrepository.com/artifact/org.apache.spark/spark-catalyst
    compile group: 'org.apache.spark', name: 'spark-catalyst_2.11', version: '2.4.5'
    // https://mvnrepository.com/artifact/org.apache.spark/spark-tags
    compile group: 'org.apache.spark', name: 'spark-tags_2.11', version: '2.4.5'
    compile group: 'org.apache.spark', name: 'spark-avro_2.11', version: '2.4.5'
    // https://mvnrepository.com/artifact/org.apache.spark/spark-hive
    compile group: 'org.apache.spark', name: 'spark-hive_2.11', version: '2.4.5'
    // https://mvnrepository.com/artifact/org.apache.xbean/xbean-asm6-shaded
    compile group: 'org.apache.xbean', name: 'xbean-asm7-shaded', version: '4.15'
   compile group: 'org.apache.hadoop', name: 'hadoop-common', version: '3.2.1'
//    compile group: 'org.apache.hadoop', name: 'hadoop-s3guard', version: '3.2.1'
    compile group: 'org.apache.hadoop', name: 'hadoop-aws', version: '3.2.1'
    compile group: 'org.apache.hadoop', name: 'hadoop-client', version: '3.2.1'
    compile group: 'com.amazonaws', name: 'aws-java-sdk-bundle', version: '1.11.271'

有同样的问题，解决方案来自已工作以加载PartitionedStagingCommitter。您还必须从中下载spark hadoop cloud jar，如解决方案中所述

我还使用了spark 3.0，这个版本的jar很好用

my spark-Default.conf中的设置

spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2
spark.hadoop.fs.s3a.committer.name                           partitioned
spark.hadoop.fs.s3a.committer.magic.enabled                  false
spark.hadoop.fs.s3a.commiter.staging.conflict-mode           append
spark.hadoop.fs.s3a.committer.staging.unique-filenames       true
spark.hadoop.fs.s3a.committer.staging.abort.pending.uploads  true
spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a    
org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
spark.sql.sources.commitProtocolClass                        
org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
spark.sql.parquet.output.committer.class                     
org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter

有同样的问题，解决方案来自已工作以加载PartitionedStagingCommitter。您还必须从中下载spark hadoop cloud jar，如解决方案中所述

我还使用了spark 3.0，这个版本的jar很好用

my spark-Default.conf中的设置

spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2
spark.hadoop.fs.s3a.committer.name                           partitioned
spark.hadoop.fs.s3a.committer.magic.enabled                  false
spark.hadoop.fs.s3a.commiter.staging.conflict-mode           append
spark.hadoop.fs.s3a.committer.staging.unique-filenames       true
spark.hadoop.fs.s3a.committer.staging.abort.pending.uploads  true
spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a    
org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
spark.sql.sources.commitProtocolClass                        
org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
spark.sql.parquet.output.committer.class                     
org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter

我从@Rajadayalan的建议中得到了一个小小的改变。除了第一个问题中的sparkSession.config（）.set（）之外，我在编写拼花地板文件时在df中添加了选项（）参数

df.distinct() .withColumn("date", date_format(col(EFFECTIVE_PERIOD_START), "yyyy-MM-dd")) .repartition(col("date")) .write() .format(fileFormat) .partitionBy("date") .mode(SaveMode.Append) .option("fs.s3a.committer.name", "partitioned") .option("fs.s3a.committer.staging.conflict-mode", "append") .option("spark.sql.sources.commitProtocolClass", "org.apache.spark.internal.io.cloud.PathOutputCommitProtocol") .option("spark.sql.parquet.output.committer.class", "org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter") .option("compression", compressionCodecName.name().toLowerCase()) .save(DOWNLOADS_NON_COMPACT_PATH);
这就产生了不同，下面的stacktrace描述了它使用的PartitionedStagingCommitter
我还可以看到_SUCCESS文件是一个JSON文件，而不是S3中的空触摸文件（_SUCCESS）

我从@Rajadayalan的建议中得到了一个小小的改变。除了第一个问题中的sparkSession.config（）.set（）之外，我在编写拼花地板文件时在df中添加了选项（）参数

df.distinct() .withColumn("date", date_format(col(EFFECTIVE_PERIOD_START), "yyyy-MM-dd")) .repartition(col("date")) .write() .format(fileFormat) .partitionBy("date") .mode(SaveMode.Append) .option("fs.s3a.committer.name", "partitioned") .option("fs.s3a.committer.staging.conflict-mode", "append") .option("spark.sql.sources.commitProtocolClass", "org.apache.spark.internal.io.cloud.PathOutputCommitProtocol") .option("spark.sql.parquet.output.committer.class", "org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter") .option("compression", compressionCodecName.name().toLowerCase()) .save(DOWNLOADS_NON_COMPACT_PATH);
这就产生了不同，下面的stacktrace描述了它使用的PartitionedStagingCommitter
我还可以看到_SUCCESS文件是一个JSON文件，而不是S3中的空触摸文件（_SUCCESS）

非常感谢。但编写器仍然以某种方式使用FileOutputCommitter。请找到我的更新问题的细节。通过一个小的修改，我能够得到这个工作。谢谢，谢谢。但编写器仍然以某种方式使用FileOutputCommitter。请找到我的更新问题的细节。通过一个小的修改，我能够得到这个工作。非常感谢。