Python pyspark使用saveAsNewAPIHadoopFile get Warning将数据流数据写入es
这是我的代码:Python pyspark使用saveAsNewAPIHadoopFile get Warning将数据流数据写入es,python,
elasticsearch,pyspark,warnings,Python,
elasticsearch,Pyspark,Warnings,这是我的代码: es_write_conf = { ¦ "es.nodes" : ES_IP, ¦ "es.port" : ES_PORT, ¦ "es.resource" : "%s/%s" % (index, doc_type), ¦ "es.input.json": "true", ¦ # "es.mapping.rich.date": "true" ¦ # "es.mapping.id": "guid
es_write_conf = {
¦ "es.nodes" : ES_IP,
¦ "es.port" : ES_PORT,
¦ "es.resource" : "%s/%s" % (index, doc_type),
¦ "es.input.json": "true",
¦ # "es.mapping.rich.date": "true"
¦ # "es.mapping.id": "guid"
}
dstream.foreachRDD(lambda es_rdd: es_rdd.saveAsNewAPIHadoopFile(
¦ path="-",
¦ outputFormatClass="org.elasticsearch.hadoop.mr.EsOutputFormat",
¦ keyClass="org.apache.hadoop.io.NullWritable",
¦ valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
¦ conf=es_write_conf))
我收到了这个警告
WARN EsOutputFormat: Speculative execution enabled for reducer - consider disabling it to prevent data corruption
如何解决此警告?我通过此解决此问题
es_write_conf = {
¦ "es.nodes" : ES_IP,
¦ "es.port" : ES_PORT,
¦ "es.resource" : "%s/%s" % (index, doc_type),
¦ "es.input.json": "true",
¦ "mapred.reduce.tasks.speculative.execution": "false",
¦ "mapred.map.tasks.speculative.execution": "false",
¦ # "es.mapping.rich.date": "true"
¦ # "es.mapping.id": "guid"
}
dstream.foreachRDD(lambda es_rdd: es_rdd.saveAsNewAPIHadoopFile(
¦ path="-",
¦ outputFormatClass="org.elasticsearch.hadoop.mr.EsOutputFormat",
¦ keyClass="org.apache.hadoop.io.NullWritable",
¦ valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
¦ conf=es_write_conf))
欢迎来到StackOverflow!代码中的符号是什么意思?谢谢,这是我从vim复制的代码,只是用来对齐:)