Java Hadoop 2.x中的Snappy压缩错误

我已经使用最新的2.x版本设置了Hadoop集群。我根据安装了snappy和hadoop snappy，以便在地图输出中启用snappy压缩

运行示例wordcount时，出现以下错误：

[dm@node1 ~]$ hadoop jar /opt/hadoop-2.0.5-alpha/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.5-alpha.jar wordcount /in /out
13/09/06 05:09:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/09/06 05:09:53 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/09/06 05:09:53 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
13/09/06 05:10:04 INFO input.FileInputFormat: Total input paths to process : 1
13/09/06 05:10:04 INFO snappy.LoadSnappy: Snappy native library loaded
13/09/06 05:10:04 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
13/09/06 05:10:04 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev d0f5a10f99f1b2af4f6610447052c5a67b8b1cc7]
13/09/06 05:10:04 INFO mapreduce.JobSubmitter: number of splits:1
13/09/06 05:10:04 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
13/09/06 05:10:04 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
13/09/06 05:10:04 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
13/09/06 05:10:04 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
13/09/06 05:10:04 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
13/09/06 05:10:04 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
13/09/06 05:10:04 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
13/09/06 05:10:04 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/09/06 05:10:04 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
13/09/06 05:10:04 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
13/09/06 05:10:04 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
13/09/06 05:10:05 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1378415309099_0001
13/09/06 05:10:06 INFO client.YarnClientImpl: Submitted application application_1378415309099_0001 to ResourceManager at node1/192.168.56.101:60832
13/09/06 05:10:06 INFO mapreduce.Job: The url to track the job: http://node1:60888/proxy/application_1378415309099_0001/
13/09/06 05:10:06 INFO mapreduce.Job: Running job: job_1378415309099_0001
13/09/06 05:10:32 INFO mapreduce.Job: Job job_1378415309099_0001 running in uber mode : false
13/09/06 05:10:32 INFO mapreduce.Job:  map 0% reduce 0%
13/09/06 05:10:53 INFO mapreduce.Job:  map 100% reduce 0%
13/09/06 05:11:02 INFO mapreduce.Job: Task Id : attempt_1378415309099_0001_m_000000_0, Status : FAILED
Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
Container killed by the ApplicationMaster.


13/09/06 05:11:03 INFO mapreduce.Job:  map 0% reduce 0%
13/09/06 05:11:07 INFO mapreduce.Job: Task Id : attempt_1378415309099_0001_m_000000_1, Status : FAILED
Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
13/09/06 05:11:13 INFO mapreduce.Job: Task Id : attempt_1378415309099_0001_m_000000_2, Status : FAILED
Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
13/09/06 05:11:19 INFO mapreduce.Job:  map 100% reduce 0%
13/09/06 05:11:19 INFO mapreduce.Job: Job job_1378415309099_0001 failed with state FAILED due to: Task failed task_1378415309099_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

13/09/06 05:11:19 INFO mapreduce.Job: Counters: 6
    Job Counters 
        Failed map tasks=4
        Launched map tasks=4
        Other local map tasks=3
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=42989
        Total time spent by all reduces in occupied slots (ms)=0

---

我在google上搜索了错误消息“error:org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy（）Z”，但没有找到这个问题的解决方案。所以我想知道如何在Hadoop 2.x中启用snappy压缩？谢谢。

该指南中的第二个参数错误

---

使用mapreduce.map.output.compress.codec。

从您提供的输出（

WARN util.NativeCodeLoader:无法为您的平台加载本机hadoop库…

等…）看起来您的本机库没有加载。我认为应该尝试使用本机库，然后在

/lib/native/

中添加Snappy库。我认为这可能会解决这个问题。希望有帮助

您的代码似乎无法加载snappy本机库。对此可能有两种解释。 您使用的snappy版本与您安装的hadoop版本不兼容。 或者您没有在类路径中添加库。

好服务器：

$ hadoop checknative -a | grep snappy
15/06/18 14:51:05 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
15/06/18 14:51:05 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
snappy:  true /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/lib/native/libsnappy.so.1

$ hadoop checknative -a | grep snappy
15/06/18 14:50:31 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
15/06/18 14:50:31 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
snappy:  true /usr/lib64/libsnappy.so.1

服务器不好：

$ hadoop checknative -a | grep snappy
15/06/18 14:51:05 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
15/06/18 14:51:05 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
snappy:  true /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/lib/native/libsnappy.so.1

$ hadoop checknative -a | grep snappy
15/06/18 14:50:31 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
15/06/18 14:50:31 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
snappy:  true /usr/lib64/libsnappy.so.1

注意，在一种情况下，“hadoop checknative”命令返回系统的libsnapy，在另一种情况下，返回一个来自hadoop发行版的libsnapy。如果您在某些主机上安装了hadoop，则会报告它们使用了系统的libsnapy，错误如下“.hadoop.mapred.YarnChild:Error运行child:java.lang.UnsatisfiedLinkError:org.apache.hadoop.util.NativeCodeLoader.buildSupportsNappy（）Z 位于org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy（本机方法） “很有可能