未找到压缩编解码器com.hadoop.Compression.lzo.LzoCodec
尝试使用压缩运行mapreduce作业未找到压缩编解码器com.hadoop.Compression.lzo.LzoCodec,hadoop,mapreduce,hadoop-lzo,Hadoop,Mapreduce,Hadoop Lzo,尝试使用压缩运行mapreduce作业 hadoop jar \ /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \ randomtextwriter \ -Ddfs.replication=1 -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.co
hadoop jar \
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
randomtextwriter \
-Ddfs.replication=1 -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzoCodec \
/tmp/randomtextwriter
使用地块将lzo分布到群集中的所有节点。即使这样,我还是犯了下面的错误
Getting below error
Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec was not found.
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:140)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:56)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:659)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2409)
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:138)
... 10 more
获取下面的错误
错误:java.lang.IllegalArgumentException:未找到压缩编解码器com.hadoop.Compression.lzo.LzoCodec。
位于org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:140)
位于org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:56)
位于org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
位于org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector。(MapTask.java:659)
位于org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
位于org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
位于org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
位于java.security.AccessController.doPrivileged(本机方法)
位于javax.security.auth.Subject.doAs(Subject.java:422)
位于org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)
位于org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
原因:java.lang.ClassNotFoundException:Class com.hadoop.compression.lzo.LzoCodec未找到
位于org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2409)
位于org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:138)
... 10多
作为临时解决方案,您可以在hadoop类路径中手动添加hadoop lzo jar
curl https://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/hadoop-lzo-0.4.19.jar
hadoop jar \ /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \ randomtextwriter \ -Ddfs.replication=1 -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzoCodec \ /tmp/randomtextwriter --libjars hadoop-lzo-0.4.19.jar
请确保下载与hadoop版本兼容的hadoop lzo。您好,它工作正常。只需要在输出目录之前添加--libjars hadoop-lzo-0.4.19.jar。您能告诉我如何在不使用libjar的情况下永久添加这些吗?您可以将这个jar添加到hadoop类路径中