Warning: file_get_contents(/data/phpspider/zhask/data//catemap/0/hadoop/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 无法将SnappyCodec与hadoop jar:NullPointerException一起使用_Java_Hadoop_Snappy - Fatal编程技术网

Java 无法将SnappyCodec与hadoop jar:NullPointerException一起使用

Java 无法将SnappyCodec与hadoop jar:NullPointerException一起使用,java,hadoop,snappy,Java,Hadoop,Snappy,我试图在一个简单的java程序中使用Hadoop的压缩库。但是,我无法使用Snappy编解码器:在方法SnappyCodec.createCompressor中执行会产生NullPointerException 注意我没有得到典型的java.lang.UnsatifiedLinkError,这是由于没有设置LD\u LIBRARY\u PATH和java\u LIBRARY\u PATH环境变量而导致的。Snappy与CDH一起正确安装,通过运行hadoop checknative报告可用,当我

我试图在一个简单的java程序中使用Hadoop的压缩库。但是,我无法使用Snappy编解码器:在方法
SnappyCodec.createCompressor
中执行会产生
NullPointerException

注意我没有得到典型的
java.lang.UnsatifiedLinkError
,这是由于没有设置LD\u LIBRARY\u PATH和java\u LIBRARY\u PATH环境变量而导致的。Snappy与CDH一起正确安装,通过运行
hadoop checknative
报告可用,当我对Snappy文件执行
hdfs dfs-text
时,Snappy解压工作

$ hadoop jar SnappyTool-0.0.1-SNAPSHOT.jar com.mycorp.SnappyCompressor
Exception in thread "main" java.lang.NullPointerException
        at org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:145)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
        at com.mycorp.SnappyCompressor.main(SnappyCompressor.java:19)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
$
$ hadoop checknative | grep snappy 2>/dev/null
$ snappy:  true /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/native/libsnappy.so.1
$ ls /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/native/
libhadoop.a       libhadoop.so.1.0.0  libnativetask.a         libsnappy.so
libhadooppipes.a  libhadooputils.a    libnativetask.so        libsnappy.so.1
libhadoop.so      libhdfs.a           libnativetask.so.1.0.0
libsnappy.so.1.1.4
$ export LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/native/
$ java -Djava.library.path=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/native/ -cp `hadoop classpath`:SnappyTool-0.0.1-SNAPSHOT.jar com.mycorp.SnappyCompressor
Exception in thread "main" java.lang.NullPointerException
        at org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:145)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
        at com.mycorp.SnappyCompressor.main(SnappyCompressor.java:19)
Java代码如下所示,最后一行是罪魁祸首:

        SnappyCodec.checkNativeCodeLoaded();
        CompressionCodec codec = new SnappyCodec();
        Compressor comp = CodecPool.getCompressor(codec);

我错过了什么?

好吧,问题是,
压缩编解码器
需要正确的配置,如中所述

配置Snappy压缩器的简单方法如下:

Configuration conf = new Configuration();
CompressionCodecFactory ccf = new CompressionCodecFactory(conf);
CompressionCodec codec = ccf.getCodecByClassName(SnappyCodec.class.getName());
Compressor comp = codec.createCompressor();
生成的jar可以使用原始问题中使用的命令行运行