Java Hadoop 2.6.1警告:WARN util.NativeCodeLoader
我正在OSX10.10.5上运行Hadoop2.6.1。 我收到这个警告: WARN util.NativeCodeLoader:无法为您的平台加载本机hadoop库。。。在适用的情况下使用内置java类 我已经读到,这个问题可能是由运行32位本机库libhadoop.so.1.0和64位版本的hadoop引起的。我已经检查了我的libhadoop.so.1.0.0版本,它是64位的Java Hadoop 2.6.1警告:WARN util.NativeCodeLoader,java,hadoop2,Java,Hadoop2,我正在OSX10.10.5上运行Hadoop2.6.1。 我收到这个警告: WARN util.NativeCodeLoader:无法为您的平台加载本机hadoop库。。。在适用的情况下使用内置java类 我已经读到,这个问题可能是由运行32位本机库libhadoop.so.1.0和64位版本的hadoop引起的。我已经检查了我的libhadoop.so.1.0.0版本,它是64位的 $ find ~/hadoop-2.6.1/ -name libhadoop.so.1.0.0 -ls 1368
$ find ~/hadoop-2.6.1/ -name libhadoop.so.1.0.0 -ls
136889669 1576 -rwxr-xr-x 1 davidlaxer staff 806303 Sep 16 14:18 /Users/davidlaxer/hadoop-2.6.1//lib/native/libhadoop.so.1.0.0
$ file /Users/davidlaxer/hadoop-2.6.1//lib/native/libhadoop.so.1.0.0
/Users/davidlaxer/hadoop-2.6.1//lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
$ env | grep HADOOP
HADOOP_HOME=/Users/davidlaxer/hadoop-2.6.1
HADOOP_COMMON_LIB_NATIVE_DIR=/Users/davidlaxer/hadoop-2.6.1/lib/native
HADOOP_INSTALL=/Users/davidlaxer/hadoop-2.6.1
HADOOP_CONF_DIR=/Users/davidlaxer/hadoop-2.6.1/etc/hadoop
HADOOP_OPTS=-Djava.library.path=/Users/davidlaxer/hadoop-2.6.1/lib
$ hadoop version
Hadoop 2.6.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b4d876d837b830405ccdb6af94742f99d49f9c04
Compiled by jenkins on 2015-09-16T21:07Z
Compiled with protoc 2.5.0
From source with checksum ba9a9397365e3ec2f1b3691b52627f
This command was run using /Users/davidlaxer/hadoop-2.6.1/share/hadoop/common/hadoop-common-2.6.1.jar
$ java -version
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
$ hadoop checknative -a
15/09/26 11:01:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false
openssl: false
15/09/26 11:01:29 INFO util.ExitUtil: Exiting with status 1
$ sudo port list zlib
Password:
zlib @1.2.8 archivers/zlib
$ sudo port list snappy
snappy @1.1.1 archivers/snappy
$ sudo port list lz4
lz4 @r130 archivers/lz4
$ sudo port list bzip2
bzip2 @1.0.6 archivers/bzip2
$ sudo port list openssl
openssl @1.0.2d devel/openssl
$env | grep CLASS
CLASSPATH=/users/davidlaxer/trunk/core/src/test/java/:/Users/davidlaxer/hadoop-2.6.1-src/hadoop-dist/target/hadoop-dist-2.6.1.jar:/Users/davidlaxer/clojure/target:/Users/davidlaxer/hadoop-2.6.1/lib/native:
有什么想法吗?警告的一个原因可能是处理器体系结构存在冲突(32位与64位)。另一种可能是本机库不在java库路径上(或者它需要的库不在路径上) 执行以下操作将为您提供更详细的信息:
hadoop checknative -a
您的HADOOP选项实际上不应该指向lib目录而不是lib/native吗
也许您应该将以下内容添加到log4j.properties文件以获取更多信息:
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=DEBUG
您还可以尝试其他一些方法:
(一)
(检查是否是许可问题)
2) 另外设置LD_LIBRARY_路径。我从源代码构建了hadoop-3.0.0-SNAPSHOT,支持本机代码, 调整了环境变量HADOOP_COMMON_LIB_NATIVE_DIR:
HADOOP_COMMON_LIB_NATIVE_DIR=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native.
现在,hadoop的两个版本(例如2.3.0和3.0快照)都没有得到警告:
HADOOP_HOME=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT
HADOOP_COMMON_LIB_NATIVE_DIR=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native
HADOOP_INSTALL=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT
HADOOP_CONF_DIR=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop
HADOOP_OPTS=-Djava.library.path=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native
$ hadoop version
Hadoop 2.3.0
Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1567123
Compiled by jenkins on 2014-02-11T13:40Z
Compiled with protoc 2.5.0
From source with checksum dfe46336fbc6a044bc124392ec06b85
This command was run using /Users/davidlaxer/hadoop-2.3.0/share/hadoop/common/hadoop-common-2.3.0.jar
$ bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Source code repository https://github.com/apache/hadoop.git -r 83e65c5fe84819b6c6da015b269fb4e46a88d105
Compiled by davidlaxer on 2015-09-26T22:46Z
Compiled with protoc 2.5.0
From source with checksum 883bd12016a9bbe21eb0ae4b6beaa
This command was run using /Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
$ bin/hadoop fs -ls /users/davidlaxer/genomics/reads/HG00103
Found 1 items
drwxr-xr-x - davidlaxer staff 102 2015-09-26 09:13 /users/davidlaxer/genomics/reads/HG00103/_temporary
David-Laxers-MacBook-Pro:hadoop-3.0.0-SNAPSHOT davidlaxer$ bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Source code repository https://github.com/apache/hadoop.git -r 83e65c5fe84819b6c6da015b269fb4e46a88d105
Compiled by davidlaxer on 2015-09-26T22:46Z
Compiled with protoc 2.5.0
From source with checksum 883bd12016a9bbe21eb0ae4b6beaa
This command was run using /Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
我也遇到了同样的问题,通过在作业配置中添加以下xml文件解决了这个问题
Configuration conf = job.getConfiguration();
//To access HDFS files
conf.addResource(new Path(hadoopHome+"/conf/core-site.xml"));//In my case hadoopHome is /usr/lib/hadoop
conf.addResource(new Path(hadoopHome+"/conf/hdfs-site.xml"));
conf.addResource(new Path(hadoopHome+"/conf/mapred-site.xml")); //To run a mapreduce job
/* Other configurations comes here*/
我也有同样的问题。幸运的是,问题已经解决了。 您可以看到下面的解决方案 几次显示警告不允许您将文件上载到HDFS。 因此,当没有警告时,就不存在这样的问题 打开hadoop env.sh并转到文件末尾,尝试添加以下行
export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"
$hadoop checknative-a 15/09/26 11:01:28警告util.NativeCodeLoader:无法为您的平台加载本机hadoop库。。。在适用的地方使用内置java类本机库检查:hadoop:false zlib:false snappy:false lz4:false bzip2:false openssl:false 15/09/26 11:01:29 INFO util.ExitUtil:以状态1env | grep类退出CLASSPATH=/users/davidlaxer/trunk/core/src/test/java/:/users/davidlaxer/hadoop-2.6.1-src/hadoop-dist/target/hadoop-dist-2.6.1.jar:/users/davidlaxer/clojure/target:/users/davidlaxer/hadoop-2.6.1/lib/native:实际上,从checknative的输出来看,您似乎需要安装更多的libraries.port(见上面的文章)显示安装了哪些“本机”库。我缺少什么?$sudo hadoop checknative-密码:15/09/26 11:26:41警告util.NativeCodeLoader:无法为您的平台加载本机hadoop库。。。在适用的地方使用内置java类本机库检查:hadoop:false zlib:false snappy:false lz4:false bzip2:false openssl:false 15/09/26 11:26:42 INFO util.ExitUtil:以状态1退出
export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"