Java dfs.data.dir中的目录无效:无法将路径:\tmp\hadoop user\dfs\data的权限设置为0755
我不熟悉hadoop框架,目前正在使用Windows7中的cygwin、hadoop-0.19.1、eclipse-3.3.1(Europa)处理大数据项目。现在我正在尝试从hadoop-0.19.1更改为hadoop-1.2.1版本 core-site.xml:Java dfs.data.dir中的目录无效:无法将路径:\tmp\hadoop user\dfs\data的权限设置为0755,java,eclipse,hadoop,cygwin,Java,Eclipse,Hadoop,Cygwin,我不熟悉hadoop框架,目前正在使用Windows7中的cygwin、hadoop-0.19.1、eclipse-3.3.1(Europa)处理大数据项目。现在我正在尝试从hadoop-0.19.1更改为hadoop-1.2.1版本 core-site.xml: <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-speci
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9100</value>
</property>
</configuration>
。我尝试了此github站点的windows修补程序:
它解决了我的问题。但若我试图在eclipse中运行简单的字数计算程序,它会在那个里给我同样的错误:“未能将路径:\tmp\hadoop user\dfs\data的权限设置为0755”
我在谷歌上搜索了很多次,但没有找到任何解决方案
任何人都请帮忙
如前所述更改hdfs-site.xml后
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>C:/cygwin/usr/tmp/hadoop-raj/dfs/logs</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>c:/cygwin/usr/hadoop-raj/dfs/data</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>c:/cygwin/usr/hadoop-raj/dfs/tmp</value>
</property>
</configuration>
dfs.replication
1.
dfs.name.dir
C:/cygwin/usr/tmp/hadoop-raj/dfs/logs
dfs.data.dir
c:/cygwin/usr/hadoop-raj/dfs/data
hadoop.tmp.dir
c:/cygwin/usr/hadoop-raj/dfs/tmp
现在我在tasktracker上发现以下错误
$ bin/hadoop tasktracker
14/02/08 15:25:22 INFO mapred.TaskTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = raj-PC/192.168.42.248
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/b ranch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_45
************************************************************/
14/02/08 15:25:23 INFO impl.MetricsConfig: loaded properties from hadoop-metrics 2.properties
14/02/08 15:25:23 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem ,sub=Stats registered.
14/02/08 15:25:23 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 s econd(s).
14/02/08 15:25:23 INFO impl.MetricsSystemImpl: TaskTracker metrics system starte d
14/02/08 15:25:23 WARN util.NativeCodeLoader: Unable to load native-hadoop libra ry for your platform... using builtin-java classes where applicable
14/02/08 15:25:23 INFO impl.MetricsSourceAdapter: MBean for source ugi registere d.
14/02/08 15:25:23 WARN impl.MetricsSystemImpl: Source name ugi already exists!
14/02/08 15:25:23 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter (org.mortbay.log) via org.mortbay.log.Slf4jLog
14/02/08 15:25:23 INFO http.HttpServer: Added global filtersafety (class=org.apa che.hadoop.http.HttpServer$QuotingInputFilter)
14/02/08 15:25:23 INFO mapred.TaskLogsTruncater: Initializing logs' truncater wi th mapRetainSize=-1 and reduceRetainSize=-1
14/02/08 15:25:23 INFO mapred.TaskTracker: Starting tasktracker with owner as ra j
14/02/08 15:25:23 INFO mapred.TaskTracker: Good mapred local directories are: /t mp/hadoop-raj/mapred/local
14/02/08 15:25:23 ERROR mapred.TaskTracker: Can not start task tracker because j ava.io.IOException: Failed to set permissions of path: \tmp\hadoop-raj\mapred\lo cal\taskTracker to 0755
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:672)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys tem.java:514)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.jav a:349)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:19 3)
at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:823)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1573)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3937)
14/02/08 15:25:23 INFO mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at raj-PC/192.168.42.248
************************************************************/
$bin/hadoop任务跟踪器
14/02/08 15:25:22信息映射。任务跟踪器:启动\u消息:
/************************************************************
启动消息:正在启动TaskTracker
启动消息:host=raj PC/192.168.42.248
启动消息:args=[]
启动消息:版本=1.2.1
启动\u消息:生成=https://svn.apache.org/repos/asf/hadoop/common/branches/b ranch-1.2-r 1503152;由“mattf”于2013年7月22日(星期一)15:23:09编制
启动消息:java=1.7.0\u 45
************************************************************/
14/02/08 15:25:23 INFO impl.MetricsConfig:从hadoop metrics 2.properties加载的属性
14/02/08 15:25:23信息impl.MetricsSourceAdapter:MBean用于源MetricsSystem,sub=Stats registered。
14/02/08 15:25:23信息impl.MetricsSystemImpl:计划的快照周期为10秒。
14/02/08 15:25:23信息实施度量系统实施:TaskTracker度量系统启动
14/02/08 15:25:23警告util.NativeCodeLoader:无法为您的平台加载本机hadoop库。。。在适用的情况下使用内置java类
14/02/08 15:25:23信息impl.MetricsSourceAdapter:MBean用于源ugi注册表d。
14/02/08 15:25:23警告impl.MetricsSystemImpl:源名称ugi已存在!
14/02/08 15:25:23信息mortbay.log:通过org.mortbay.log.slf4jloggeradapter(org.mortbay.log)登录到org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
14/02/08 15:25:23信息http.HttpServer:添加了全局过滤器安全性(class=org.apa che.hadoop.http.HttpServer$QuotingInputFilter)
14/02/08 15:25:23 INFO mapred.TaskLogsTruncater:使用mapRetainSize=-1和reduceRetainSize=-1初始化日志的截断器
14/02/08 15:25:23信息映射。TaskTracker:启动TaskTracker,所有者为ra j
14/02/08 15:25:23 INFO mapred.TaskTracker:Good mapred本地目录是:/t mp/hadoop raj/mapred/local
14/02/08 15:25:23错误映射。任务跟踪器:无法启动任务跟踪器,因为j ava.io.IOException:未能设置路径:\tmp\hadoop raj\mapred\lo的权限cal\taskTracker至0755
位于org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
位于org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:672)
位于org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys tem.java:514)
位于org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
位于org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:19 3)
位于org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:823)
位于org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1573)
位于org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3937)
14/02/08 15:25:23信息映射。任务跟踪器:关机\u消息:
/************************************************************
关闭消息:关闭raj PC/192.168.42.248上的TaskTracker
************************************************************/
这是一个Cygwin问题。您需要将所有本地目录更改为Cygwin目录,例如c:/Cygwin/usr/tmp/hadoop-user/dfs/data
一定要这样做
$ bin/hadoop datanode
14/02/07 22:24:09 INFO datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = raj-PC/101.63.181.80
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/b ranch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_45
************************************************************/
14/02/07 22:24:10 INFO impl.MetricsConfig: loaded properties from hadoop-metrics 2.properties
14/02/07 22:24:10 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem ,sub=Stats registered.
14/02/07 22:24:10 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 s econd(s).
14/02/07 22:24:10 INFO impl.MetricsSystemImpl: DataNode metrics system started
14/02/07 22:24:11 INFO impl.MetricsSourceAdapter: MBean for source ugi registere d.
14/02/07 22:24:11 WARN impl.MetricsSystemImpl: Source name ugi already exists!
14/02/07 22:24:11 WARN util.NativeCodeLoader: Unable to load native-hadoop libra ry for your platform... using builtin-java classes where applicable
14/02/07 22:24:11 WARN datanode.DataNode: Invalid directory in dfs.data.dir: Fai led to set permissions of path: \tmp\hadoop-raj\dfs\data to 0755
14/02/07 22:24:11 ERROR datanode.DataNode: All directories in dfs.data.dir are i nvalid.
14/02/07 22:24:11 INFO datanode.DataNode: Exiting Datanode
14/02/07 22:24:11 INFO datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at raj-PC/101.63.181.80
************************************************************/
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>C:/cygwin/usr/tmp/hadoop-raj/dfs/logs</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>c:/cygwin/usr/hadoop-raj/dfs/data</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>c:/cygwin/usr/hadoop-raj/dfs/tmp</value>
</property>
</configuration>
$ bin/hadoop tasktracker
14/02/08 15:25:22 INFO mapred.TaskTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = raj-PC/192.168.42.248
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/b ranch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_45
************************************************************/
14/02/08 15:25:23 INFO impl.MetricsConfig: loaded properties from hadoop-metrics 2.properties
14/02/08 15:25:23 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem ,sub=Stats registered.
14/02/08 15:25:23 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 s econd(s).
14/02/08 15:25:23 INFO impl.MetricsSystemImpl: TaskTracker metrics system starte d
14/02/08 15:25:23 WARN util.NativeCodeLoader: Unable to load native-hadoop libra ry for your platform... using builtin-java classes where applicable
14/02/08 15:25:23 INFO impl.MetricsSourceAdapter: MBean for source ugi registere d.
14/02/08 15:25:23 WARN impl.MetricsSystemImpl: Source name ugi already exists!
14/02/08 15:25:23 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter (org.mortbay.log) via org.mortbay.log.Slf4jLog
14/02/08 15:25:23 INFO http.HttpServer: Added global filtersafety (class=org.apa che.hadoop.http.HttpServer$QuotingInputFilter)
14/02/08 15:25:23 INFO mapred.TaskLogsTruncater: Initializing logs' truncater wi th mapRetainSize=-1 and reduceRetainSize=-1
14/02/08 15:25:23 INFO mapred.TaskTracker: Starting tasktracker with owner as ra j
14/02/08 15:25:23 INFO mapred.TaskTracker: Good mapred local directories are: /t mp/hadoop-raj/mapred/local
14/02/08 15:25:23 ERROR mapred.TaskTracker: Can not start task tracker because j ava.io.IOException: Failed to set permissions of path: \tmp\hadoop-raj\mapred\lo cal\taskTracker to 0755
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:672)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys tem.java:514)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.jav a:349)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:19 3)
at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:823)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1573)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3937)
14/02/08 15:25:23 INFO mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at raj-PC/192.168.42.248
************************************************************/