Java Map减少丢工”；EPERM：不允许操作“；使用copyToLocalFile（--）方法时。？

我已经编写了MapReduce代码来将文件从HDFS复制到本地，当我运行MapReduce作业时，它会抛出下面的错误

            Log Upload Time: Thu Jun 29 10:51:22 +0530 2017

            Log Length: 78421677

            Showing 1000000 bytes of 78421677 total. Click 
            here
             for the full log.
          t org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
    at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:393)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:922)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:903)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:800)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:368)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2016)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1985)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1961)
    at com.mani.pmml_mr.PMMLMapper.map(PMMLMapper.java:64)
    at com.mani.pmml_mr.PMMLMapper.map(PMMLMapper.java:35)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
EPERM: Operation not permitted
        at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
        at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:708)
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
        at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
        at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
        at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:393)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:922)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:903)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:800)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:368)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2016)
        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1985)
        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1961)

---

有人能帮我一下吗。

您的datanode是否已启动并正在运行？你能看到hdfs上的文件吗？这是整个stacktrace还是此错误上方有1或2行？是否配置了cgroups？@Philantrover我已验证所有数据节点都已启动并正在运行，并且可以在HDFS中看到文件。我已经更新了日志信息。请您再检查一下好吗。@jedijs:不，我还没有配置cgroup。必须配置它吗。我已为输入和输出路径授予777权限。您可能已为输出目录授予777权限，但

hadoop

可能对

output

的父目录没有足够的权限。尝试将输出路径设置为

/tmp/output

，看看它是否有效

17/06/29 10:51:16 INFO mapreduce.Job: Job job_1495430640647_0231 completed successfully