Java 仅在主机上运行的mapreduce任务

Java 仅在主机上运行的mapreduce任务,java,hadoop,mapreduce,bigdata,yarn,Java,Hadoop,Mapreduce,Bigdata,Yarn,我正在尝试使用Hadoop2.6在两节点集群上运行Mapreduce。 我的mapreduce仅在主节点上运行,而不是在从节点上运行 2015-05-18 14:46:43,570 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT] 2015-05-18 14:46:44,311 INFO org.apache.hadoop

我正在尝试使用Hadoop2.6在两节点集群上运行Mapreduce。 我的mapreduce仅在主节点上运行,而不是在从节点上运行

2015-05-18 14:46:43,570 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT]
2015-05-18 14:46:44,311 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
2015-05-18 14:46:44,311 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
2015-05-18 14:46:44,311 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
2015-05-18 14:46:44,312 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
2015-05-18 14:46:44,312 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
2015-05-18 14:46:44,312 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
2015-05-18 14:46:44,325 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
2015-05-18 14:46:44,325 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.NodeManager
2015-05-18 14:46:44,348 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-05-18 14:46:44,388 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-05-18 14:46:44,388 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system started
2015-05-18 14:46:44,402 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
2015-05-18 14:46:44,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: per directory file limit = 8192
2015-05-18 14:46:44,419 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: usercache path : file:/tmp/hadoop-hduser/nm-local-dir/usercache_DEL_1431940604404
2015-05-18 14:46:44,446 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
2015-05-18 14:46:44,467 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.
2015-05-18 14:46:44,467 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Adding auxiliary service httpshuffle, "mapreduce_shuffle"
2015-05-18 14:46:44,489 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:  Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@8662911
2015-05-18 14:46:44,489 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:  Using ResourceCalculatorProcessTree : null
2015-05-18 14:46:44,489 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Physical memory check enabled: true
2015-05-18 14:46:44,489 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Virtual memory check enabled: true
2015-05-18 14:46:44,491 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: NodeManager configured with 8 G physical memory allocated to containers, which is more than 80% of the total physical memory available (3.8 G). Thrashing might happen.
2015-05-18 14:46:44,494 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Initialized nodemanager for null: physical-memory=8192 virtual-memory=17204 virtual-cores=8
2015-05-18 14:46:44,588 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-05-18 14:46:44,648 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 48733
2015-05-18 14:46:44,785 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the server
2015-05-18 14:46:44,785 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Blocking new container-requests as container manager rpc server is still starting.
2015-05-18 14:46:44,786 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-05-18 14:46:44,786 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 48733: starting
2015-05-18 14:46:44,805 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: Updating node address : egap-OptiPlex-3020:48733
2015-05-18 14:46:44,819 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-05-18 14:46:44,819 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8040
2015-05-18 14:46:44,821 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the server
2015-05-18 14:46:44,822 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-05-18 14:46:44,822 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8040: starting
2015-05-18 14:46:44,822 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer started on port 8040
2015-05-18 14:46:44,867 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
2015-05-18 14:46:44,876 INFO org.apache.hadoop.mapred.ShuffleHandler: httpshuffle listening on port 13562
2015-05-18 14:46:44,877 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager started at egap-OptiPlex-3020/127.0.1.1:48733
2015-05-18 14:46:44,877 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager bound to 0.0.0.0/0.0.0.0:0
2015-05-18 14:46:44,882 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:8042
2015-05-18 14:46:44,981 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-05-18 14:46:44,983 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.nodemanager is not defined
2015-05-18 14:46:44,993 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-05-18 14:46:44,994 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context node
2015-05-18 14:46:44,994 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2015-05-18 14:46:44,994 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2015-05-18 14:46:44,997 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /node/*
2015-05-18 14:46:44,997 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2015-05-18 14:46:45,019 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8042
2015-05-18 14:46:45,019 INFO org.mortbay.log: jetty-6.1.26
2015-05-18 14:46:45,051 INFO org.mortbay.log: Extract jar:file:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
2015-05-18 14:46:45,300 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
2015-05-18 14:46:45,300 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 8042
2015-05-18 14:46:45,607 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2015-05-18 14:46:45,646 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /10.182.1.29:8031
2015-05-18 14:46:45,713 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: []
2015-05-18 14:46:45,737 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[]
2015-05-18 14:46:45,924 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: Rolling master-key for container-tokens, got key with id -185600296
2015-05-18 14:46:45,926 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: Rolling master-key for container-tokens, got key with id -442584774
2015-05-18 14:46:45,926 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as egap-OptiPlex-3020:48733 with total resource of <memory:8192, vCores:8>
2015-05-18 14:46:45,926 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying ContainerManager to unblock new container-requests
2015-05-18 14:46:43570 INFO org.apache.hadoop.warn.server.nodemanager.nodemanager:[TERM,HUP,INT]的注册UNIX信号处理程序
2015-05-18 14:46:44311 INFO org.apache.hadoop.warn.event.AsyncDispatcher:注册org.apache.hadoop.warn.server.nodemanager.containermanager.containermanager.ContainerEventType类org.apache.hadoop.warn.server.nodemanager.containermanager.containermanager$ContainerEventDispatcher
2015-05-18 14:46:44311 INFO org.apache.hadoop.warn.event.AsyncDispatcher:为org.apache.hadoop.warn.server.nodemanager.containermanager.application.ApplicationEventType类注册org.apache.hadoop.warn.server.nodemanager.containermanager.containermanager$ApplicationEventDispatcher
2015-05-18 14:46:44311 INFO org.apache.hadoop.warn.event.AsyncDispatcher:为org.apache.hadoop.warn.server.nodemanager.containermanager.localizer.event.LocalizationEventType类注册org.apache.hadoop.warn.server.nodemanager.containermanager.localizer.ResourceLocalizationService类
2015-05-18 14:46:44312信息org.apache.hadoop.warn.event.AsyncDispatcher:为org.apache.hadoop.warn.server.nodemanager.containermanager.AuxServicesEventType类注册org.apache.hadoop.warn.server.nodemanager.containermanager.AuxServices
2015-05-18 14:46:44312信息org.apache.hadoop.warn.event.AsyncDispatcher:注册org.apache.hadoop.warn.server.nodemanager.containermanager.monitor.ContainerMonitor类的org.apache.hadoop.warn.server.nodemanager.containermanager.monitor.ContainerMonitorImpl
2015-05-18 14:46:44312信息org.apache.hadoop.warn.event.AsyncDispatcher:为类org.apache.hadoop.warn.server.nodemanager.containermanager.launcher.ContainerLauncher.ContainerLauncher.EventType注册org.apache.hadoop.warn.server.nodemanager.containermanager.launcher.ContainerLauncher
2015-05-18 14:46:44325信息org.apache.hadoop.warn.event.AsyncDispatcher:为类org.apache.hadoop.warn.server.nodemanager.ContainerManagerReventType注册org.apache.hadoop.warn.server.nodemanager.containermanager.ContainerManagerImpl类
2015-05-18 14:46:44325信息org.apache.hadoop.warn.event.AsyncDispatcher:为类org.apache.hadoop.warn.server.nodemanager.NodeManagerEventType注册org.apache.hadoop.warn.server.nodemanager.nodemanager.nodemanager
2015-05-18 14:46:44348 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:从hadoop-metrics2.properties加载的属性
2015-05-18 14:46:44388 INFO org.apache.hadoop.metrics2.impl.metricsystemimpl:计划的快照时间为10秒。
2015-05-18 14:46:44388 INFO org.apache.hadoop.metrics2.impl.metricsystemimpl:NodeManager度量系统启动
2015-05-18 14:46:44402 INFO org.apache.hadoop.warn.event.AsyncDispatcher:为org.apache.hadoop.warn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType类注册org.apache.hadoop.warn.server.nodemanager.containermanager.loghandler.NoAggregatingLogHandler
2015-05-18 14:46:44402 INFO org.apache.hadoop.warn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:每个目录文件限制=8192
2015-05-18 14:46:44419 INFO org.apache.hadoop.warn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:用户缓存路径:文件:/tmp/hadoop hduser/nm local dir/usercache\u DEL_1431940604404
2015-05-18 14:46:44446 INFO org.apache.hadoop.warn.event.AsyncDispatcher:为org.apache.hadoop.warn.server.nodemanager.containermanager.localizer.event.LocalizerEventType类注册org.apache.hadoop.warn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
2015-05-18 14:46:44467警告org.apache.hadoop.WARN.server.nodemanager.containermanager.AuxServices:配置中名为“mapreduce_shuffle”的辅助服务用于类org.apache.hadoop.mapred.ShuffleHandler,其名称为“httpshuffle”。因为这些不同的工具尝试发送服务数据和读取服务元数据可能会有问题,除非引用配置中的名称。
2015-05-18 14:46:44467 INFO org.apache.hadoop.warn.server.nodemanager.containermanager.AuxServices:添加辅助服务httpshuffle,“mapreduce_shuffle”
2015-05-18 14:46:44489 INFO org.apache.hadoop.warn.server.nodemanager.containermanager.monitor.ContainerMonitorImpl:使用ResourceCalculatorPlugin:org.apache.hadoop.warn.util。LinuxResourceCalculatorPlugin@8662911
2015-05-18 14:46:44489 INFO org.apache.hadoop.warn.server.nodemanager.containermanager.monitor.ContainerMonitorImpl:使用ResourceCalculatorProcessTree:null
2015-05-18 14:46:44489 INFO org.apache.hadoop.warn.server.nodemanager.containermanager.monitor.ContainerMonitorImpl:已启用物理内存检查:true
2015-05-18 14:46:44489 INFO org.apache.hadoop.warn.server.nodemanager.containermanager.monitor.ContainerMonitorImpl:已启用虚拟内存检查:true
2015-05-18 14:46:44491 WARN org.apache.hadoop.WARN.server.nodemanager.containermanager.monitor.ContainerMonitorImpl:nodemanager配置了分配给容器的8G物理内存,占可用物理内存总量(3.8G)的80%以上。可能会发生颠簸。
2015-05-18 14:46:44494 INFO org.apache.hadoop.warn.server.nodemanager.NodeStatusUpdaterImpl:Initialized nodemanager for null:physical memory=8192 virtual memory=17204 virtual cores=8
2015-05-18 14:46:44588 INFO org.apache.hadoop.ipc.CallQueueManager:使用callQueue类java.util.concurrent.LinkedBlockingQueue
2015-05-18 14:46:44648 INFO org.apache.hadoop.ipc.Server:启动端口48733的套接字读取器#1
2015-05-18 14:46:44785 INFO org.apache.hadoop.warn.factories.impl.pb.RpcServerFactoryPBImpl:将协议org.apache.hadoop.warn.api.ContainerManagementProtocolPB添加到服务器
2.