如何在使用Spring纱线时将资源分配给容器?

如何在使用Spring纱线时将资源分配给容器?,spring,hadoop,resources,yarn,Spring,Hadoop,Resources,Yarn,我正在[github][1]上尝试Spring纱线示例,该示例由gradle构建。我成功地在Thread上运行了custom amservice示例 但我不知道如何为容器分配特殊资源。我尝试在CustomAppmaster的StaticEventingAppmaster类中重写onContainerAllocated和onContainerLaunched方法并分配资源,如下所示 @Override protected void onContainerAllocated(Container co

我正在[github][1]上尝试Spring纱线示例,该示例由gradle构建。我成功地在Thread上运行了custom amservice示例

但我不知道如何为容器分配特殊资源。我尝试在CustomAppmaster的StaticEventingAppmaster类中重写onContainerAllocated和onContainerLaunched方法并分配资源,如下所示

@Override
protected void onContainerAllocated(Container container) {
       //==allocate resource
        Resource resource = new ResourcePBImpl();
        resource.setMemory(1300);
        resource.setVirtualCores(7);
        container.setResource(resource);
        //====
        if (getMonitor() instanceof ContainerAware) {
                ((ContainerAware)getMonitor()).onContainer(Arrays.asList(container));
        }
        getLauncher().launchContainer(container, getCommands());
}
@Override
protected void onContainerLaunched(Container container) {
       //==allocate resource
        Resource resource = new ResourcePBImpl();
        resource.setMemory(1300);
        resource.setVirtualCores(7);
        container.setResource(resource);
        //====
        if (getMonitor() instanceof ContainerAware) {
                ((ContainerAware)getMonitor()).onContainer(Arrays.asList(container));
        }
}
在日志中,它似乎是有效的:

2014-12-30 20:06:35,524 DEBUG [AbstractPollingAllocator] - response has 1 new containers
2014-12-30 20:06:35,525 DEBUG [AbstractPollingAllocator] - new container: container_1419934738198_0004_01_000003

////  this line shows the memory is 1300 and cpu core is 7
2014-12-30 20:06:35,525 DEBUG [DefaultContainerMonitor] - Reporting container=Container: [ContainerId: container_1419934738198_0004_01_000003, NodeId: yarn-master1:57799, NodeHttpAddress: yarn-master1:8042, Resource: <memory:1300, vCores:7>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.170:57799 }, ]

2014-12-30 20:06:35,526 DEBUG [DefaultContainerMonitor] - State after reportContainer: DefaultContainerMonitor [allocated=[container_1419934738198_0004_01_000003,], running=[container_1419934738198_0004_01_000002,], completed=[], failed=[]]

////  this line shows the memory is 1300 and cpu core is 7
2014-12-30 20:06:35,526 DEBUG [DefaultContainerLauncher] - Launching container: Container: [ContainerId: container_1419934738198_0004_01_000003, NodeId: yarn-master1:57799, NodeHttpAddress: yarn-master1:8042, Resource: <memory:1300, vCores:7>, Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.170:57799 }, ] with commands $JAVA_HOME/bin/java,org.springframework.yarn.container.CommandLineContainerRunner,container-context.xml,yarnContainer,1><LOG_DIR>/Container.stdout,2><LOG_DIR>/Container.stderr
关键点是,在日志中:。当前使用情况:使用了86.6 MB的1 GB物理内存,而不是1.3 GB


所以我认为我的方法没有起作用。有谁能告诉我如何正确分配资源吗?

这是《纱线》中的一个问题领域,我相信随着越来越多的非MR应用程序在《纱线》中使用,这一领域最终会变得更好。我相信您的设置是正确的,但纱线的一点奇怪行为导致了这些问题。目前,从应用程序的角度来看,我们所能做的很少,因为大多数mem设置都是在纱线本身中强制执行的,来自应用程序的请求只是请求

纱线上的Spring XD依赖于同样的东西,值得检查我们在其文档中写的内容:。请参阅“配置内存保留”部分

我会尽力确保同样的信息也会传到我们的SpringHadoop和SpringThreadRef文档中

2014-12-30 20:07:05,929 DEBUG [AbstractPollingAllocator] - response has 1 completed containers

//The Same container was stopped because it beyond the limits.
2014-12-30 20:07:05,932 DEBUG [AbstractPollingAllocator] - completed container: container_1419934738198_0004_01_000003 with status=ContainerStatus: [ContainerId: container_1419934738198_0004_01_000003, State: COMPLETE, Diagnostics: Container [pid=10587,containerID=container_1419934738198_0004_01_000003] is running beyond virtual memory limits. Current usage: 86.6 MB of 1 GB physical memory used; 31.8 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1419934738198_0004_01_000003 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 10587 32315 10587 10587 (bash) 2 3 12652544 353 /bin/bash -c /home/novelbio/software/jdk//bin/java org.springframework.yarn.container.CommandLineContainerRunner container-context.xml yarnContainer 1>/home/novelbio/software/hadoop/logs/userlogs/application_1419934738198_0004/container_1419934738198_0004_01_000003/Container.stdout 2>/home/novelbio/software/hadoop/logs/userlogs/application_1419934738198_0004/container_1419934738198_0004_01_000003/Container.stderr 
    |- 10761 10587 10587 10587 (java) 108 10 34135896064 21811 /home/novelbio/software/jdk//bin/java org.springframework.yarn.container.CommandLineContainerRunner container-context.xml yarnContainer 

, ExitStatus: 0, ]