如果使用spring集成的远程分区中不存在资源路径,则spring批处理中的MessageChannelPartitionHandler的行为如何?
我正在使用SpringBatch远程分区。下面是我的配置如果使用spring集成的远程分区中不存在资源路径,则spring批处理中的MessageChannelPartitionHandler的行为如何?,spring,batch-processing,spring-batch,spring-integration,spring-batch-admin,Spring,Batch Processing,Spring Batch,Spring Integration,Spring Batch Admin,我正在使用SpringBatch远程分区。下面是我的配置 <task:executor id="taskExecutor" pool-size="50" /> <rabbit:template id="computeAmqpTemplate" connection-factory="rabbitConnectionFactory" routing-key="computeQueue" reply-timeout="${compute.partition.tim
<task:executor id="taskExecutor" pool-size="50" />
<rabbit:template id="computeAmqpTemplate"
connection-factory="rabbitConnectionFactory" routing-key="computeQueue"
reply-timeout="${compute.partition.timeout}">
</rabbit:template>
<int:channel id="computeOutboundChannel">
<int:dispatcher task-executor="taskExecutor" />
</int:channel>
<int:channel id="computeInboundStagingChannel" />
<amqp:outbound-gateway request-channel="computeOutboundChannel"
reply-channel="computeInboundStagingChannel" amqp-template="computeAmqpTemplate"
mapped-request-headers="correlationId, sequenceNumber, sequenceSize, STANDARD_REQUEST_HEADERS"
mapped-reply-headers="correlationId, sequenceNumber, sequenceSize, STANDARD_REQUEST_HEADERS" />
<beans:bean id="computeMessagingTemplate"
class="org.springframework.integration.core.MessagingTemplate"
p:defaultChannel-ref="computeOutboundChannel"
p:receiveTimeout="${compute.partition.timeout}" />
<beans:bean id="computePartitionHandler"
class="org.springframework.batch.integration.partition.MessageChannelPartitionHandler"
p:stepName="computeStep" p:gridSize="${compute.grid.size}"
p:messagingOperations-ref="computeMessagingTemplate" />
<int:aggregator ref="computePartitionHandler"
send-partial-result-on-expiry="true" send-timeout="${compute.step.timeout}"
input-channel="computeInboundStagingChannel" />
<amqp:inbound-gateway concurrent-consumers="${compute.consumer.concurrency}"
request-channel="computeInboundChannel"
reply-channel="computeOutboundStagingChannel" queue-names="computeQueue"
connection-factory="rabbitConnectionFactory"
mapped-request-headers="correlationId, sequenceNumber, sequenceSize, STANDARD_REQUEST_HEADERS"
mapped-reply-headers="correlationId, sequenceNumber, sequenceSize, STANDARD_REQUEST_HEADERS" />
<int:channel id="computeInboundChannel" />
<int:service-activator ref="stepExecutionRequestHandler"
input-channel="computeInboundChannel" output-channel="computeOutboundStagingChannel" />
<int:channel id="computeOutboundStagingChannel" />
<beans:bean id="computePartitioner"
class="org.springframework.batch.core.partition.support.MultiResourcePartitioner"
p:resources="file:${spring.tmp.batch.dir}/#{jobParameters[batch_id]}/shares_rics/shares_rics_*.txt"
scope="step" />
<beans:bean id="computeFileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader"
p:resource="#{stepExecutionContext[fileName]}" p:lineMapper-ref="stLineMapper"
scope="step" />
<beans:bean id="computeItemWriter"
class="com.st.batch.foundation.writers.ComputeItemWriter"
p:symfony-ref="symfonyStepScoped" p:timeout="${compute.item.timeout}"
p:batchId="#{jobParameters[batch_id]}" scope="step" />
<step id="computeStep">
<tasklet transaction-manager="transactionManager">
<chunk reader="computeFileItemReader" writer="computeItemWriter"
commit-interval="${compute.commit.interval}" />
</tasklet>
</step>
<flow id="computeFlow">
<step id="computeStep.master">
<partition partitioner="computePartitioner"
handler="computePartitionHandler" />
</step>
</flow>
<job id="computeJob" restartable="true">
<flow id="computeJob.computeFlow" parent="computeFlow" />
</job>
问题是,在MultiResourcePartitioner中,我传递模式以查找文件并创建与文件数量相等的分区。但目录是在运行时有条件地创建的
如果目录不存在(输入文件不可用),我希望此步骤成功,然后继续下一步
现在,工作只是暂时的,什么都做不了。它既不认为该步骤成功,也不抛出异常,因此甚至不会失败。在这一步它只是变为空闲
<beans:bean id="computePartitioner"
class="org.springframework.batch.core.partition.support.MultiResourcePartitioner"
p:resources="file:${spring.tmp.batch.dir}/#{jobParameters[batch_id]}/shares_rics/shares_rics_*.txt"
scope="step" />
有什么办法处理这个问题吗?我只想把这一步看作是成功的,然后进入下一步。
Upate:
刚刚用创建分区步骤测试了它,该步骤运行在本地服务器而不是远程服务器上,若文件不存在,则默认情况下会将该步骤标记为已完成。所以问题不在于MultiResourcePartitioner,而是在使用分区步骤在具有上述配置的远程服务器上运行时出现的问题
我猜,它的聚合逻辑一直在等待响应,即使没有发送步骤执行消息?这是因为默认的SequenceSizeReleaseStrategy依赖于IntegrationMessageHeaderAccessor.SEQUENCE\u SIZE标头属性,并且由于根本没有消息,聚合器无法访问SEQUENCE\u SIZE
@MessageEndpoint
public class MessageChannelPartitionHandler implements PartitionHandler {
public Collection<StepExecution> handle(StepExecutionSplitter stepExecutionSplitter,
StepExecution masterStepExecution) throws Exception {
Set<StepExecution> split = stepExecutionSplitter.split(masterStepExecution, gridSize);
int count = 0;
if (replyChannel == null) {
replyChannel = new QueueChannel();
}//end if
for (StepExecution stepExecution : split) {
Message<StepExecutionRequest> request = createMessage(count++, split.size(), new StepExecutionRequest(
stepName, stepExecution.getJobExecutionId(), stepExecution.getId()), replyChannel);
if (logger.isDebugEnabled()) {
logger.debug("Sending request: " + request);
}
messagingGateway.send(request);
}
Message<Collection<StepExecution>> message = messagingGateway.receive(replyChannel);
if (logger.isDebugEnabled()) {
logger.debug("Received replies: " + message);
}
Collection<StepExecution> result = message.getPayload();
return result;
}
private Message<StepExecutionRequest> createMessage(int sequenceNumber, int sequenceSize,
StepExecutionRequest stepExecutionRequest, PollableChannel replyChannel) {
return MessageBuilder.withPayload(stepExecutionRequest).setSequenceNumber(sequenceNumber)
.setSequenceSize(sequenceSize)
.setCorrelationId(stepExecutionRequest.getJobExecutionId() + ":" + stepExecutionRequest.getStepName())
.setReplyChannel(replyChannel)
.build();
}
}
@MessageEndpoint
公共类MessageChannelPartitionHandler实现PartitionHandler{
公共集合句柄(StepExecutionSplitter StepExecutionSplitter,
StepExecution(masterStepExecution)引发异常{
设置拆分=stepExecutionSplitter.split(masterStepExecution,gridSize);
整数计数=0;
if(replyChannel==null){
replyChannel=new QueueChannel();
}//如果结束
用于(步骤执行步骤执行:拆分){
Message request=createMessage(count++,split.size(),new-StepExecutionRequest(
stepName、stepExecution.getJobExecutionId()、stepExecution.getId()、replyChannel);
if(logger.isDebugEnabled()){
调试(“发送请求:+请求”);
}
messagingGateway.send(请求);
}
Message Message=messagingGateway.receive(replyChannel);
if(logger.isDebugEnabled()){
调试(“收到回复:“+消息”);
}
收集结果=message.getPayload();
返回结果;
}
私有消息createMessage(int sequenceNumber、int sequenceSize、,
StepExecutionRequest StepExecutionRequest,可污染通道回复通道){
返回MessageBuilder.withPayload(stepExecutionRequest).setSequenceNumber(sequenceNumber)
.setSequenceSize(序列大小)
.setCorrelationId(stepExecutionRequest.getJobExecutionId()+”:“+stepExecutionRequest.getStepName())
.setReplyChannel(replyChannel)
.build();
}
}
如果没有步骤执行请求,即拆分计数为0,则它不会进入forloop内部,因此不会发送任何消息,但在for loop之后,它将等待接收响应。
解决方案是什么?使用检查目录并返回CONTINUE或SKIP(或任何其他有意义的值)我在spring批处理错误跟踪器Jira上提出了一个问题 MessageChannelPartitionHandler的问题是,到目前为止它还不能处理这种情况。临时修复方法是重写MessageChannelPartitionHandler.handle,检查消息集的大小,并按照上面提出的建议返回NULL 这个方法应该是这样的,它对我有效
public Collection<StepExecution> handle(StepExecutionSplitter stepExecutionSplitter,
StepExecution masterStepExecution) throws Exception {
Set<StepExecution> split = stepExecutionSplitter.split(masterStepExecution, gridSize);
if(split.size() == 0) {
return null;
}
int count = 0;
if (replyChannel == null) {
replyChannel = new QueueChannel();
}//end if
for (StepExecution stepExecution : split) {
Message<StepExecutionRequest> request = createMessage(count++, split.size(), new StepExecutionRequest(
stepName, stepExecution.getJobExecutionId(), stepExecution.getId()), replyChannel);
if (logger.isDebugEnabled()) {
logger.debug("Sending request: " + request);
}
messagingGateway.send(request);
}
if (logger.isDebugEnabled()) {
logger.debug("No message sent but waiting for reply: ");
}
Message<Collection<StepExecution>> message = messagingGateway.receive(replyChannel);
if (logger.isDebugEnabled()) {
logger.debug("Received replies: " + message);
}
Collection<StepExecution> result = message.getPayload();
return result;
}
公共集合句柄(StepExecutionSplitter StepExecutionSplitter,
StepExecution(masterStepExecution)引发异常{
设置拆分=stepExecutionSplitter.split(masterStepExecution,gridSize);
if(split.size()==0){
返回null;
}
整数计数=0;
if(replyChannel==null){
replyChannel=new QueueChannel();
}//如果结束
用于(步骤执行步骤执行:拆分){
Message request=createMessage(count++,split.size(),new-StepExecutionRequest(
stepName、stepExecution.getJobExecutionId()、stepExecution.getId()、replyChannel);
if(logger.isDebugEnabled()){
调试(“发送请求:+请求”);
}
messagingGateway.send(请求);
}
if(logger.isDebugEnabled()){
debug(“未发送消息,但正在等待答复:”);
}
Message Message=messagingGateway.receive(replyChannel);
if(logger.isDebugEnabled()){
调试(“收到回复:“+消息”);
}
收集结果=message.getPayload();
返回结果;
}
如果我使用decider,我将不得不在所有有相同问题的步骤中使用它。是否有我可以应用于所有步骤的任何内容,即,如果任何步骤的输入文件不可用,则这些步骤应标记为已完成,并应继续到下一步可能会将FlatFileItemReader设置为。将strict
设置为false是一种解决方案?在“非严格”模式下,“缺少资源”条件不会引发任何错误,但返回“无数据”。我尝试过,但不起作用。如果我在一台机器上使用分区,它工作得很好。但当我使用MessageChannelPartitionHandler将其用于远程机器时,它就不起作用了