Spring batch Spring批处理分区-分区步骤未执行
我使用批处理分区来根据DB表中某一列的某个范围获取和处理记录。分区步骤stagingPartitionStep在作业配置中的几个步骤之后出现,如下所示:Spring batch Spring批处理分区-分区步骤未执行,spring-batch,Spring Batch,我使用批处理分区来根据DB表中某一列的某个范围获取和处理记录。分区步骤stagingPartitionStep在作业配置中的几个步骤之后出现,如下所示: <batch:job id="myBatchJob"> <batch:step id="mainStep"> <batch:tasklet task-executor="myTaskExecutor" transaction-manager="batchTransactionMan
<batch:job id="myBatchJob">
<batch:step id="mainStep">
<batch:tasklet task-executor="myTaskExecutor" transaction-manager="batchTransactionManager">
<batch:chunk reader="myDataReader"
processor="MyDataProcessor" writer="MyDataWriter" commit-interval="50">
</batch:chunk>
<batch:listeners>
<batch:listener ref="myItemListener" />
</batch:listeners>
</batch:tasklet>
<batch:next on="COMPLETED" to="checkConditionStep" />
</batch:step>
<batch:step id="checkConditionStep">
<batch:tasklet task-executor="checkConditionExecutor"
transaction-manager="batchTransactionManager" ref="checkConditionTasklet">
</batch:tasklet>
<batch:next on="FAILED" to="updateStagingTableStep" />
<batch:next on="COMPLETED" to="stagingPartitionStep" />
</batch:step>
<batch:step id="updateStagingTableStep">
<batch:tasklet task-executor="checkConditionExecutor"
transaction-manager="batchTransactionManager" ref="updateStagingTasklet">
</batch:tasklet>
</batch:step>
<batch:step id="stagingPartitionStep">
<batch:partition step="processStagingStep" partitioner="stagingProcessPartitioner">
<batch:handler grid-size="10" task-executor="stagingProcessTaskExecutor" />
</batch:partition>
</batch:step>
<batch:job>
分区器和分区步骤:
<bean id="stagingProcessPartitioner"
class="com.mycom.batch.partitioner.StagingProcessPartitioner"
scope="step">
</bean>
<batch:step id="processStagingStep">
<batch:tasklet transaction-manager="batchTransactionManager">
<batch:chunk reader="stagingProcessorDataReader" writer="stagingProcessorDataWriter"
commit-interval="50">
</batch:chunk>
</batch:tasklet>
</batch:step>
任务执行者:
<bean id="myTaskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="20" />
<property name="maxPoolSize" value="20" />
</bean>
<bean id="stagingProcessTaskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="10" />
<property name="maxPoolSize" value="10" />
<property name="allowCoreThreadTimeOut" value="true" />
</bean>
<bean id="checkConditionStep"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="1" />
<property name="maxPoolSize" value="1" />
</bean>
分区器实现:分区器为每个分区步骤创建ExecutionContext,并放置一个前缀值,该值将在JDBC查询中用于根据凭证号创建分区
public class StagingProcessPartitioner implements Partitioner{
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> partitionMap = new HashMap<String, ExecutionContext>();
for(int threadId = 0; threadId < gridSize; threadId++){
ExecutionContext context = new ExecutionContext();
String stepName = "step" + threadId;
context.put("voucherSuffix", threadId);
partitionMap.put(stepName, context);
LOGGER.info("Created ExecutionContext for partioned step : " + stepName);
}
return partitionMap;
}}
公共类StagingProcessPartitioner实现了分区器{
公共地图分区(int gridSize){
Map partitionMap=newhashmap();
对于(int-threadId=0;threadId
数据读取器:步骤上下文中的voucherSuffix在JDBC查询中用于创建数据分区。因此,应在以0、1、2、9结尾的凭证编号上创建10个分区
<bean id="stagingProcessorDataReader"
class="org.springframework.batch.item.database.JdbcPagingItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="queryProvider" ref="stagingDataQueryProvider" />
<property name="parameterValues">
<map>
<entry key="department" value="#{jobParameters[department]}" />
<entry key="joiningDate" value="#{jobParameters[joiningDate]}" />
<entry key="voucherSuffix" value="#{stepExecutionContext[voucherSuffix]}" />
</map>
</property>
<property name="pageSize" value="1000" />
<property name="rowMapper" ref="myDataRowMapper"/>
</bean>
<bean id="stagingDataQueryProvider"
class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="selectClause" value="SELECT EMP_ID, EMP_NAME, DOB, ADDRESS1, ADDRESS2" />
<property name="fromClause" value="EMP_STG_TBL" />
<property name="whereClause" value="WHERE DEPT_ID=:department AND DOJ=:joiningDate AND VCHR LIKE '%:voucherSuffix'" />
<property name="sortKeys">
<map>
<entry key="EMP_ID" value="ASCENDING"></entry>
</map>
</property>
</bean>
问题是,当作业被执行时,每个步骤都会很好地执行,直到分区步骤。partioner创建执行上下文,该上下文可以从日志语句中确认,但步骤ProcessingStaginStep未执行,作业完成时状态为COMPLETED。此作业和分区步骤配置是否正确
下面是日志语句 2015-02-23 03:03:04信息myTaskScheduler-3 SimpleScheduler:146-执行步骤:[checkConditionStep] 2015-02-23 03:03:04信息myTaskScheduler-3 SimpleScheduler:146-执行步骤:[分段步骤] 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建了ExecutionContext:step0 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建ExecutionContext:step1 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建ExecutionContext:step2 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建ExecutionContext:step3 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建ExecutionContext:step4 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建ExecutionContext:step5 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建ExecutionContext:step6 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建ExecutionContext:step7 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建ExecutionContext:step8 2015-02-23 03:03:04信息myTaskScheduler-3 StagingProcessPartitioner:29-为已分区的步骤创建ExecutionContext:step9 2015-02-23 03:03:05信息myTaskScheduler-3 SimpleJobLauncher:136-作业:[FlowJob:[name=myBatchJob]]已完成,具有以下参数:[]和以下状态:[已完成]
是否正在执行子步骤?如果是,每个步骤的结果是什么?另外,分区器中的ID是如何与数据库中的VCHR字段关联的?是的,所有子步骤都正常完成,并且正在调用分区器,我可以看到从分区方法生成的日志语句。因为我需要根据以0,1,2结尾的voucherNumber对记录进行分区,…9我正在传递threadId,它的范围也在0…9之间,将在查询中使用。子步骤是ProcessingStaginStep,所以我很困惑,因为您的问题表明它没有被执行,但您刚才说它是…调用分区器的步骤stagingPartitionStep已执行,但分区步骤ProcessingStaginStep未执行。这是什么作业存储库中每个子步骤执行(processStagingStep的子步骤执行)的结果?