Spring integration Spring集成-批处理在15分钟内完成18000个作业

我在下面有一个场景，目前正在利用Spring集成作为技术来实现

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

• 我有大约18000个员工Id数据
• 对于每个员工，需要启动一个流程，执行一个HTTP调用以从邮件日历服务器检索员工配置文件信息，然后执行一个HTTP调用以检索其他信息，然后可能需要在单个任务中再发送3-5个HTTP调用
• 我需要在15分钟内为超过50000名员工完成此过程
• 我需要这整个批处理过程每15分钟运行一次又一次
• 假设每个作业需要5秒钟完成。。我还需要30分钟才能完成

=================

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

初始思维

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

我可以使用spring集成实现以下功能： -为每个员工创建一个职位-18000个职位。作业请求可能只包含一个员工ID，因此请求非常轻。 -一次将所有作业添加到int:queue，以便它触发输入通道-CalendarSynRequestChannel -让一名民意调查人员——100名兼职工人在15分钟内完成工作

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

问题：

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

• 这是进行这种批处理的好方法吗？我关心的是队列的大小，一次可以支持18000个工作
• 我是否应该使用基于文件的方法将所有员工id存储在多个文件中，然后由轮询器获取？然而，这也会使设计复杂化，因为工作人员可能同时存在读取/写入/删除文件的问题

当前解决方案：

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

---

任何遇到类似问题的人都可能会对如何使用Spring Integration解决问题有一些见解

为什么不执行以下Spring批处理作业：

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

读取员工数据的读取器

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

进行HTTP调用的处理器

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

将结果写入日志文件的编写器（例如）

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

然后利用TaskScheduler（spring批处理框架）每15分钟安排一次执行，甚至可以在固定延迟的情况下安排更好的执行

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>

---

如果您想更并行地执行，请使用org.springframework.batch.integration.async.AsyncItemProcessor（和writer）

您好，这两种方法实际上都可以使用spring集成或spring批处理。我想要实现的是。。。如果我把18000个项目放入队列或其他什么东西，我会有问题吗。。。即使是在SpringBatch中，我也需要有一个机制将18000分为更小的块，这样工人们就可以一个接一个或一块地捡起来。尝试看看是否将18000个项目放入队列中，以及它现在是否会破坏任何东西。每个请求都非常小，因为它只包含员工ID

<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
    <int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>

<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>