Design patterns 如何构造一个所有数据源共享相同结构的管道体系结构？

​ 我正在尝试构建一个数据流管道，在该管道中，多个N个数据源会将我的数据流到一个节点管理器，然后保存或临时保存到postgresql，类似于聊天，用户发送消息，然后另一方接收数据并保存，但是，我想首先面对一些问题：我应该将每个数据源中的数据流化为dict列表还是单个dict，并且节点管理器it将对所有数据进行分组以执行批量插入

例如：

我的方法

• 任务队列芹菜（2个任务，一个每24小时运行一次，另一个每小时运行一次）
• 节点管理器，用作聊天以流式传输数据
• 刮取器将数据流式传输到节点管理器
• 节点管理器保存数据<我不确定DICT列表或单个JSON对象是否=对DB的许多查询

cvedetails ​ 对于速度cvssv2、cvssv3，已将其移除 ​

​ ​ NVD ​

{
   "cveid":"CVE-2009-20001",
   "summary":"An issue was discovered in MantisBT before 2.24.5. It associates a unique cookie string with each user. This string is not reset upon logout (i.e., the user session is still considered valid and active), allowing an attacker who somehow gained access to a user's cookie to login as them.",
   "vulnerability_type":"613",
   "affected_products":"Mantisbt Mantisbt *",
   "advisory_url":"https://mantisbt.org/bugs/view.php?id=27976",
   "detection_date":"2021-03-07"
}

{
   "cveid":"CVE-1999-0002",
   "summary":"Buffer overflow in NFS mountd gives root access to remote attackers, mostly in Linux systems.",
   "vulnerability_type":"CWE-119",
   "affected_products":"N/A",
   "cvssv2":{
      "version":"2.0",
      "vectorString":"AV:N/AC:L/Au:N/C:C/I:C/A:C",
      "accessVector":"NETWORK",
      "accessComplexity":"LOW",
      "authentication":"NONE",
      "confidentialityImpact":"COMPLETE",
      "integrityImpact":"COMPLETE",
      "availabilityImpact":"COMPLETE",
      "baseScore":"Decimal(""10.0"")"
   },
   "cvssv3":{
      
   },
   "advisory_url":[
      "http://www.securityfocus.com/bid/121",
      "ftp://patches.sgi.com/support/free/security/advisories/19981006-01-I",
      "http://www.ciac.org/ciac/bulletins/j-006.shtml"
   ],
   "detection_date":datetime.datetime(1998,10,12,4,0,"tzinfo=tzutc())"
}