elasticsearch 将基于内容的过滤记录到多个索引中

我目前正在从一个S3存储桶中提取JSON日志文件，其中包含定义为RawLog的不同类型的日志，以及另一个值MessageSourceType（还有更多我不关心的元数据字段）。文件上的每一行都是一个单独的日志，以防出现差异

我现在把这些都放在一个索引中，正如我在下面的配置中所看到的，然而，我理想的情况是将它们分割成单独的索引。例如，如果MessageSourceType=Syslog-Linux主机，那么我需要logstash将RawLog提取为Syslog并将其放入名为logs Syslog的索引中，而如果MessageSourceType=MS Windows事件日志XML，我希望它将RawLog提取为XML并将其放入名为logs-MS_Event_logs的索引中

filter {
   mutate {
     replace => [ "message", "%{message}" ]
   }
   json {
        source => "message"
        remove_field => "message"
   }
}

output {
   elasticsearch {
     hosts => ["http://xx.xx.xx.xx:xxxx","http://xx.xx.xx.xx:xxxx"]
     index => "logs-received"
 }

另外，对于一点上下文，这里是其中一个日志的示例：

{"MsgClassTypeId":"3000","Direction":"0","ImpactedZoneEnum":"0","message":"<30>Feb 13 23:45:24 xx.xx.xx.xx Account=\"\" Action=\"\" Aggregate=\"False\" Amount=\"\" Archive=\"True\" BytesIn=\"\" BytesOut=\"\" CollectionSequence=\"825328\" Command=\"\" CommonEventId=\"3\" CommonEventName=\"General Operations\" CVE=\"\" DateInserted=\"2/13/2021 11:45:24 PM\" DInterface=\"\" DIP=\"\" Direction=\"0\" DirectionName=\"Unknown\" DMAC=\"\" DName=\"\" DNameParsed=\"\" DNameResolved=\"\" DNATIP=\"\" DNATPort=\"-1\" Domain=\"\" DomainOrigin=\"\" DPort=\"-1\" DropLog=\"False\" DropRaw=\"False\" Duration=\"\" EntityId=\"" EventClassification=\"-1\" EventCommonEventID=\"-1\" FalseAlarmRating=\"0\" Forward=\"False\" ForwardToLogMart=\"False\" GLPRAssignedRBP=\"-1\" Group=\"\" HasBeenInserted_EMDB=\"False\" HasBeenQueued_Archiving=\"True\" HasBeenQueued_EventProcessor=\"False\" HasBeenQueued_LogProcessor=\"True\" Hash=\"\" HostID=\"44\" IgnoreGlobalRBPCriteria=\"False\" ImpactedEntityId=\"0\" ImpactedEntityName=\"\" ImpactedHostId=\"-1\" ImpactedHostName=\"\" ImpactedLocationKey=\"\" ImpactedLocationName=\"\" ImpactedNetworkId=\"-1\" ImpactedNetworkName=\"\" ImpactedZoneEnum=\"0\" ImpactedZoneName=\"\" IsDNameParsedValue=\"True\" IsRemote=\"True\" IsSNameParsedValue=\"True\" ItemsIn=\"\" ItemsOut=\"\" LDSVERSION=\"1.1\" Login=\"\" LogMartMode=\"13627389\" LogSourceId=\"158\" LogSourceName=\"ip-xx-xx-xx-xx.eu-west-2.computer.internal Linux Syslog\" MediatorMsgID=\"0\" MediatorSessionID=\"1640\" MsgClassId=\"3999\" MsgClassName=\"Other Operations\" MsgClassTypeId=\"3000\" MsgClassTypeName=\"Operations\" MsgCount=\"1\" MsgDate=\"2021-02-13T23:45:24.0000000+00:00\" MsgDateOrigin=\"0\" MsgSourceHostID=\"44\" MsgSourceTypeId=\"88\" MsgSourceTypeName=\"Syslog - Linux Host\" NormalMsgDate=\"2021-02-13T23:45:24.0540000Z\" Object=\"\" ObjectName=\"\" ObjectType=\"\" OriginEntityId=\"0\" OriginEntityName=\"\" OriginHostId=\"-1\" OriginHostName=\"\" OriginLocationKey=\"\" OriginLocationName=\"\" OriginNetworkId=\"-1\" OriginNetworkName=\"\" OriginZoneEnum=\"0\" OriginZoneName=\"\" ParentProcessId=\"\" ParentProcessName=\"\" ParentProcessPath=\"\" PID=\"-1\" Policy=\"\" Priority=\"4\" Process=\"\" ProtocolId=\"-1\" ProtocolName=\"\" Quantity=\"\" Rate=\"\" Reason=\"\" Recipient=\"\" RecipientIdentity=\"\" RecipientIdentityCompany=\"\" RecipientIdentityDepartment=\"\" RecipientIdentityDomain=\"\" RecipientIdentityID=\"-1\" RecipientIdentityTitle=\"\" ResolvedImpactedName=\"\" ResolvedOriginName=\"\" ResponseCode=\"\" Result=\"\" RiskRating=\"0\" RootEntityId=\"9\" Sender=\"\" SenderIdentity=\"\" SenderIdentityCompany=\"\" SenderIdentityDepartment=\"\" SenderIdentityDomain=\"\" SenderIdentityID=\"-1\" SenderIdentityTitle=\"\" SerialNumber=\"\" ServiceId=\"-1\" ServiceName=\"\" Session=\"\" SessionType=\"\" Severity=\"\" SInterface=\"\" SIP=\"\" Size=\"\" SMAC=\"\" SName=\"\" SNameParsed=\"\" SNameResolved=\"\" SNATIP=\"\" SNATPort=\"-1\" SPort=\"-1\" Status=\"\" Subject=\"\" SystemMonitorID=\"9\" ThreatId=\"\" ThreatName=\"\" UniqueID=\"7d4c4ed3-a2fc-44bc-a7ec-0b8b68e7f456\" URL=\"\" UserAgent=\"\" UserImpactedIdentity=\"\" UserImpactedIdentityCompany=\"\" UserImpactedIdentityDomain=\"\" UserImpactedIdentityID=\"-1\" UserImpactedIdentityTitle=\"\" UserOriginIdentity=\"\" UserOriginIdentityCompany=\"\" UserOriginIdentityDepartment=\"\" UserOriginIdentityDomain=\"\" UserOriginIdentityID=\"-1\" UserOriginIdentityTitle=\"\" VendorInfo=\"\" VendorMsgID=\"\" Version=\"\" RawLog=\"02 13 2021 23:45:24 xx.xx.xx.xx <SYSD:INFO> Feb 13 23:45:24 euw2-ec2--001 metricbeat[3031]: 2021-02-13T23:45:24.264Z#011ERROR#011[logstash.node_stats]#011node_stats/node_stats.go:73#011error making http request: Get \\\"https://xx.xx.xx.xx:9600/\\\": dial tcp xx.xx.xx.xx:9600: connect: connection refused\"","CollectionSequence":"825328","NormalMsgDate":"2021-02-13T23:45:24.0540000Z"}

{MsgClassTypeId:“3000”，“方向”：“0”，“受影响区域”：“0”，“消息”：“2月13日23:45:24 xx.xx.xx.xx帐户=\”\“操作=\”\“聚合=\”假\“金额=\”\“存档=”真\“字节数=”字节数=“\”集合顺序=\“825328\”命令=“\”CommonEventId=”3\“CommonEventName=”一般操作\“CVE=”日期插入=”2021年2月13日11:45:24下午\“DInterface=\”\“DIP=”\“0\”DirectionName=“Unknown\”DMAC=“\”Dnamesparsed=“\”Dnamesolved=“\”Dnamesolved=“\”DNATIP=“\”DNATPort=“-1\”DomainOrigin=“\”DPort=“-1\”Dropog=“False\”DropRaw=“False\”持续时间=“EntityClassification=”EventId=“Event1=”公共事件ID=“-Event1=”\“FalseArmRating=\'0\'Forward=\'False\'ForwardToLogMart=\'False\'GLPRAssignedRBP=\'1\'Group=\'HasBeenInserted\'U EMDB=\'False\'HasBeenQueued\'U存档='True\'Hasbeenqueed\'U事件处理器='False\'Hasbeenqueed\'LogProcessor='True\'Hash='HostID='44\'IgnoreGlobalRBPCCriteria='False\'ImpactedEntityId='0\'ImpactedHostId=\“-1\”ImpactedHostName=\“\”ImpactedLocationKey=\“\”ImpactedLocationName=\“\”ImpactedNetworkId=\“-1\”ImpactedNetworkName=\“\”ImpactedZoneNum=\“0\”ImpactedZoneName=\“\”IsDNameParsedValue=\“True\”IsSNameParsedValue=“True\”ItemsIn=\“\”ItemsOut=\”LDSVERSION=“1.1\”Log=“136279”登录模式LogSourceId=\'158\'LogSourceName=\'ip-xx-xx-xx-xx.eu-west-2.computer.internal Linux Syslog\'MediatorMsgID=\'0\'MediatorSessionID=\'1640\'MsgClassId=\'3999\'MsgClassName=\'Other Operations\'3000\'MsgClassTypeName=\'Operations\'MsgCount='1\'MsgDate='2021-02-13T23:45:24.0000000+00:00\'MsgDate='0\'MsgSourceHostID=\'44\'MsgSourceTypeId=\'88\'MsgSourceTypeName=\'Syslog-Linux主机\'NormalMsgDate=\'2021-02-13T23:45:24.0540000Z\'Object=\'ObjectName=\'ObjectType=\'OriginEntityId=\'0\'OriginEntityName=\'OriginHostId=\'1\'OriginLocationKey=\'OriginationName='OriginNetworkId='OriginNetworkName=\“\”OriginZoneNum=\“0\”OriginZoneName=\“\”ParentProcessId=\“\”ParentProcessName=\“\”ParentProcessPath=\“\”PID=\“-1\”Policy=\“\”Priority=\”4\”Process=\“\”ProtocolId=\”-1\”ProtocolName=“\”数量=“\”速率=“\”原因=“\”收件人=“\”收件人标识=“\”收件人标识公司=“\”RecipientIdentityDepartment=\“\”RecipientIdentityDomain=\“\”RecipientIdentityID=\“-1\”RecipientIdentityTitle=\“\”ResolvedIdPactedName=\“\”ResponseCode=\“\”Result=\“\”RiskRating=\“0\”RootEntityId=\“9\”发件人=\“\”发件人=“\”发件人身份=“\”发件人身份公司=\“\”发件人身份部门=“\”发件人身份=““\”SenderIdentityID=\“-1\”SenderIdentityTitle=\“\”SerialNumber=\“\”ServiceId=\“-1\”ServiceName=\“\”SessionType=\“\”SessionType=\“\”Severity=\“\”SInterface=\“\”SIP=“\”Size=“\”SMAC=“\”SName=“\”Snamesolved=“\”SNATIP=“\”SNATPort=“-1\”SPort=“-1\”Status=\”受试者监控系统9\”ThreatName=\“\”UniqueID=\“7d4c4ed3-a2fc-44bc-a7ec-0b8b68e7f456\”URL=\“\”UserAgent=\“\”UserImpactedIdentity=\“\”UserImpactedIdentityCompany=\“\”UserImpactedIdentityDomain=\”UserImpactedIdentityID=\”-1\“UserImpactedIdentityTitle=“\”UserOriginIdentity=\”UserOriginIdentityCompany=\“\”UserOriginIdentityPartment=\”UserOriginIdentityDomain=\“\”UserOriginIdentityID=\“-1\”UserOriginIdentityTitle=\“\”VendorMsgID=\”Version=\“\”RawLog=\”0213 2021 23:45:24 xx.xx.xx.xx.xx二月13日23:45:24 euw2-ec2--001节拍[3031]：2021-02-13T23:45:24.264Z\35011错误[logstash.node\u]#011node_stats/node_stats.go:73#011发出http请求时出错：Get\\\”https://xx.xx.xx.xx:9600/\\\“：拨号tcp xx.xx.xx.xx:9600:connect:connection-sequence\”，“CollectionSequence:“825328”，“NormalMsgDate:“2021-02-13T23:45:24.0540000Z”}

我有点不确定实现这一目标的最佳方法，我想你们可能会有一些建议。我已经调查了grok，认为这可能实现我的目标，但我不确定从何处开始。

您可以在

过滤器

部分中执行此操作，并根据正在解析的日志类型定义目标索引

filter {
  ... other filters ...

  if [MsgSourceTypeName] == "Syslog - Linux Host" {
    mutate {
      add_field => {
        "[@metadata][target_index]" => "logs-syslog"
      }
    }
  }
  else if [MsgSourceTypeName] == "MS Windows Event Logging XML" {
    mutate {
      add_field => {
        "[@metadata][target_index]" => "logs-ms_event_log"
      }
    }
  }
}
output {
   elasticsearch {
     hosts => ["http://xx.xx.xx.xx:xxxx","http://xx.xx.xx.xx:xxxx"]
     index => "%{[@metadata][target_index]}"
   }
}

---

你的输入在哪里？这两种日志的输入相同？你可以做你想做的事情，但是你需要在管道中基于一些文件名使用条件。你能分享一个其他类型日志的例子吗？只需添加

logs-MS\u Event\u logs

应该是

logs-MS\u Event\u logs

，因为索引名需要小写。Tr如果@leandrojmp是我明目张胆地复制/粘贴的，那么我会在if语句中使用我的配置来提取所需的部分作为syslog或xml吗？另外，我假设我会在将消息提取为json后使用条件，否则我将无法匹配条件？没错，您首先需要提取消息，然后检查字段，然后根据条件对消息应用不同的解析。谢谢，运气不好，但我认为这是由于我的提取/pa中的错误造成的