elasticsearch 无法使用logstash（7.6.2）和xml筛选器插件设置解析和设置时间戳

环境详情： 麋鹿堆7.6.2 视窗10

在通过Logstash进行索引时，我无法从日志中替换/设置弹性搜索时间戳。它只是添加一个新字段，而不是替换原始字段。它只添加了“_dateparsefailure”标记，没有任何其他信息

我怀疑日期过滤器不起作用

我的示例日志数据：

<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819710045" level="WARN" thread="27"><log4j:message>registrarCheck.bookingWizardController.TryUpdatebookingCareOptions(): bookingCareOptionId: CenterBasedCare, bookingId: 5745493, bookingregistrarsCount: 5, IsEditbooking: False, IsEditbookingStep2Modified: False, IsMemberShip: False</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="283" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760731" level="ERROR" thread="15"><log4j:message>ERROR from EasyDraft API for funding accountid-&gt;0-&gt;Name: firstname lastname-&gt;Card number is invalid</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="139" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760856" level="ERROR" thread="15"><log4j:message>Error in controller: effective username: user1, identity username: user1, machine name: webserver1 
Client Name: [zzz Test ESomeApplication], Contract Id: [7ee17d62-d292-e511-b173-005056991898]
, Person Id: [143658262]
, Client ID: [b33442b3-d192-e511-b173-005056991898], Contract Relationship ID: [4529625]
, Person Fullname: [firstname lastname].
, Full Name: [firstname lastname], CRM ID: [a64c97b1-8a80-e811-b738-005056991899]</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException: Exception of type 'SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException' was thrown.
   at SomeOrganization.SomeApplication.BusinessLogic.PaymentAccount.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\SomeApplication\SomeOrganization.SomeApplication.BusinessLogic\PaymentAccount.cs:line 415
   at Csla.BusinessBase`1.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 163
   at Csla.BusinessBase`1.Csla.Core.ISavable.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 350
   at SomeOrganization.Shared.Web.ApplicationBlocks.Controllers.CustomCslaMvcController.SaveObject[T](T item, Action`1 updateModel, Boolean forceUpdate) in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.Web.ApplicationBlocks\Controllers\CustomCslaMvcController.cs:line 171</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530824089499" level="ERROR" thread="41"><log4j:message>Error Occured while Save Login in Class Login &amp; Method : Save For Username : tegh14</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-2-131752976869399121" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>System.Security.Authentication.AuthenticationException: We can�t find that username and/or password.  If you are trying to register for the first time using your employer�s credentials, select the Create Your Profile link below. If you are having trouble accessing the site, feel free to call us at none-one-CARES in the United States or Canada, 0800 000 000 in the United Kingdom, or 0800 000 000 in Ireland.
   at SomeOrganization.Shared.BusinessLogic.Security.Login.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.BusinessLogic\Security\Login.cs:line 547</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" timestamp="1587880949425" level="WARN" thread="47"><log4j:message>User mphilpunla-&gt;LoginWithSAML-&gt;lobuniqueId 19153694</log4j:message><log4j:properties><log4j:data name="log4jmachinename" value="webserver2" /><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-132323544167926323" /><log4j:data name="log4net:UserName" value="SomeOrganisation\!svc-lob-apps" /><log4j:data name="log4net:Identity" value="" /><log4j:data name="log4net:HostName" value="webserver2" /></log4j:properties><log4j:locationInfo class="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="E:\TFS2018agent\agent\_work\96\s\Shared\SomeOrganisation.Shared.ApplicationBlocks\Logging\Logging.cs" line="294" /></log4j:event>

---

我在这里遗漏了什么？

UNIX模式要求您的时间是以秒为单位的UNIX时间，这是一个10位数字，但您的

时间戳

字段是以毫秒为单位的UNIX时间，这是一个13位数字

您应该改用

UNIX\u MS

模式

date {
    match => [ "timestamp","UNIX_MS" ]
    target => "@timestamp"
    remove_field => ["timestamp"]
}

编辑：

如果

timestamp

字段是一个加强项，则上述过滤器可以正常工作，但是

xml

过滤器似乎将数据存储在一个数组中，即使您只有一个信息，因此本例中的

timestamp

字段位于索引0中，然后过滤器中的字段需要是

[timestamp][0]

date {
    match => [ "[timestamp][0]","UNIX_MS" ]
    target => "@timestamp"
    remove_field => ["timestamp"]
}

使用以下消息进行模拟

{ "msg": "sample message", "timestamp": ["1530819710045"] }

输出为：

{
    "@timestamp" => 2018-07-05T19:41:50.045Z,
          "host" => "elk",
      "@version" => "1",
           "msg" => "sample message"
}

---

我也尝试过使用“UNIX_MS”，但仍然不起作用。结果仍然是一样的。此外，时间戳字段不会像日期过滤器中提到的那样被删除。我也尝试过使用不同的目标字段，但转换仍然不成功。我想我明白了问题所在，xml筛选器解析数据并将其存储在一个数组中，即使它只是一个信息，因此您的

时间戳

字段是一个数组，您的日期在索引0中，我将更新我的答案，以便您可以重试。

{ "msg": "sample message", "timestamp": ["1530819710045"] }

{
    "@timestamp" => 2018-07-05T19:41:50.045Z,
          "host" => "elk",
      "@version" => "1",
           "msg" => "sample message"
}