<img src="//i.stack.imgur.com/RUiNP.png" height="16" width="18" alt="" class="sponsor tag img">elasticsearch 使用logstash将日志文件推送到elastic_<img Src="//i.stack.imgur.com/RUiNP.png" Height="16" Width="18" Alt="" Class="sponsor Tag Img">elasticsearch_Jmeter_Logstash_Logstash Grok_Grok

elasticsearch 使用logstash将日志文件推送到elastic

jmeter logstash

elasticsearch 使用logstash将日志文件推送到elastic,elasticsearch,jmeter,logstash,logstash-grok,grok,elasticsearch,Jmeter,Logstash,Logstash Grok,Grok,我们有一个来自Jmeter的错误日志文件，需要使用logstash将其推送到elastic 这是日志文件，我们要从第5行推送数据 2021-05-14 20:32:49,822 INFO o.a.j.u.JMeterUtils: Setting Locale to en_EN 2021-05-14 20:32:49,842 INFO o.a.j.JMeter: Loading user properties from: user.properties 2021-05-14 20:32:49,8

我们有一个来自Jmeter的错误日志文件，需要使用logstash将其推送到elastic

这是日志文件，我们要从第5行推送数据

 2021-05-14 20:32:49,822 INFO o.a.j.u.JMeterUtils: Setting Locale to en_EN
2021-05-14 20:32:49,842 INFO o.a.j.JMeter: Loading user properties from: user.properties
2021-05-14 20:32:49,843 INFO o.a.j.JMeter: Loading system properties from: system.properties
2021-05-14 20:32:49,844 WARN o.a.j.JMeter: LogLevel: ERROR
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-4: ; response received: {"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-7: ; response received: {"timestamp":"2021-05-14T14:32:51.689+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-20: ; response received: {"timestamp":"2021-05-14T14:32:51.850+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: LEX 2-1: ; response received: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>
2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-5: ; response received: {"timestamp":"2021-05-14T14:32:51.687+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}

当我们尝试推送时，会出现错误Grok parse failure\u grokparsefailure

不确定我在这里遗漏了什么

像这样的东西应该适合你：

^%{GREEDYDATA}ERROR%{GREEDYDATA}response received: %{GREEDYDATA:error_message}

%{TIMESTAMP_ISO8601:timestamp} %{WORD:LogType} %{GREEDYDATA:data1} PostProcessor: %{GREEDYDATA:data2} ; response received: %{GREEDYDATA:responseRecd}

例1：

2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-4: ; response received: {"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}

输出：

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "Http URL/API Test 1-4:"
    ]
  ],
  "responseRecd": [
    [
      "{"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}"
    ]
  ]
}

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "LEX 2-1:"
    ]
  ],
  "responseRecd": [
    [
      "<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>"
    ]
  ]
}

例2：

2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: LEX 2-1: ; response received: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>

2021-05-14 20:32:51824错误o.a.j.e.j.JSR223后处理器：LEX 2-1:；收到的回复：TREXhgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3

输出：

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "Http URL/API Test 1-4:"
    ]
  ],
  "responseRecd": [
    [
      "{"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}"
    ]
  ]
}

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "LEX 2-1:"
    ]
  ],
  "responseRecd": [
    [
      "<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>"
    ]
  ]
}

{
“时间戳”：[
[
"2021-05-14 20:32:51,824"
]
],
“日志类型”：[
[
“错误”
]
],
“数据1”：[
[
“o.a.j.e.j.JSR223”
]
],
“数据2”：[
[
“第2-1条：
]
],
“回应者CD”：[
[
“TREXhgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3”
]
]
}

我已经给出了虚拟名称data1和data2。你可以根据你的需要把它们进一步分开。

请使用进行调试。

类似的内容应该适合您：

%{TIMESTAMP_ISO8601:timestamp} %{WORD:LogType} %{GREEDYDATA:data1} PostProcessor: %{GREEDYDATA:data2} ; response received: %{GREEDYDATA:responseRecd}

例1：

2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: Http URL/API Test 1-4: ; response received: {"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}

输出：

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "Http URL/API Test 1-4:"
    ]
  ],
  "responseRecd": [
    [
      "{"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}"
    ]
  ]
}

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "LEX 2-1:"
    ]
  ],
  "responseRecd": [
    [
      "<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>"
    ]
  ]
}

例2：

2021-05-14 20:32:51,824 ERROR o.a.j.e.J.JSR223 PostProcessor: LEX 2-1: ; response received: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>

2021-05-14 20:32:51824错误o.a.j.e.j.JSR223后处理器：LEX 2-1:；收到的回复：TREXhgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3

输出：

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "Http URL/API Test 1-4:"
    ]
  ],
  "responseRecd": [
    [
      "{"timestamp":"2021-05-14T14:32:51.688+0000","status":404,"error":"Not Found","message":"No message available","path":"/healths"}"
    ]
  ]
}

{
  "timestamp": [
    [
      "2021-05-14 20:32:51,824"
    ]
  ],
  "LogType": [
    [
      "ERROR"
    ]
  ],
  "data1": [
    [
      "o.a.j.e.J.JSR223"
    ]
  ],
  "data2": [
    [
      "LEX 2-1:"
    ]
  ],
  "responseRecd": [
    [
      "<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><ns3:systemContext xmlns:ns3="urn:abc.mon.utils.v1.context"><systemId>TREX</systemId><correlationId>hgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3</correlationId></ns3:systemContext></SOAP-ENV:Header><SOAP-ENV:Body><ns3:retrieveStopPayResponse xmlns:ns3="urn:abc.tailhead.readyforcheck.processStopPay"><retrieveGagaPayResponse recordFound="false"><checkNumber/><stopPayDate/></retrieveGagaPayResponse></ns3:retrieveStopPayResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>"
    ]
  ]
}

{
“时间戳”：[
[
"2021-05-14 20:32:51,824"
]
],
“日志类型”：[
[
“错误”
]
],
“数据1”：[
[
“o.a.j.e.j.JSR223”
]
],
“数据2”：[
[
“第2-1条：
]
],
“回应者CD”：[
[
“TREXhgfhjghj-c62dsfsdfdb-4aff-a871-8f32088943b3”
]
]
}

我已经给出了虚拟名称data1和data2。你可以根据你的需要把它们进一步分开。请使用进行调试