elasticsearch 解析log4j日志的方法

在下面的grok中，我们希望在logstash发送的数据中添加一些字段，以便kibana进行分析。但是，除了消息字段，从kibana看不到其他字段

grok {
    match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{SKYLOGLEVEL:loglevel} %{THREAD:thread} %{RMOTEIP:remoteipaddress} %{JAVACLASS:logclass} %{CUSTOM_TRACE_EXCEPTION:exception} %{CUSTOM_TRACE_CAUSED_BY:causedby} %{GREEDYDATA:details}"}
    match => {"exception" => "%{CUSTOM_TRACE_EXCEPTION:exception}"}
    match => {"thread" => "%{THREAD:thread}"}   
    match => {"loglevel" => "%{ACMELOGLEVEL:loglevel}"}
    match => {"logclass" => "%{JAVACLASS:logclass}"}
    match => {"remoteip" => "%{RMOTEIP:remoteipaddress}"}
    break_on_match => false 
}

任何确定问题的指针都会有所帮助

谢谢

桑

请在下面的“我的日志文件”中找到一个例外：

2013-04-05 00:00:02101错误[scheduler_Worker-6]（DataProcessor.java:412）RemoteException>
轴断层
故障代码：{http://schemas.xmlsoap.org/soap/envelope/}服务器
故障子代码：
faultString:0005:没有与指定条件匹配的数据
故障因素：
故障节点：
故障详情：
{http://www.bea.com/wli/sb/context}故障：0005没有数据与指定的条件匹配GetNumber_responseCreate Number Responseresponse管道
0005:没有与指定标准匹配的数据1
位于org.apache.axis.message.SOAPFaultBuilder.createFault（SOAPFaultBuilder.java:222）
位于org.apache.axis.message.SOAPFaultBuilder.endElement（SOAPFaultBuilder.java:129）
位于org.apache.axis.encoding.DeserializationContext.endElement（DeserializationContext.java:1087）
位于com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement（未知源代码）
位于com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.ScannedElement（未知来源）
位于com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next（未知源）
2013-04-05 00:07:36535信息[TP-Processor8]10.136.59.190（WTSDK.java:504）WTSDK-命令：V.1
ACDG.WA/ACMEXS/E../PQF7436
VQZ。
VMF///33080
WM DAH 11417.FAX/BG/RTI/CAM/OZI
2013-04-05 00:07:36557信息[TP-Processor8]10.136.59.190（WTSDK.java:505）WTSDK-PID:PQF7436
2013-04-05 00:07:40120信息[TP-Processor8]10.136.59.190（WTSDK.java:517）WTSDK：使用PID PQF7436解析前的响应时间==3560毫秒
2013-04-05 00:07:40126信息[TP-Processor8]10.136.59.190（WTSDK.java:547）WTSDK:解析后的响应字符串：WM DAH PERQF11417

从logstash生成的输出中摘录

{@timestamp:“2016-03-07T23:59:47.306Z”，“消息”：“2013-04-05 00:00:02101错误[scheduler_Worker-6]（DataProcessor.java:412）RemoteException>\nAxisFault\n故障代码：{http://schemas.xmlsoap.org/soap/envelope/}服务器\n faultSubcode:\n faultString:0005:没有与指定条件匹配的数据\n faultActor:\n faultNode:\n faultDetail:\n\t{http://www.bea.com/wli/sb/context}故障：0005没有数据与指定的标准相匹配。指定的标准是：编号GetNumber\u响应创建航班号响应行程响应管道\n0005:没有数据与指定的标准相匹配1\n\t org.apache.axis.message.SOAPFaultBuilder.createFault（SOAPFaultBuilder.java:222）\n\t org.apache.axis.message.SOAPFaultBuilder.endElement（SOAPFaultBuilder.java:129）\n\tat org.apache.axis.encoding.DeserializationContext.endElement（DeserializationContext.java:1087）\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement（未知源）\n\tat com.sun.org.apache.xerces.internal.impl.xmlDocumentFragmentScanneImpl.ScannerElement（未知源）\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next（未知源）\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next（未知源）\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.ScandDocument（未知源）\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse（未知源）\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse（未知源）\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse（未知源）\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse（未知源）\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParser.parse（未知源）\n\tat org.apache.axis.encoding.DeserializationContext.parse（DeserializationContext.java:227）\n\tat org.apache.axis.SOAPPart.getassoapendevelope（SOAPPart.java:696）\n\tat org.apache.axis.Message.getSOAPEnvelope（Message.java:435）\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke（MustUnderstandChecker.java:62）\n\tat org.apache.axis.client.AxisClient.invoke（AxisClient.java:206）\n\tat org.apache.axis.client.Call.invoke（Call.java:2784）\n\tat org.apache.axis.client.Call.invoke（Call.java:2443）\n\tat org.apache.axis.client.Call.invoke（Call.java:2366）\n\tat org.apache.axis.client.Call.Call.Call.invoke（Call.java:1812）\n\tat com.acme.RequestBindingStub.GetNumber（RequestBindingStub.java:1563）\n\tat com.acme.DataProcessor.callOGSTime（DataProcessor.java:398）\n\tat com.acme.DataProcessor.processData（DataProcessor.java:290）\n\tat sun.reflect.gentedMethodAccessor601.invoke（未知源）\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke（未知源）\n\tat java.lang.reflect.Method.invoke（未知源）\n\tat org.springframework.util.MethodInvoker.invoke（MethodInvoker.java:273）\n\tat org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal（MethodInvokingJobDetailFactoryBean.java:264）\n\tat org.springframework.scheduling.quartz.QuartzJobBean.execute（QuartzJobBean.java:86）\n\tat org.quartz.core.JobRunShell.run（JobRunShell.java:203）\n\tat org.quartz.quartz.QuartzJobBean.execute（quartz.QuartzJobBean.java:86）\n\tat org.quartz.quartz.siml.SimpleThreadPool.workerthreadpool$WorkerThread.run（SimpleThreadPool.java:520）”，“@version:“1”，“tags:”：[“多“LVRJ8YRJX1”}，“计数”：1，“字段”：null，“输入类型”：“日志”，“偏移量”：3744，“源”：“C:\\logs\\bagassist\u x-Copy.log”，“类型”：“日志”
2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6          ]                 (DataProcessor.java:412 ) RemoteException > 
AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server
 faultSubcode: 
 faultString: 0005: No Data matched the criteria Specified
 faultActor: 
 faultNode: 
 faultDetail: 
    {http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>getNumber</con:node><con:pipeline>getNumber_response</con:pipeline><con:stage>Create Number Response</con:stage><con:path>response-pipeline</con:path></con:location>

0005: No Data matched the criteria Specified1
    at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)
    at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)
    at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
2013-04-05 00:07:36,535 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:504 ) WTSDK- Command: V.1
ACDG.WA/ACMEXS/E…/PQF7436
VQZ.
VMF////33080
WM DAH 11417.FAX/BG/RTI/CAM/OZI
2013-04-05 00:07:36,557 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:505 ) WTSDK- PID: PQF7436
2013-04-05 00:07:40,120 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:517 ) WTSDK: Response Time before parsing using PID PQF7436 == 3560 ms
2013-04-05 00:07:40,126 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:547 ) WTSDK: Response string after parsing: WM DAH PERQF11417  

{"@timestamp":"2016-03-07T23:59:47.306Z","message":"2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6          ]                 (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetNumber</con:node><con:pipeline>GetNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified1\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.RequestBindingStub.GetNumber(RequestBindingStub.java:1563)\n\tat com.acme.DataProcessor.callOGSTime(DataProcessor.java:398)\n\tat com.acme.DataProcessor.processData(DataProcessor.java:290)\n\tat sun.reflect.GeneratedMethodAccessor601.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)\n\tat org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)\n\tat org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:203)\n\tat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)","@version":"1","tags":["multiline","beats_input_codec_multiline_applied"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":3744,"source":"C:\\logs\\bagassist_x - Copy.log","type":"log","host":"LVRJ8YRJX1"}
{"@timestamp":"2016-03-07T23:59:47.306Z","message":"2013-04-05 00:00:02,319 ERROR [scheduler_Worker-6          ]                 (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetNumber</con:node><con:pipeline>GetNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.RequestBindingStub.GetNumber(RequestBindingStub.java:1563)\n\tat com.acme.DataProcessor.callOGSTime(DataProcessor.java:398)\n\tat com.acme.DataProcessor.processData(DataProcessor.java:290)\n\tat sun.reflect.GeneratedMethodAccessor601.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)\n\tat org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)\n\tat org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:203)\n\tat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)","@version":"1","tags":["multiline","beats_input_codec_multiline_applied"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":7569,"source":"C:\\logs\\bagassist_x - Copy.log","type":"log","host":"LVRJ8YRJX1"}

<pattern>%d %-5level [%-28thread] [%-15X{remoteIpAddress}] (%35logger{0}:%-3L\) %message%n</pattern>                                                            

grok {
  match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} % 
   {SKYLOGLEVEL:loglevel} %{THREAD:thread} %{RMOTEIP:remoteipaddress} %
   {JAVACLASS:logclass} %{CUSTOM_TRACE_EXCEPTION:exception} %
   {CUSTOM_TRACE_CAUSED_BY:causedby} %{GREEDYDATA:details}"
}

- Input: 09-05-18 10:40:57,384 43 11296 [Timer-3] INFO  abc.com.task.CheckPendingTask  -  ---- process START!!! ---- 
- Pattern: %{TIMESTAMP_ISO8601:timestamp} %{NUMBER:line} %{NUMBER:relativeTime} %{NOTSPACE:thread} %{LOGLEVEL:loglevel}  %{JAVACLASS:class}  - %{GREEDYDATA:message}
- Output:
{
  "timestamp": [
    [
      "09-05-18 10:40:57,384"
    ]
  ],
  "YEAR": [
    [
      "09"
    ]
  ],
  "MONTHNUM": [
    [
      "05"
    ]
  ],
  "MONTHDAY": [
    [
      "18"
    ]
  ],
  "HOUR": [
    [
      "10",
      null
    ]
  ],
  "MINUTE": [
    [
      "40",
      null
    ]
  ],
  "SECOND": [
    [
      "57,384"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "line": [
    [
      "43"
    ]
  ],
  "BASE10NUM": [
    [
      "43",
      "11296"
    ]
  ],
  "relativeTime": [
    [
      "11296"
    ]
  ],
  "thread": [
    [
      "[Timer-3]"
    ]
  ],
  "loglevel": [
    [
      "INFO"
    ]
  ],
  "class": [
    [
      "abc.com.task.CheckPendingTask"
    ]
  ],
  "message": [
    [
      " ---- process START!!! ---- "
    ]
  ]
}