<img src="//i.stack.imgur.com/RUiNP.png" height="16" width="18" alt="" class="sponsor tag img">elasticsearch Logstash到kibana多行不适用于自定义消息_<img Src="//i.stack.imgur.com/RUiNP.png" Height="16" Width="18" Alt="" Class="sponsor Tag Img">elasticsearch_Logstash_Kibana 4_Logstash Grok

elasticsearch Logstash到kibana多行不适用于自定义消息

logstash

elasticsearch Logstash到kibana多行不适用于自定义消息,elasticsearch,logstash,kibana-4,logstash-grok,elasticsearch,Logstash,Kibana 4,Logstash Grok,我正试图通过logstash将我的日志文件推送到elasticsearch，并在kibana上显示它。它适用于单行日志记录。但是，当涉及到多行过滤器时，它会失败以下是我的多行日志输入示例： 2016-06-02T04:02:29,720 INFO Thread-25-match-entity-bolt a52488cc-316b-402e-af58-3b8a663cd76a STDIO invoke Error processing message:{ "eid": "f9f16541-

我正试图通过logstash将我的日志文件推送到elasticsearch，并在kibana上显示它。它适用于单行日志记录。但是，当涉及到多行过滤器时，它会失败
以下是我的多行日志输入示例：

2016-06-02T04:02:29,720 INFO  Thread-25-match-entity-bolt a52488cc-316b-402e-af58-3b8a663cd76a STDIO invoke Error processing message:{
  "eid": "f9f16541-4fab-4131-a82e-e3ddf6fcd949",
  "entityInfo": {
    "entityType": "style",
    "defaultLocale": "en-US"
  },
  "systemInfo": {
    "tenantId": "t1"
  },
  "attributesInfo": {
    "externalId": 1514,
    "attributesRead": {
      "IsEntityVariantsValid": false,
      "IsEntityExtensionsValid": false
    },
    "attributesUpdated": {
      "DateAttribute": "2016-06-01T00:00:00.0000000",
      "IsEntitySelfValid": true,
      "IsEntityMetaDataValid": true,
      "IsEntityCommonAttributesValid": true,
      "IsEntityCategoryAttributesValid": true,
      "IsEntityRelationshipsValid": true
    }
  },
  "jsAttributesInfo": {
    "jsRelationship": {
      "entityId": "CottonMaterial001",
      "parentEntityId": "Apparel",
      "category": "Apparel",
      "categoryName": "Apparel",
      "categoryPath": "Apparel",
      "categoryNamePath": "Apparel",
      "variant": "1514",
      "variantPath": "1035/1514",
      "container": "Demo Master",
      "containerName": "Demo Master",
      "containerPath": "DemoOrg/Demo Master/Apparel",
      "organization": "DemoOrg",
      "segment": "A"
    },
    "jsChangeContext": {
      "entityAction": "update",
      "user": "cfadmin",
      "changeAgent": "EntityEditor.aspx",
      "changeAgentType": "PIM",
      "changeInterface": "Entity",
      "sourceTimestamp": "2016-06-01T19:48:19.4162475+05:30",
      "ingestTimestamp": "2016-06-01T19:48:19.4162475+05:30"
    }
  }
}

到目前为止，我已经尝试了这些日志存储配置：

input {
  file {
    path => "path_to_logs/logs.log"
    start_position => "beginning"
  }
}
filter{
    multiline {
      negate => "true"
      pattern => "^%{TIMESTAMP_ISO8601} "
      what => "previous"
    }
    grok{
      match => { "message" => "^%{TIMESTAMP_ISO8601:JigsawTimestamp}%{SPACE}%{LOGLEVEL:JigsawLoglevel}%{SPACE}%{HOSTNAME:ThreadName}%{SPACE}%{UUID:GUID}%{SPACE}%{JAVACLASS:JigsawClassName}%{SPACE}%{WORD:JigsawMethodName}%{SPACE}%{GREEDYDATA:JigsawLogMessage}" } 
    }
}
output {

  if "_grokparsefailure" not in [tags] {
    elasticsearch { 
      hosts => ["localhost:9200"] 
    }
  }
}

第二条：

input {
  file {
    path => "path_to_logs/logs.log"
    start_position => "beginning"
    codec => multiline {
      negate => "true"
      pattern => "^%{TIMESTAMP_ISO8601} "
      what => "previous"
    }
  }
}
filter{
    grok{
      match => { "message" => "^%{TIMESTAMP_ISO8601:JigsawTimestamp}%{SPACE}%{LOGLEVEL:JigsawLoglevel}%{SPACE}%{HOSTNAME:ThreadName}%{SPACE}%{UUID:GUID}%{SPACE}%{JAVACLASS:JigsawClassName}%{SPACE}%{WORD:JigsawMethodName}%{SPACE}%{GREEDYDATA:JigsawLogMessage}" } 
    }
}
output {

  if "_grokparsefailure" not in [tags] {
    elasticsearch { 
      hosts => ["localhost:9200"] 
    }
  }
}

我也尝试过这种模式：

pattern => "^\s"

然而，这些都没有帮助。他们都被贴上了grokparsefailure标签。我希望JSON行是单个消息的一部分。请指出此筛选器中的错误。

在grok筛选器中，有几个错误使您无法查看任何日志

在INFO之后的示例数据中，有两个空格

对于字段JigsawClassName，您使用JAVACLASS作为输入，这对于日志来说是错误的

为什么JAVACLASS是错误的

其实施情况如下：

JAVACLASS（？[a-zA-Z0-9-]+）+[a-zA-Z0-9$]+

根据上面的JAVACLASS，文本中至少需要一个句点（.）符号。然而，在您的日志中，它只是STDIO

将您的grok match替换为以下内容：-

match => { "message" => "^%{TIMESTAMP_ISO8601:JigsawTimestamp}%{SPACE}%{LOGLEVEL:JigsawLoglevel}%{SPACE}%{SPACE}%{HOSTNAME:ThreadName}%{SPACE}%{UUID:GUID}%{SPACE}%{WORD:JigsawClassName}%{SPACE}%{WORD:JigsawMethodName}%{SPACE}%{GREEDYDATA:JigsawLogMessage}" }

为了便于理解，还可以使用output通过添加stdout插件将其重定向到控制台，如下所示：-

output {
if "_grokparsefailure" not in [tags] {
elasticsearch { 
  hosts => ["localhost:9200"] 
}
stdout { codec => rubydebug }
}

这将使您在使用Logstash处理数据时更容易理解错误。