elasticsearch geoip查找失败弹性堆栈日志存储,elasticsearch,logstash,elastic-stack,logstash-grok,geoip,elasticsearch,Logstash,Elastic Stack,Logstash Grok,Geoip" /> elasticsearch geoip查找失败弹性堆栈日志存储,elasticsearch,logstash,elastic-stack,logstash-grok,geoip,elasticsearch,Logstash,Elastic Stack,Logstash Grok,Geoip" />
Fatal编程技术网
  • elasticsearch/
  • elasticsearch geoip查找失败弹性堆栈日志存储

elasticsearch geoip查找失败弹性堆栈日志存储

logstash

elasticsearch geoip查找失败弹性堆栈日志存储,elasticsearch,logstash,elastic-stack,logstash-grok,geoip,elasticsearch,Logstash,Elastic Stack,Logstash Grok,Geoip,使用filebeat将apache日志从Windows系统发送到linux EC2中的我的logstash服务器，然后发送到elastic search和Kibana 弹性搜索和Kibana-5.3 Logstash和filebeat-5.3 filebeat.yml： filebeat.prospectors: - input_type: log # Paths that should be crawled and fetched. Glob based paths. paths:

使用filebeat将apache日志从Windows系统发送到linux EC2中的我的logstash服务器，然后发送到elastic search和Kibana

弹性搜索和Kibana-5.3 Logstash和filebeat-5.3

filebeat.yml：

filebeat.prospectors:

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - C:\Users\Sagar\Desktop\elastic_test4\data\log\*

output.logstash:
  # The Logstash hosts
  hosts: ["10.101.00.11:5044"]
  template.name: "filebeat-poc"
  template.path: "filebeat.template.json"
  template.overwrite: false
Ubuntu Linux EC2实例中的logstash.conf

input {
  beats {
    port => 5044
  }
}
filter {
  grok {
      match => {
        "message" => "%{COMBINEDAPACHELOG}"
      }
  }
  geoip {
      source => "clientip"
      target => "geoip"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
  }
   mutate {
      convert => [ "[geoip][coordinates]", "float"]
  }
 }
output {
  elasticsearch {
  hosts => ["elastic-instance-1.es.amazonaws.com:80"]
  index => "apache-%{+YYYY.MM.dd}"
  document_type => "apache_logs"
 }
  stdout { codec => rubydebug }
}
我的虚拟日志文件

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846
64.242.88.10 - - [07/Mar/2004:16:06:51 -0800] "GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1" 200 4523
64.242.88.10 - - [07/Mar/2004:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Mar/2004:16:11:58 -0800] "GET /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 200 7352
64.242.88.10 - - [07/Mar/2004:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
64.242.88.10 - - [07/Mar/2004:16:23:12 -0800] "GET /twiki/bin/oops/TWiki/AppendixFileSystem?template=oopsmore¶m1=1.12¶m2=1.12 HTTP/1.1" 200 11382
64.242.88.10 - - [07/Mar/2004:16:24:16 -0800] "GET /twiki/bin/view/Main/PeterThoeny HTTP/1.1" 200 4924
64.242.88.10 - - [07/Mar/2004:16:29:16 -0800] "GET /twiki/bin/edit/Main/Header_checks?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12851
64.242.88.10 - - [07/Mar/2004:16:30:29 -0800] "GET /twiki/bin/attach/Main/OfficeLocations HTTP/1.1" 401 12851
64.242.88.10 - - [07/Mar/2004:16:31:48 -0800] "GET /twiki/bin/view/TWiki/WebTopicEditTemplate HTTP/1.1" 200 3732
64.242.88.10 - - [07/Mar/2004:16:32:50 -0800] "GET /twiki/bin/view/Main/WebChanges HTTP/1.1" 200 40520
64.242.88.10 - - [07/Mar/2004:16:33:53 -0800] "GET /twiki/bin/edit/Main/Smtpd_etrn_restrictions?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12851
我能够将这些日志发送到elastic和kibana仪表板。管道已设置且正在工作，但geoip未工作

这是我的kibana搜索输出

{
        "_index": "apache-2017.06.15",
        "_type": "apache_logs",
        "_id": "AVyqJhi6ItD-cRj2_AW6",
        "_score": 1,
        "_source": {
          "@timestamp": "2017-06-15T05:06:48.038Z",
          "offset": 154,
          "@version": "1",
          "input_type": "log",
          "beat": {
            "hostname": "sagar-machine",
            "name": "sagar-machine",
            "version": "5.3.2"
          },
          "host": "by-df164",
          "source": """C:\Users\Sagar\Desktop\elastic_test4\data\log\apache-log.log""",
          "message": """64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846""",
          "type": "log",
          "tags": [
            "beats_input_codec_plain_applied",
            "_grokparsefailure",
            "_geoip_lookup_failure"
          ]
        }
      }
知道我为什么要面对这个问题吗。

您有一个
\u grokparsefailure
，因此
clientip
字段不存在。这会导致
\u geoip\u lookup\u失败
，因为
geoip
筛选器正在寻找不存在的
clientip
字段

您的日志匹配的是
%{commonapacachelog}
模式，而不是您正在使用的模式。因此，您的配置如下所示：

filter {
  grok {
      match => {
        "message" => "%{COMMONAPACHELOG}"
      }
   }
   ...
}
在使用正确的模式后，您应该注意到
clientip
字段存在，在此之后，希望
geoip
过滤器能够工作。：）

您有一个
\u grokparsefailure
，因此
clientip
字段不存在。这会导致
\u geoip\u lookup\u失败
，因为
geoip
筛选器正在寻找不存在的
clientip
字段

您的日志匹配的是
%{commonapacachelog}
模式，而不是您正在使用的模式。因此，您的配置如下所示：

filter {
  grok {
      match => {
        "message" => "%{COMMONAPACHELOG}"
      }
   }
   ...
}
在使用正确的模式后，您应该注意到
clientip
字段存在，在此之后，希望
geoip
过滤器能够工作。：）

我不知道您的日志格式是否适合apache。因为你的日志看起来像这样

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846

149.148.126.144 - - [10/Sep/2017:06:30:44 -0700] "GET /apps/cart.jsp?appID=6944 HTTP/1.0" 200 4981 "http://hernandez.net/app/main/search/homepage.php" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/5322 (KHTML, like Gecko) Chrome/13.0.896.0 Safari/5322"
标准apache日志如下所示

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846

149.148.126.144 - - [10/Sep/2017:06:30:44 -0700] "GET /apps/cart.jsp?appID=6944 HTTP/1.0" 200 4981 "http://hernandez.net/app/main/search/homepage.php" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/5322 (KHTML, like Gecko) Chrome/13.0.896.0 Safari/5322"
我建议您对即将发布的apache日志格式进行标准化。否则，默认的grok配置将不适用于您。然后，您必须为自定义日志编写自己的grok模式。这将解析即将出现的日志行

除此之外，还有很多原因让你犯这样的错误

您没有在filebeat中注释“filebeat模板” 配置我们将在您直接发送时使用的filebeat模板 从filebeat到elastic的日志

更改filebeat的配置

filebeat.prospectors:
- input_type: log
  paths: C:\Users\Sagar\Desktop\elastic_test4\data\log\*.log

output.logstash:
  hosts: ["10.101.00.11:5043"]
您必须将'ingest geoip'过滤器插件安装到elastic中 搜索。如果您没有使用任何外部数据库或服务

您可以使用下面的命令安装弹性插件

elasticsearch-plugin install ingest-geoip
我不确定你的弹性实例，因为它是默认的 侦听9200端口，而不是80端口

您必须更改logstash的配置脚本。下面是这样的东西

input {
    beats {
        host => "10.101.00.11"
        port => "5044"
    }
}

filter {
    grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
    geoip { source => "clientip" }
}

output {
    elasticsearch {
        #hosts => ["elastic-instance-1.es.amazonaws.com:80"]
        hosts => ["elastic-instance-1.es.amazonaws.com:9200"]
        index => "apache-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
}
应用这些配置后，您的输出将如下所示

{
  "_index": "apache-2017.09.21",
  "_type": "log",
  "_id": "AV6kqsr3A-YOTHfOm2US",
  "_version": 1,
  "_score": null,
  "_source": {
    "request": "/apps/cart.jsp?appID=9421",
    "agent": "\"Mozilla/5.0 (Windows 95; sl-SI; rv:1.9.2.20) Gecko/2017-08-19 13:55:15 Firefox/12.0\"",
    "geoip": {
      "city_name": "Beijing",
      "timezone": "Asia/Shanghai",
      "ip": "106.121.102.198",
      "latitude": 39.9289,
      "country_name": "China",
      "country_code2": "CN",
      "continent_code": "AS",
      "country_code3": "CN",
      "region_name": "Beijing",
      "location": {
        "lon": 116.3883,
        "lat": 39.9289
      },
      "region_code": "11",
      "longitude": 116.3883
    },
    "offset": 11050275,
    "auth": "-",
    "ident": "-",
    "input_type": "log",
    "verb": "POST",
    "source": "C:\\Users\\admin\\Desktop\\experiment\\Elastic\\access_log_20170915-005134.log",
    "message": "106.121.102.198 - - [19/Dec/2017:05:54:29 -0700] \"POST /apps/cart.jsp?appID=9421 HTTP/1.0\" 200 4984 \"http://cross.com/login/\" \"Mozilla/5.0 (Windows 95; sl-SI; rv:1.9.2.20) Gecko/2017-08-19 13:55:15 Firefox/12.0\"",
    "type": "log",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "referrer": "\"http://cross.com/login/\"",
    "@timestamp": "2017-09-21T13:39:55.047Z",
    "response": "200",
    "bytes": "4984",
    "clientip": "106.121.102.198",
    "@version": "1",
    "beat": {
      "hostname": "DESKTOP-16QDF02",
      "name": "DESKTOP-16QDF02",
      "version": "5.5.2"
    },
    "host": "DESKTOP-16QDF02",
    "httpversion": "1.0",
    "timestamp": "19/Dec/2017:05:54:29 -0700"
  },
  "fields": {
    "@timestamp": [
      1506001195047
    ]
  },
  "sort": [
    1506001195047
  ]
}
我希望这就是您正在寻找的解决方案。

我不知道您的日志格式是否适用于apache。因为你的日志看起来像这样

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846

149.148.126.144 - - [10/Sep/2017:06:30:44 -0700] "GET /apps/cart.jsp?appID=6944 HTTP/1.0" 200 4981 "http://hernandez.net/app/main/search/homepage.php" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/5322 (KHTML, like Gecko) Chrome/13.0.896.0 Safari/5322"
标准apache日志如下所示

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846

149.148.126.144 - - [10/Sep/2017:06:30:44 -0700] "GET /apps/cart.jsp?appID=6944 HTTP/1.0" 200 4981 "http://hernandez.net/app/main/search/homepage.php" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/5322 (KHTML, like Gecko) Chrome/13.0.896.0 Safari/5322"
我建议您对即将发布的apache日志格式进行标准化。否则，默认的grok配置将不适用于您。然后，您必须为自定义日志编写自己的grok模式。这将解析即将出现的日志行

除此之外，还有很多原因让你犯这样的错误

您没有在filebeat中注释“filebeat模板” 配置我们将在您直接发送时使用的filebeat模板 从filebeat到elastic的日志

更改filebeat的配置

filebeat.prospectors:
- input_type: log
  paths: C:\Users\Sagar\Desktop\elastic_test4\data\log\*.log

output.logstash:
  hosts: ["10.101.00.11:5043"]
您必须将'ingest geoip'过滤器插件安装到elastic中 搜索。如果您没有使用任何外部数据库或服务

您可以使用下面的命令安装弹性插件

elasticsearch-plugin install ingest-geoip
我不确定你的弹性实例，因为它是默认的 侦听9200端口，而不是80端口

您必须更改logstash的配置脚本。下面是这样的东西

input {
    beats {
        host => "10.101.00.11"
        port => "5044"
    }
}

filter {
    grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
    geoip { source => "clientip" }
}

output {
    elasticsearch {
        #hosts => ["elastic-instance-1.es.amazonaws.com:80"]
        hosts => ["elastic-instance-1.es.amazonaws.com:9200"]
        index => "apache-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
}
应用这些配置后，您的输出将如下所示

{
  "_index": "apache-2017.09.21",
  "_type": "log",
  "_id": "AV6kqsr3A-YOTHfOm2US",
  "_version": 1,
  "_score": null,
  "_source": {
    "request": "/apps/cart.jsp?appID=9421",
    "agent": "\"Mozilla/5.0 (Windows 95; sl-SI; rv:1.9.2.20) Gecko/2017-08-19 13:55:15 Firefox/12.0\"",
    "geoip": {
      "city_name": "Beijing",
      "timezone": "Asia/Shanghai",
      "ip": "106.121.102.198",
      "latitude": 39.9289,
      "country_name": "China",
      "country_code2": "CN",
      "continent_code": "AS",
      "country_code3": "CN",
      "region_name": "Beijing",
      "location": {
        "lon": 116.3883,
        "lat": 39.9289
      },
      "region_code": "11",
      "longitude": 116.3883
    },
    "offset": 11050275,
    "auth": "-",
    "ident": "-",
    "input_type": "log",
    "verb": "POST",
    "source": "C:\\Users\\admin\\Desktop\\experiment\\Elastic\\access_log_20170915-005134.log",
    "message": "106.121.102.198 - - [19/Dec/2017:05:54:29 -0700] \"POST /apps/cart.jsp?appID=9421 HTTP/1.0\" 200 4984 \"http://cross.com/login/\" \"Mozilla/5.0 (Windows 95; sl-SI; rv:1.9.2.20) Gecko/2017-08-19 13:55:15 Firefox/12.0\"",
    "type": "log",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "referrer": "\"http://cross.com/login/\"",
    "@timestamp": "2017-09-21T13:39:55.047Z",
    "response": "200",
    "bytes": "4984",
    "clientip": "106.121.102.198",
    "@version": "1",
    "beat": {
      "hostname": "DESKTOP-16QDF02",
      "name": "DESKTOP-16QDF02",
      "version": "5.5.2"
    },
    "host": "DESKTOP-16QDF02",
    "httpversion": "1.0",
    "timestamp": "19/Dec/2017:05:54:29 -0700"
  },
  "fields": {
    "@timestamp": [
      1506001195047
    ]
  },
  "sort": [
    1506001195047
  ]
}
我希望这就是您正在寻找的解决方案。

您可能需要确保apache日志的模式正确：

SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
对于grok匹配的模式，您可以查看上的详细信息

除此之外，你也可以看看

您可能必须确保apache日志的模式正确：

SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
对于grok匹配的模式，您可以查看上的详细信息

除此之外，你也可以看看

虽然这些链接很有用，但最好将相关部分复制到格式化的代码块中。虽然这些链接很有用，但最好将相关部分复制到格式化的代码块中。