<img src="//i.stack.imgur.com/RUiNP.png" height="16" width="18" alt="" class="sponsor tag img">elasticsearch 如何使用logtash和filebeat过滤日志文件_<img Src="//i.stack.imgur.com/RUiNP.png" Height="16" Width="18" Alt="" Class="sponsor Tag Img">elasticsearch_Logstash_Kibana_Elastic Stack_Elk

elasticsearch 如何使用logtash和filebeat过滤日志文件

logstash kibana

elasticsearch 如何使用logtash和filebeat过滤日志文件,elasticsearch,logstash,kibana,elastic-stack,elk,elasticsearch,Logstash,Kibana,Elastic Stack,Elk,我正在尝试筛选文件夹中存储的3个错误文件，但无法将其筛选出来。Kibana将所有三个文件合并为一个文件，不过滤任何数据。我想在filebeats中正确显示错误文件，并使用不同的标记名（文件名）。任何帮助都将被告知 filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /usr/share/filebeat/mylog/*.log output.logstash: hosts: ["logs

我正在尝试筛选文件夹中存储的3个错误文件，但无法将其筛选出来。Kibana将所有三个文件合并为一个文件，不过滤任何数据。我想在filebeats中正确显示错误文件，并使用不同的标记名（文件名）。任何帮助都将被告知

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/mylog/*.log

output.logstash:
  hosts: ["logstash:5044"]

processors:
  - add_docker_metadata:
      host: "unix:///host_docker/docker.sock"
    add_tags:
      tags: ["accesslog"]

logstash.conf

input {
  beats {
    port => 5044
  }
}
filter {
  grok {
    match => { "message" => ["%{TIMESTAMP_ISO8601:timestamp} \[%{LOGLEVEL:log_level}\] \<%{DATA:field_misc}\> %{GREEDYDATA:message}"] }
  }
  json {
    source => "message"
  }
  date{
    match => ["timestamp", "UNIX_MS"]
    target => "@timestamp"
  }
  ruby {
    code => "event.set('indexDay', event.get('[@timestamp]').time.localtime('+09:00').strftime('%Y%m%d'))"
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    template => "/usr/share/logstash/templates/logstash.template.json"
    template_name => "logstash"
    template_overwrite => true
    index => "logstash-%{indexDay}"
    codec => json
  }
  stdout {
    codec => rubydebug
  }
}

事件日志

2020/07/15 19:25:12 adgroup:bidid:9f9dde92c6cd11ea931617530a7d0003:7e b5 82 0a 7e 91 f0 20 db 6b 9f 3a cb 3d 79 91 ab 05 3b 2d  view_count Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN 1
2020/07/15 19:25:12     1594841112  2.0 172.21.0.1:39436    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208
2020/07/15 19:34:44 adgroup:bidid:9f9dde92c6cd11ea931617530a7d0003:7e b5 82 0a 7e 91 f0 20 db 6b 9f 3a cb 3d 79 91 ab 05 3b 2d  view_count map[accname:SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN accnamenoex:Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN adGroupId:Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN adType:4 app_id:SpotX_Roku OS_46041 aud: bidprice:22.00 brand:Roku bundleid:46041 cAdType:0 campaignId:SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video carrier: cat: city:Goodland creativeId:2079953 dealid:30907.885f3.46db device:7 deviceid:e0d2403fd48b73ed7245eff2021b3d9d dma:678 exchId: extendId: geocriteria:0 height:1080 idl: imp_count:3 ip:68.232.59.208 latitude: longitude: model:Digital Video player platform: playersize: pubId:198919 skip: slotvis: state:Kansas timestamp:1594839703 ua:Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) url:https://channelstore.roku.com/details/46041 urlhost:app:Sling TV view_count:1 width:1920 winprice:0.0006818 zip:67735]
2020/07/15 19:34:44 adgroup:bidid:9f9dde92c6cd11ea931617530a7d0003:7e b5 82 0a 7e 91 f0 20 db 6b 9f 3a cb 3d 79 91 ab 05 3b 2d  view_count Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN 1
2020/07/15 19:34:44     1594841684  2.0 172.21.0.1:39600    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208
2020/07/15 20:11:17     1594843877  2.0 172.21.0.1:40362    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208
2020/07/15 20:56:49     1594846609  2.0 172.21.0.1:41216    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208

通过添加字段，可以在Filebeat级别标记不同类型的日志文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /PATH/TO/ACCESS.log
  fields:
    category: access_log
- type: log
  enabled: true
  paths:
    - /PATH/TO/ERROR.log
  fields:
    category: error_log

上面的配置为每个日志类别处理的文档添加了一个新字段

您可以在Kibana中输入（例如）字段名“fields.category:error_log”以过滤错误日志

我们如何使用logstash.conf文件中的标记过滤json错误文件和其他文件？@aws noob如果您需要在logstash中处理日志，则可以使用新添加的字段过滤事件。例如：如果[字段][类别]=“错误日志”。

2020/07/15 19:25:12 adgroup:bidid:9f9dde92c6cd11ea931617530a7d0003:7e b5 82 0a 7e 91 f0 20 db 6b 9f 3a cb 3d 79 91 ab 05 3b 2d  view_count Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN 1
2020/07/15 19:25:12     1594841112  2.0 172.21.0.1:39436    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208
2020/07/15 19:34:44 adgroup:bidid:9f9dde92c6cd11ea931617530a7d0003:7e b5 82 0a 7e 91 f0 20 db 6b 9f 3a cb 3d 79 91 ab 05 3b 2d  view_count map[accname:SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN accnamenoex:Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN adGroupId:Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN adType:4 app_id:SpotX_Roku OS_46041 aud: bidprice:22.00 brand:Roku bundleid:46041 cAdType:0 campaignId:SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video carrier: cat: city:Goodland creativeId:2079953 dealid:30907.885f3.46db device:7 deviceid:e0d2403fd48b73ed7245eff2021b3d9d dma:678 exchId: extendId: geocriteria:0 height:1080 idl: imp_count:3 ip:68.232.59.208 latitude: longitude: model:Digital Video player platform: playersize: pubId:198919 skip: slotvis: state:Kansas timestamp:1594839703 ua:Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) url:https://channelstore.roku.com/details/46041 urlhost:app:Sling TV view_count:1 width:1920 winprice:0.0006818 zip:67735]
2020/07/15 19:34:44 adgroup:bidid:9f9dde92c6cd11ea931617530a7d0003:7e b5 82 0a 7e 91 f0 20 db 6b 9f 3a cb 3d 79 91 ab 05 3b 2d  view_count Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN 1
2020/07/15 19:34:44     1594841684  2.0 172.21.0.1:39600    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208
2020/07/15 20:11:17     1594843877  2.0 172.21.0.1:40362    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208
2020/07/15 20:56:49     1594846609  2.0 172.21.0.1:41216    {[User-Agent:[curl/7.61.1]][Accept:[*/*]]}  2.0 9f9dde92c6cd11ea931617530a7d0003    17  3416706 video   0.681800    view        SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video  SpotX2.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video.Viaero_Wireless_Fort_Morgan_19328_122697_1_OTT_V2_Video_package_Sinclair_TubiTV_FuboTV_MN    198919  2079953 https://channelstore.roku.com/details/46041         1080    1920    7       DMA: 678| DMAName: | Country: | State: Kansas| City: Goodland| Metro: | Region: | GeoCriteriaId: 0| CityStateCountry: | Longitude: | Latitude: | Zipcode: 67735 22.00   Roku    Digital Video player    Mozilla/5.0 (QSP; Roku; AP; 5.4.12.227) 4   0.015000    app:Sling TV    SpotX_Roku OS_46041 46041   e0d2403fd48b73ed7245eff2021b3d9d    30907.885f3.46db    68.232.59.208

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /PATH/TO/ACCESS.log
  fields:
    category: access_log
- type: log
  enabled: true
  paths:
    - /PATH/TO/ERROR.log
  fields:
    category: error_log