Logstash 格罗克罐头'；不要创建可选字段
我正在尝试使用grok解析logstash中的日志。 在错误日志中还有两个可选字段，如requestUrl和requestMethod。 但是当我使用grok将日志解析为logstash时，这些错误日志中的字段grok无法创建这两个字段 mypipeline.conf：Logstash 格罗克罐头'；不要创建可选字段,logstash,logstash-grok,Logstash,Logstash Grok,我正在尝试使用grok解析logstash中的日志。 在错误日志中还有两个可选字段，如requestUrl和requestMethod。 但是当我使用grok将日志解析为logstash时，这些错误日志中的字段grok无法创建这两个字段 mypipeline.conf： input { beats { type=>"mytest" port => 5044 } } filter{ if [fields][log_type
input {
beats {
type=>"mytest"
port => 5044
}
}
filter{
if [fields][log_type] == "gbase"
{
if [level] in [ "Error", "Fatal" ]
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{URI:requestUrl} %{USER:requestMethod} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
else
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}" ]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
}
if [fields][log_type] == "finance"
{
if [level] in [ "Error", "Fatal" ]
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:receivedAt} \[%{INT:processId}]\ %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{URI:requestUrl} %{WORD:requestMethod} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
else
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:receivedAt} \[%{INT:processId}]\ %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
#match => [ "time" , "dd/MMM/yyyy:HH:mm:ss Z" ]
target=> "@timestamp"
}
}
output {
if [fields][log_type] == "finance"
{
elasticsearch
{
hosts => ["http://localhost:9200"]
index => "finance-%{+YYYY.MM.dd}"
user => "something"
password => "something"
}
}
if [fields][log_type] == "gbase"
{
elasticsearch
{
hosts => ["http://localhost:9200"]
index => "gbase-%{+YYYY.MM.dd}"
user => "something"
password => "something"
}
}
stdout { codec => rubydebug }
}
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\Git\gbase.API\API\Logs\*.log
tags: ["gbaseapi"]
fields:
log_type: gbase
multiline.pattern: ^\d\d\d\d-\d\d-\d\d
multiline.negate: true
multiline.match: after
multiline.max_lines : 1000
- type: log
enabled: true
paths:
- D:\Git\finance.api\FinanceAPI\logs\*.log
tags: ["financeapi"]
fields:
log_type: finance
multiline.pattern: ^\d\d\d\d-\d\d-\d\d
multiline.negate: true
multiline.match: after
multiline.max_lines : 1000
2020-10-10 21:14:59.8746 22152 WARN Validation bhavin 192.168.43.244 Sensitive data logging is enabled. Log entries and exception messages may include sensitive application data, this mode should only be enabled during development.
2020-10-10 21:15:35.2745 22152 DEBUG QuartzSchedulerThread bhavin 192.168.43.244 Batch acquisition of 0 triggers
2020-10-10 21:15:35.2783 22152 ERROR DeveloperExceptionPageMiddleware bhavin 192.168.43.244 http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.System.NullReferenceException: Object reference not set to an instance of an object.
{
"input" => {
"type" => "log"
},
"processId" => "34076",
"log" => {
"file" => {
"path" => "D:\\Git\\gbase.API\\API\\Logs\\log-2020-10-10.log"
},
"flags" => [
[0] "multiline"
],
"offset" => 43633
},
"clientIp" => "192.168.43.244",
"fields" => {
"log_type" => "gbase"
},
"timestamp" => "2020-10-10 21:29:00.1492",
"ecs" => {
"version" => "1.5.0"
},
"user" => "bhavin",
"agent" => {
"name" => "Bhavin",
"hostname" => "Bhavin",
"type" => "filebeat",
"version" => "7.9.2",
"id" => "b5decaac-61a3-4dbc-860b-4423a0379391",
"ephemeral_id" => "208d7c9d-bdcd-45e9-8f82-ccdb08292203"
},
"level" => "ERROR",
"message" => "http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.GurukulERP.Logic.InvalidDataException: Provide a valid PhoneNumber.\n at GurukulERP.API.Controllers.CRM.LeadsController.New(AdmissionLead model) in D:\\Git\\gbase.API\\API\\Controllers\\CRM\\LeadsController.cs:line 515\n at lambda_method(Closure , Object , Object[] )\n at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at Microsoft.AspNetCore.Diagnostics..Invoke(HttpContext context) at GurukulERP.API.Controllers.CRM.LeadsController.New(AdmissionLead model) in D:\\Git\\gbase.API\\API\\Controllers\\CRM\\LeadsController.cs:line 515\n at lambda_method(Closure , Object , Object[] )\n at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at Microsoft.AspNetCore.Diagnostics..Invoke(HttpContext context)",
"host" => {
"name" => "Bhavin",
"hostname" => "Bhavin",
"architecture" => "x86_64",
"mac" => [
[0] "04:92:26:4a:aa:7b",
[1] "52:3e:aa:6a:59:4c",
[2] "50:3e:aa:6a:59:4c",
[3] "00:ff:73:c0:8e:ca",
[4] "50:3e:aa:6a:59:4c"
],
"id" => "2540a614-ed14-468c-82b6-3c2f1de5b687",
"ip" => [
[ 0] "fe80::8511:1aea:78ea:8722",
[ 1] "169.254.135.34",
[ 2] "fe80::a1dc:1a7d:8369:4ee5",
[ 3] "169.254.78.229",
[ 4] "fe80::c9b5:254f:10b3:191d",
[ 5] "169.254.25.29",
[ 6] "fe80::ac32:76b1:509a:bc5e",
[ 7] "169.254.188.94",
[ 8] "2405:204:808b:8dbb:d456:182e:dd87:ca6",
[ 9] "2405:204:808b:8dbb:3132:508b:53c1:132e",
[10] "fe80::d456:182e:dd87:ca6",
[11] "192.168.43.244"
],
"os" => {
"name" => "Windows 10 Enterprise",
"platform" => "windows",
"kernel" => "10.0.18362.1082 (WinBuild.160101.0800)",
"version" => "10.0",
"family" => "windows",
"build" => "18362.1082"
}
},
"@version" => "1",
"type" => "mytest",
"@timestamp" => 2020-10-10T15:59:02.229Z,
"tags" => [
[0] "gbaseapi",
[1] "beats_input_codec_plain_applied",
[2] "_dateparsefailure"
],
"logger" => "DeveloperExceptionPageMiddleware"
}
{
“输入”=>{
“键入”=>“日志”
},
“processId”=>“34076”，
“日志”=>{
“文件”=>{
“路径”=>“D:\\Git\\gbase.API\\API\\Logs\\log-2020-10-10.log”
},
“标志”=>[
[0]“多行”
],
“偏移量”=>43633
},
“clientIp”=>“192.168.43.244”，
“字段”=>{
“日志类型”=>“gbase”
},
“时间戳”=>“2020-10-10 21:29:00.1492”，
“ecs”=>{
“版本”=>“1.5.0”
},
“用户”=>“bhavin”，
“代理”=>{
“name”=>“Bhavin”，
“主机名”=>“Bhavin”，
“键入”=>“文件节拍”，
“版本”=>“7.9.2”，
“id”=>“B5DECAC-61a3-4dbc-860b-4423a0379391”，
“临时id”=>“208d7c9d-bdcd-45e9-8f82-ccdb08292203”
},
“级别”=>“错误”，
“消息”=>”http://bhavin/Leads/New POST执行请求时发生未经处理的异常。GurukulERP.Logic.InvalidDataException:提供有效的电话号码。\n在D:\\Git\\gbase.API\\API\\Controllers\\CRM\\LeadsController.cs中的GurukulERP.API.Controllers.CRM.LeadsController.New（AdmissionLead model）处：lambda\u方法第515行\n（闭包，对象，对象[]）\n位于Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute（对象目标，对象[]参数）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute（IActionResultTypeMapper，ObjectMethodExecutor，对象控制器，对象[]参数）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync（）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next（状态和下一个、范围和范围、对象和状态、布尔值和已完成）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync（）\n--从引发异常的上一个位置开始的堆栈结束跟踪---\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow（ActionExecutedContextSealed上下文）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next（状态和下一个、范围和范围、对象和状态、布尔值和isCompleted）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync（）\n--从引发异常的上一个位置开始的堆栈结束跟踪----\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g|0（ResourceInvoker invoker，Task lastTask，State next，Scope Scope，Object State，Boolean isCompleted）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow（ResourceExecutedContextSealed context）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.next（State&next，Scope&Scope，Object&State，Boolean&isCompleted）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync（）\n--来自引发异常的上一个位置的堆栈结束跟踪----\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g_u等待| 17_0（ResourceInvoker invoker invoker，Task，IDisposable作用域）\n位于Microsoft.AspNetCore.Routing.EndpointMiddleware.g_uwaitRequestTask | 6_0（端点、任务请求任务、ILogger记录器）\n位于Microsoft.AspNetCore.Session.SessionMiddleware.Invoke（HttpContext上下文）\n位于Microsoft.AspNetCore.Session.SessionMiddleware.Invoke（HttpContext上下文）\n位于Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke（HttpContext上下文）\n位于Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke（HttpContext上下文）\n位于Swashback.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke（HttpContext上下文）\n位于Swashback.AspNetCore.Swagger.Swagger中间件.Invoke（HttpContext HttpContext，ISwaggerProvider swaggerProvider）\n位于Microsoft.AspNetCore.Diagnostics..Invoke（HttpContext context）位于GurukulERP.API.Controllers.CRM.LeadsController.New（AdmissionLead model）在D:\\Git\\gbase.API\\API\\Controllers\\CRM\\LeadsController.cs:lambda_方法（闭包，对象，对象[]）的第515行\n在Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute（对象目标，对象[]参数）的Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute的第515行（位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync（）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next的IActionResultTypeMapper映射器、ObjectMethodExecutor执行器、对象控制器、对象[]参数）\n\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync（）\n---从引发异常的上一个位置开始的堆栈结束跟踪---\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow（ActionExecutedContextSealed上下文）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next（State&Next、Scope&Scope、Object&State、Boolean&isCompleted）\n位于Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnelFilterAsync（）\n--从引发异常的上一个位置开始的堆栈结束跟踪---\n在Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g_等待| 24_0（ResourceInvoker invoker，Task lastTask，State next，Scope Scope，Object State，Boolean isCompleted）\n在Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow（ResourceExecuteContextse）
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}" }
overwrite => "message"
}
if [message] =~ "^http:" {
grok {
match => { "message" => "%{URI:requestUrl} %{USER:requestMethod} %{GREEDYDATA:message}" }
overwrite => "message"
}
}
}
http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.System.NullReferenceException: Object reference not set to an instance of an object.