Logstash 使用Syslog存储多行

在Logstash和multiline一起工作时遇到一些困难

我使用的容器将所有stdout日志条目作为syslog转发给logstash

这是logstash接收的最终内容。这里有多行应该表示两个事件

<14>2015-02-09T14:25:01Z logspout dev_zservice_1[1]: 2015-02-10 11:55:38.496  INFO 1 --- [tp1302304527-19] c.z.service.DefaultInvoiceService        : Creating with DefaultInvoiceService started...
<14>2015-02-09T14:25:01Z logspout dev_zservice_1[1]: 2015-02-10 11:55:48.596  WARN 1 --- [tp1302304527-19] o.eclipse.jetty.servlet.ServletHandler   : 
<14>2015-02-09T14:25:01Z logspout dev_zservice_1[1]: 
<14>2015-02-09T14:25:01Z logspout dev_zservice_1[1]: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]
<14>2015-02-09T14:25:01Z logspout dev_zservice_1[1]:    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978)
<14>2015-02-09T14:25:01Z logspout dev_zservice_1[1]:    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
<14>2015-02-09T14:25:01Z logspout dev_zservice_1[1]:    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
<14>2015-02-09T14:25:01Z logspout dev_zservice_1[1]:    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)

第二个事件包含此消息，其中包含三个问题：

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]: 

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]:    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978)

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]:    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]:    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]:    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)

<14>2015-02-10T12:59:09Z logspout dev_nginx_1[1]: 192.168.59.3 - - [10/Feb/2015:12:59:09 +0000] "POST /api/invoice/ HTTP/1.1" 500 1115 "http://192.168.59.103/"; "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36" "-"

2015-02-10T12:59:09Z日志喷口开发服务[1]：
2015-02-10T12:59:09Z Logspoutdev_zservice_1[1]：org.springframework.web.util.NestedServletException:请求处理失败；嵌套异常为org.springframework.dao.DataAccessResourceFailureException:在等待与AnyServerSelector{}匹配的服务器时，在10000毫秒后超时。群集状态的客户端视图为{type=Unknown，servers=[{address=mongo:27017，type=Unknown，state=Connecting，exception={com.mongodb.MongoException$Network:exception打开套接字}，由{java.net.unknownhostexption:mongo:Unknown error}引起]；嵌套异常为com.mongodb.MongoTimeoutException：等待与AnyServerSelector{}匹配的服务器时，10000毫秒后超时。群集状态的客户端视图为{type=Unknown，servers=[{address=mongo:27017，type=Unknown，state=Connecting，异常={com.mongodb.MongoException$Network:exception打开套接字}，原因是{java.net.UnknownHostException:mongo:unknown error}]
2015-02-10T12:59:09Z Logspoutdev_zservice_1[1]：位于org.springframework.web.servlet.FrameworkServlet.processRequest（FrameworkServlet.java:978）
2015-02-10T12:59:09Z Logspoutdev_zservice_1[1]：位于org.springframework.web.servlet.FrameworkServlet.doPost（FrameworkServlet.java:868）
2015-02-10T12:59:09Z Logspoutdev_zservice_1[1]：位于javax.servlet.http.HttpServlet.service（HttpServlet.java:707）
2015-02-10T12:59:09Z Logspoutdev_zservice_1[1]：位于org.springframework.web.servlet.FrameworkServlet.service（FrameworkServlet.java:842）
2015-02-10T12:59:09Z Logspoutdev_nginx_1[1]：192.168.59.3---[10/Feb/2015:12:59:09+0000]“POST/api/invoice/HTTP/1.1”500 1115”http://192.168.59.103/“Mozilla/5.0（Macintosh；英特尔Mac OS X 10_10_2）AppleWebKit/537.36（KHTML，类似Gecko）Chrome/40.0.2214.94 Safari/537.36”-”

消息文本包含一行带有不属于此处的dev\u nginx\u 1条目。这应视为一个单独的事件

每一行都包含前缀。

2015-02-10T12:59:09Z logspout开发服务[1]：

每行有一个额外的新行

问题。 为什么dev_nginx_1条目本身不是一个事件。为什么它被认为属于前一个？ 如何去掉消息每行中的syslog前缀。 如何去掉额外的新行？

至于（1），您在多行中使用了

container\u name

。这是时间戳后的字段。在您的示例中，它们都是“logspout”。我觉得似乎是对的

至于（2），每一行都带有前缀和时间戳，因此默认情况下您希望它们在那里用

log\u message

替换

message

，但是我看不到您正在设置

log\u message

。那么，您认为前缀和时间戳是如何被删除的呢？

对于（1），用

%{SYSLOGHOST:container\u name}%{DATA}替换多行模式中的
%{SYSLOGHOST}%{DATA:container\u name}
（当您在工作中使用时）

对于（2）和（3），您可以尝试以下内容：

mutate {
    gsub => [ "message", "<\d+>.*?:\s", "", "message", "\n(\n)", "\1" ]
}

变异{
gsub=>[“message”、“*？：\s”、“、”message”、“\n（\n）”、“\1”]
}

这里，
gsub
设置正在执行两个操作：

检查字段“message”，找到从“”到冒号后跟空格的子字符串，并用空字符串替换这些子字符串
检查字段“message”，找到由两个连续换行符组成的子字符串，并将其替换为一个换行符。它使用组
\1
反向引用执行替换（\n）

，因为如果您尝试使用

\n

本身，Logstash实际上会将其替换为

\\n

，这是行不通的

---

注意：

多行

现在不推荐用作过滤器，因此您需要在输入中使用它作为编解码器（或者它内置于FileBeat之类的东西）。

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]: 

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]:    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978)

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]:    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]:    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

<14>2015-02-10T12:59:09Z logspout dev_zservice_1[1]:    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)

<14>2015-02-10T12:59:09Z logspout dev_nginx_1[1]: 192.168.59.3 - - [10/Feb/2015:12:59:09 +0000] "POST /api/invoice/ HTTP/1.1" 500 1115 "http://192.168.59.103/"; "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36" "-"

mutate {
    gsub => [ "message", "<\d+>.*?:\s", "", "message", "\n(\n)", "\1" ]
}