Parsing 如何在具有多个时间戳和行的日志存储配置中解析/筛选此复杂日志文件_Parsing_<img Src="//i.stack.imgur.com/RUiNP.png" Height="16" Width="18" Alt="" Class="sponsor Tag Img">elasticsearch_Logstash_Logstash Grok_Logstash Configuration

Parsing 如何在具有多个时间戳和行的日志存储配置中解析/筛选此复杂日志文件

parsing logstash

Parsing 如何在具有多个时间戳和行的日志存储配置中解析/筛选此复杂日志文件,parsing,elasticsearch,logstash,logstash-grok,logstash-configuration,Parsing,elasticsearch,Logstash,Logstash Grok,Logstash Configuration,我有以下日志文件 2017-06-13 13:00:01,494 - INFO [Line: 48]: Begin logging 2017-06-13 13:00:01,494 - DEBUG [Line: 89]: Writing to lockfile. Lockfile location: /etc/conf/the/back_up.txt 2017-06-13 13:00:03,521 - WARNING [Line: 449]: Snapshotting is not enabl

我有以下日志文件

2017-06-13 13:00:01,494 - INFO [Line: 48]: Begin logging

2017-06-13 13:00:01,494 - DEBUG [Line: 89]: Writing to lockfile. Lockfile location: /etc/conf/the/back_up.txt

2017-06-13 13:00:03,521 - WARNING [Line: 449]: Snapshotting is not enabled

2017-06-13 13:06:15,663 - INFO [Line: 898]: stderr: 

17/06/13 13:00:13 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=true,maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/DAT/ABC], targetPath=/etc/conf/the/back_up.txt, tar getPathExists=true, preserveRawXattrs=false}

17/06/13 13:00:13 INFO impl.TimelineClientImpl: Timeline service address: http://ip-192-168-X-XX.xyz:9000/v1/example/
17/06/13 13:00:14 INFO tools.DistCp: DistCp job log path: /var/tar/xar

17/06/13 13:00:20 INFO mapreduce.JobSubmitter: number of splits:22

17/06/13 13:00:21 INFO impl.YarnClientImpl: Submitted application application_1495940390018_0989

17/06/13 13:00:21 INFO mapreduce.Job: Running job: job_1495940390018_0989 
17/06/13 13:00:29 INFO mapreduce.Job: Job job_1495940390018_0989 running in uber mode : false
17/06/13 13:00:29 INFO mapreduce.Job: map 0% reduce 0%
17/06/13 13:00:46 INFO mapreduce.Job: map 11% reduce 0%
17/06/13 13:00:47 INFO mapreduce.Job: map 17% reduce 0%
17/06/13 13:00:48 INFO mapreduce.Job: map 18% reduce 0%
17/06/13 13:00:49 INFO mapreduce.Job: map 23% reduce 0%
17/06/13 13:00:50 INFO mapreduce.Job: map 28% reduce 0%
17/06/13 13:00:51 INFO mapreduce.Job: map 29% reduce 0%
17/06/13 13:00:52 INFO mapreduce.Job: map 32% reduce 0%
17/06/13 13:00:53 INFO mapreduce.Job: map 37% reduce 0%
17/06/13 13:00:54 INFO mapreduce.Job: map 38% reduce 0%
17/06/13 13:00:55 INFO mapreduce.Job: map 41% reduce 0%
17/06/13 13:00:56 INFO mapreduce.Job: map 44% reduce 0%
17/06/13 13:00:57 INFO mapreduce.Job: map 45% reduce 0%
17/06/13 13:00:58 INFO mapreduce.Job: map 47% reduce 0%
17/06/13 13:00:59 INFO mapreduce.Job: map 48% reduce 0%
17/06/13 13:01:00 INFO mapreduce.Job: map 49% reduce 0%
17/06/13 13:01:07 INFO mapreduce.Job: map 54% reduce 0%
17/06/13 13:01:08 INFO mapreduce.Job: map 57% reduce 0%
17/06/13 13:01:10 INFO mapreduce.Job: map 59% reduce 0%
17/06/13 13:01:11 INFO mapreduce.Job: map 60% reduce 0%
17/06/13 13:01:13 INFO mapreduce.Job: map 62% reduce 0%
17/06/13 13:01:14 INFO mapreduce.Job: map 63% reduce 0%
17/06/13 13:01:15 INFO mapreduce.Job: map 64% reduce 0%
17/06/13 13:01:16 INFO mapreduce.Job: map 65% reduce 0%
17/06/13 13:01:31 INFO mapreduce.Job: map 76% reduce 0%
17/06/13 13:01:35 INFO mapreduce.Job: map 77% reduce 0%
17/06/13 13:01:39 INFO mapreduce.Job: map 78% reduce 0%
17/06/13 13:01:44 INFO mapreduce.Job: map 79% reduce 0%
17/06/13 13:01:48 INFO mapreduce.Job: map 80% reduce 0%
17/06/13 13:01:52 INFO mapreduce.Job: map 81% reduce 0%
17/06/13 13:01:55 INFO mapreduce.Job: map 82% reduce 0%
17/06/13 13:01:58 INFO mapreduce.Job: map 83% reduce 0%
17/06/13 13:02:01 INFO mapreduce.Job: map 84% reduce 0%
17/06/13 13:02:06 INFO mapreduce.Job: map 85% reduce 0%
17/06/13 13:02:09 INFO mapreduce.Job: map 86% reduce 0%
17/06/13 13:02:12 INFO mapreduce.Job: map 87% reduce 0%
17/06/13 13:02:16 INFO mapreduce.Job: map 88% reduce 0%
17/06/13 13:02:18 INFO mapreduce.Job: map 89% reduce 0%
17/06/13 13:02:23 INFO mapreduce.Job: map 90% reduce 0%
17/06/13 13:02:28 INFO mapreduce.Job: map 91% reduce 0%
17/06/13 13:02:36 INFO mapreduce.Job: map 92% reduce 0%
17/06/13 13:02:42 INFO mapreduce.Job: map 93% reduce 0%
17/06/13 13:02:47 INFO mapreduce.Job: map 94% reduce 0%
17/06/13 13:02:51 INFO mapreduce.Job: map 95% reduce 0%
17/06/13 13:02:57 INFO mapreduce.Job: map 96% reduce 0%
17/06/13 13:03:04 INFO mapreduce.Job: map 97% reduce 0%
17/06/13 13:03:10 INFO mapreduce.Job: map 98% reduce 0%
17/06/13 13:03:30 INFO mapreduce.Job: map 99% reduce 0%
17/06/13 13:03:58 INFO mapreduce.Job: map 100% reduce 0%

17/06/13 13:06:15 INFO mapreduce.Job: Job job_1495940390018_0989 completed successfully 

17/06/13 13:06:15 INFO mapreduce.Job: Counters: 33
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=30634
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1810172
HDFS: Number of bytes written=6602
HDFS: Number of read operations=21710
HDFS: Number of large read operations=0
HDFS: Number of write operations=4461
Job Counters 
Launched map tasks=22
Other local map tasks=22
Total time spent by all maps in occupied slots (ms)=09878
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=170939
Total vcore-milliseconds taken by all map tasks=17049
Total megabyte-milliseconds taken by all map tasks=1747536
Map-Reduce Framework
Map input records=417
Map output records=175
Input split bytes=262
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=3338
CPU time spent (ms)=3180
Physical memory (bytes) snapshot=480768
Virtual memory (bytes) snapshot=61798624
Total committed heap usage (bytes)=2965728
File Input Format Counters 
Bytes Read=17510
File Output Format Counters 
Bytes Written=6616
org.apache.hadoop.tools.mapred.CopyMapper$Counter
BYTESSKIPPED=11361
COPY=1242
SKIP=3175
2017-06-13 13:06:15,668 - INFO [Line: 904]: Distcp -log output stored in /var/AB/CY/

2017-06-13 13:06:15,673 - INFO [Line: 132]: End logging 

2017-06-13 13:07:01,494 - INFO [Line: 48]: Begin logging

. <similar to above logs>

. <similar to above logs>

. <similar to above logs>

2017-06-13 13:07:15,673 - INFO [Line: 132]: End logging

2017-06-13 13:00:01494-信息[行：48]：开始记录
2017-06-13 13:00:01494-调试[行：89]：写入锁文件。锁文件位置：/etc/conf/the/back\u up.txt
2017-06-13 13:00:03521-警告[行：449]：快照未启用
2017-06-13 13:06:15663-信息[行：898]：标准：
17/06/13 13:00:13 INFO tools.DistCp:Input Options:DistCpOptions{atomicCommit=false，syncFolder=true，deletemission=false，ignoreFailures=true，maxMaps=20，sslConfigurationFile='null'，copyrategy='uniformsize'，sourceFileListing=null，sourcepath=[/DAT/ABC]，targetPath=/etc/conf/the/back_up.txt，tar getPathExists=true，preserveRawXattrs=false}
17/06/13 13:00:13信息impl.TimelineClientImpl:Timeline服务地址：http://ip-192-168-X-XX.xyz:9000/v1/example/
17/06/13 13:00:14信息工具。DistCp:DistCp作业日志路径：/var/tar/xar
17/06/13 13:00:20信息mapreduce.JobSubmitter:拆分数：22
2013年6月17日13:00:21信息建议客户建议：提交的申请申请
17/06/13 13:00:21信息mapreduce。作业：正在运行作业：作业_1495940390018 _0989
17/06/13 13:00:29信息mapreduce.作业：作业作业在uber模式下运行：false
17/06/13 13:00:29信息地图还原。作业：地图0%还原0%
17/06/13 13:00:46信息地图还原。作业：地图11%还原0%
17/06/13 13:00:47信息地图还原。作业：地图17%还原0%
17/06/13 13:00:48信息地图还原。作业：地图18%还原0%
17/06/13 13:00:49信息地图还原。作业：地图23%还原0%
17/06/13 13:00:50信息地图还原。作业：地图28%还原0%
17/06/13 13:00:51信息地图还原。工作：地图29%还原0%
17/06/13 13:00:52信息地图还原。作业：地图32%还原0%
17/06/13 13:00:53信息地图还原。作业：地图37%还原0%
17/06/13 13:00:54信息地图还原。作业：地图38%还原0%
17/06/13 13:00:55信息地图还原。作业：地图41%还原0%
17/06/13 13:00:56信息地图还原。作业：地图44%还原0%
17/06/13 13:00:57信息地图还原。作业：地图45%还原0%
17/06/13 13:00:58信息地图还原。作业：地图47%还原0%
17/06/13 13:00:59信息地图还原。作业：地图48%还原0%
17/06/13 13:01:00信息地图还原。作业：地图49%还原0%
17/06/13 13:01:07信息地图还原。工作：地图54%还原0%
17/06/13 13:01:08信息地图还原。作业：地图57%还原0%
17/06/13 13:01:10信息地图还原。作业：地图59%还原0%
17/06/13 13:01:11信息地图还原。作业：地图60%还原0%
17/06/13 13:01:13信息地图还原。作业：地图62%还原0%
17/06/13 13:01:14信息地图还原。作业：地图63%还原0%
17/06/13 13:01:15信息mapreduce。作业：映射64%减少0%
17/06/13 13:01:16信息地图还原。作业：地图65%还原0%
17/06/13 13:01:31信息地图还原。作业：地图76%还原0%
17/06/13 13:01:35信息地图还原。作业：地图77%还原0%
17/06/13 13:01:39信息地图还原。作业：地图78%还原0%
17/06/13 13:01:44信息地图还原。工作：地图79%还原0%
17/06/13 13:01:48信息地图还原。作业：地图80%还原0%
17/06/13 13:01:52信息地图还原。作业：地图81%还原0%
17/06/13 13:01:55信息mapreduce.工作：映射82%减少0%
17/06/13 13:01:58信息地图还原。作业：地图83%还原0%
17/06/13 13:02:01信息地图还原。作业：地图84%还原0%
17/06/13 13:02:06信息地图还原。作业：地图85%还原0%
17/06/13 13:02:09信息地图还原。作业：地图86%还原0%
17/06/13 13:02:12信息地图还原。工作：地图87%还原0%
17/06/13 13:02:16信息地图还原。作业：地图88%还原0%
17/06/13 13:02:18信息地图还原。作业：地图89%还原0%
17/06/13 13:02:23信息地图还原。作业：地图90%还原0%
17/06/13 13:02:28信息地图还原。作业：地图91%还原0%
17/06/13 13:02:36信息地图还原。作业：地图92%还原0%
17/06/13 13:02:42信息地图还原。作业：地图93%还原0%
17/06/13 13:02:47信息地图还原。作业：地图94%还原0%
17/06/13 13:02:51信息地图还原。作业：地图95%还原0%
17/06/13 13:02:57信息地图还原。作业：地图96%还原0%
17/06/13 13:03:04信息地图还原。作业：地图97%还原0%
17/06/13 13:03:10信息地图还原。作业：地图98%还原0%
17/06/13 13:03:30信息地图还原。作业：地图99%还原0%
17/06/13 13:03:58信息地图还原。作业：地图100%还原0%
17/06/13 13:06:15信息mapreduce。作业：作业作业成功完成
17/06/13 13:06:15信息地图还原。工作：计数器：33
文件系统计数器
文件：读取的字节数=0
文件：写入的字节数=30634
文件：读取操作数=0
文件：大型读取操作数=0
文件：写入操作数=0
HDFS:读取的字节数=1810172
HDFS:写入的字节数=6602
HDFS:读取操作数=21710
HDFS:大型读取操作数=0
HDFS:写入操作数=4461
工作计数器
已启动的地图任务=22
其他本地地图任务=22
所有地图在占用插槽中花费的总时间（毫秒）=09878
占用的插槽中所有减少项花费的总时间（ms）=0
所有map任务花费的总时间（毫秒）=170939
所有映射任务占用的vcore总毫秒数=17049
所有映射任务占用的总MB毫秒=1747536
Map-Reduce框架
地图输入记录=417
地图输出记录=175
输入拆分字节=262
溢出的记录=0
失败的洗牌=0
合并地图输出=0
GC经过的时间（ms）=3338
所用CPU时间（毫秒）=3180
物理内存（字节）快照=480768
虚拟内存（字节）快照=61798624
提交的堆使用总量（字节）=2965728
文件输入格式计数器
读取字节数=17510
文件输出格式计数器
写入的字节数=6616
org.apache.hadoop.tools.mapred.CopyMapper$Counter
BYTESSKIPPED=11361
副本=1242
跳过=3175
2017-06-13 13:06:15668-信息[行：904]：Distcp-日志输出存储在/var/AB/CY中/
2017-06-13 13:06:15673-信息[行：132]：结束日志记录
2017-06-13 13:07:01494-信息[行：48]：开始记录
. 
. 
. 
2017-06-13 13:07:15673-信息[行：132]：结束日志记录

。。

。 .

等等

然后，对于不同的作业ID，它将与上面的部分类似，以“开始日志记录”开头，以“结束日志记录”结尾，如上面粗体显示的

所以我这里的问题是：如何在log stash config中解析该日志，在这里我希望看到

input {
  stdin {
    codec => multiline {
      pattern => "End logging"
      what => "next"
      negate => true
    }
  }
}

output {
  stdout {
    codec => "rubydebug"
  }
}

{
      "@version" => "1",
       "message" => "2017-06-13 13:00:01,494 - INFO [Line: 48]: Begin logging\n\n2017-06-13 13:00:01,494 - DEBUG [Line: 89]: Writing to lockfile. Lockfile location: /etc/conf/the/back_up.txt\n\n2017-06-13 13:00:03,521 - WARNING [Line: 449]: Snapshotting is not enabled\n\n2017-06-13 13:06:15,663 - INFO [Line: 898]: stderr: \n\n17/06/13 13:00:13 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=true,maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/DAT/ABC], targetPath=/etc/conf/the/back_up.txt, tar getPathExists=true, preserveRawXattrs=false}\n\n17/06/13 13:00:13 INFO impl.TimelineClientImpl: Timeline service address: http://ip-192-168-X-XX.xyz:9000/v1/example/\n17/06/13 13:00:14 INFO tools.DistCp: DistCp job log path: /var/tar/xar\n\n17/06/13 13:00:20 INFO mapreduce.JobSubmitter: number of splits:22\n\n17/06/13 13:00:21 INFO impl.YarnClientImpl: Submitted application application_1495940390018_0989\n\n17/06/13 13:00:21 INFO mapreduce.Job: Running job: job_1495940390018_0989 \n17/06/13 13:00:29 INFO mapreduce.Job: Job job_1495940390018_0989 running in uber mode : false\n17/06/13 13:00:29 INFO mapreduce.Job: map 0% reduce 0%\n17/06/13 13:00:46 INFO mapreduce.Job: map 11% reduce 0%\n17/06/13 13:00:47 INFO mapreduce.Job: map 17% reduce 0%\n17/06/13 13:00:48 INFO mapreduce.Job: map 18% reduce 0%\n17/06/13 13:00:49 INFO mapreduce.Job: map 23% reduce 0%\n17/06/13 13:00:50 INFO mapreduce.Job: map 28% reduce 0%\n17/06/13 13:00:51 INFO mapreduce.Job: map 29% reduce 0%\n17/06/13 13:00:52 INFO mapreduce.Job: map 32% reduce 0%\n17/06/13 13:00:53 INFO mapreduce.Job: map 37% reduce 0%\n17/06/13 13:00:54 INFO mapreduce.Job: map 38% reduce 0%\n17/06/13 13:00:55 INFO mapreduce.Job: map 41% reduce 0%\n17/06/13 13:00:56 INFO mapreduce.Job: map 44% reduce 0%\n17/06/13 13:00:57 INFO mapreduce.Job: map 45% reduce 0%\n17/06/13 13:00:58 INFO mapreduce.Job: map 47% reduce 0%\n17/06/13 13:00:59 INFO mapreduce.Job: map 48% reduce 0%\n17/06/13 13:01:00 INFO mapreduce.Job: map 49% reduce 0%\n17/06/13 13:01:07 INFO mapreduce.Job: map 54% reduce 0%\n17/06/13 13:01:08 INFO mapreduce.Job: map 57% reduce 0%\n17/06/13 13:01:10 INFO mapreduce.Job: map 59% reduce 0%\n17/06/13 13:01:11 INFO mapreduce.Job: map 60% reduce 0%\n17/06/13 13:01:13 INFO mapreduce.Job: map 62% reduce 0%\n17/06/13 13:01:14 INFO mapreduce.Job: map 63% reduce 0%\n17/06/13 13:01:15 INFO mapreduce.Job: map 64% reduce 0%\n17/06/13 13:01:16 INFO mapreduce.Job: map 65% reduce 0%\n17/06/13 13:01:31 INFO mapreduce.Job: map 76% reduce 0%\n17/06/13 13:01:35 INFO mapreduce.Job: map 77% reduce 0%\n17/06/13 13:01:39 INFO mapreduce.Job: map 78% reduce 0%\n17/06/13 13:01:44 INFO mapreduce.Job: map 79% reduce 0%\n17/06/13 13:01:48 INFO mapreduce.Job: map 80% reduce 0%\n17/06/13 13:01:52 INFO mapreduce.Job: map 81% reduce 0%\n17/06/13 13:01:55 INFO mapreduce.Job: map 82% reduce 0%\n17/06/13 13:01:58 INFO mapreduce.Job: map 83% reduce 0%\n17/06/13 13:02:01 INFO mapreduce.Job: map 84% reduce 0%\n17/06/13 13:02:06 INFO mapreduce.Job: map 85% reduce 0%\n17/06/13 13:02:09 INFO mapreduce.Job: map 86% reduce 0%\n17/06/13 13:02:12 INFO mapreduce.Job: map 87% reduce 0%\n17/06/13 13:02:16 INFO mapreduce.Job: map 88% reduce 0%\n17/06/13 13:02:18 INFO mapreduce.Job: map 89% reduce 0%\n17/06/13 13:02:23 INFO mapreduce.Job: map 90% reduce 0%\n17/06/13 13:02:28 INFO mapreduce.Job: map 91% reduce 0%\n17/06/13 13:02:36 INFO mapreduce.Job: map 92% reduce 0%\n17/06/13 13:02:42 INFO mapreduce.Job: map 93% reduce 0%\n17/06/13 13:02:47 INFO mapreduce.Job: map 94% reduce 0%\n17/06/13 13:02:51 INFO mapreduce.Job: map 95% reduce 0%\n17/06/13 13:02:57 INFO mapreduce.Job: map 96% reduce 0%\n17/06/13 13:03:04 INFO mapreduce.Job: map 97% reduce 0%\n17/06/13 13:03:10 INFO mapreduce.Job: map 98% reduce 0%\n17/06/13 13:03:30 INFO mapreduce.Job: map 99% reduce 0%\n17/06/13 13:03:58 INFO mapreduce.Job: map 100% reduce 0%\n\n17/06/13 13:06:15 INFO mapreduce.Job: Job job_1495940390018_0989 completed successfully \n\n17/06/13 13:06:15 INFO mapreduce.Job: Counters: 33\nFile System Counters\nFILE: Number of bytes read=0\nFILE: Number of bytes written=30634\nFILE: Number of read operations=0\nFILE: Number of large read operations=0\nFILE: Number of write operations=0\nHDFS: Number of bytes read=1810172\nHDFS: Number of bytes written=6602\nHDFS: Number of read operations=21710\nHDFS: Number of large read operations=0\nHDFS: Number of write operations=4461\nJob Counters \nLaunched map tasks=22\nOther local map tasks=22\nTotal time spent by all maps in occupied slots (ms)=09878\nTotal time spent by all reduces in occupied slots (ms)=0\nTotal time spent by all map tasks (ms)=170939\nTotal vcore-milliseconds taken by all map tasks=17049\nTotal megabyte-milliseconds taken by all map tasks=1747536\nMap-Reduce Framework\nMap input records=417\nMap output records=175\nInput split bytes=262\nSpilled Records=0\nFailed Shuffles=0\nMerged Map outputs=0\nGC time elapsed (ms)=3338\nCPU time spent (ms)=3180\nPhysical memory (bytes) snapshot=480768\nVirtual memory (bytes) snapshot=61798624\nTotal committed heap usage (bytes)=2965728\nFile Input Format Counters \nBytes Read=17510\nFile Output Format Counters \nBytes Written=6616\norg.apache.hadoop.tools.mapred.CopyMapper$Counter\nBYTESSKIPPED=11361\nCOPY=1242\nSKIP=3175\n2017-06-13 13:06:15,668 - INFO [Line: 904]: Distcp -log output stored in /var/AB/CY/\n\n2017-06-13 13:06:15,673 - INFO [Line: 132]: End logging ",
          "tags" => [
        [0] "multiline"
    ]
}
{
      "@version" => "1",
       "message" => "\n2017-06-13 13:07:01,494 - INFO [Line: 48]: Begin logging\n\n. <similar to above logs>\n\n. <similar to above logs>\n\n. <similar to above logs>\n\n2017-06-13 13:07:15,673 - INFO [Line: 132]: End logging",
          "tags" => [
        [0] "multiline"
    ]
}