Python 使用ELK堆栈从日志创建json对象

我有一些由质量中心（HP Alm）和Jenkins之间的桥梁生成的日志：如下所示：

    Building in workspace D:\Tools\Jenkins\workspace\*******-JenkinsWithQC
[Dani-JenkinsWithQC] $ D:\Tools\Jenkins\workspace\*******-JenkinsWithQC\HpToolsLauncher.exe -paramfile props20022014150821066.txt
"Started..."
Timeout is set to: 5
Run mode is set to: RUN_REMOTE
============================================================================
Starting test set execution
Test set name: JenkinsIntegartionTest, Test set id: 2457
"Number of tests in set: "2
Test 1: [1]Login will run on host: si0vm839
Test 2: [1]Logout will run on host: si0vm839
"Scheduler started at:20.02.2014 15:08:28
-------------------------------------------------------------------------------------------------------
20.02.2014 15:08:29 Running: [1]Login
20.02.2014 15:08:29 Running test: [1]Login, Test id: 938, Test instance id: 1412
Test: [1]Login, Id: 1412, Execution status: Running
Test: [1]Login, Id: 1412, Execution status: Error, Message: Access is denied
20.02.2014 15:08:33 Test complete: [1]Login
-------------------------------------------------------------------------------------------------------
20.02.2014 15:08:33 Running: [1]Logout
20.02.2014 15:08:33 Running test: [1]Logout, Test id: 939, Test instance id: 1413
Test: [1]Logout, Id: 1413, Execution status: Running
Test: [1]Logout, Id: 1413, Execution status: Error, Message: Access is denied
==============\nJob timed out!\n==============
================================================
Run status: Job failed, total tests: 2, succeeded: 0, failures: 0, errors: 2
Build step 'Execute HP tests from HP ALM' changed build result to FAILURE
Finished: FAILURE

{
    'Build': {
         'TestSet': [
             {
                  'name' .   : 'execution',
                  'id'       : 2457,
                  'startedAt': '20.02.2014 15:08:28',
                  'nbOfTests': 2,
                  'tests' .  : [
                      {
                           'name': "[1]Login",
                           'host': "si0vm839"
                      }
                  ]

             } 
         ]
    }
}

我想从这些日志创建一个JSON，如下所示：

    Building in workspace D:\Tools\Jenkins\workspace\*******-JenkinsWithQC
[Dani-JenkinsWithQC] $ D:\Tools\Jenkins\workspace\*******-JenkinsWithQC\HpToolsLauncher.exe -paramfile props20022014150821066.txt
"Started..."
Timeout is set to: 5
Run mode is set to: RUN_REMOTE
============================================================================
Starting test set execution
Test set name: JenkinsIntegartionTest, Test set id: 2457
"Number of tests in set: "2
Test 1: [1]Login will run on host: si0vm839
Test 2: [1]Logout will run on host: si0vm839
"Scheduler started at:20.02.2014 15:08:28
-------------------------------------------------------------------------------------------------------
20.02.2014 15:08:29 Running: [1]Login
20.02.2014 15:08:29 Running test: [1]Login, Test id: 938, Test instance id: 1412
Test: [1]Login, Id: 1412, Execution status: Running
Test: [1]Login, Id: 1412, Execution status: Error, Message: Access is denied
20.02.2014 15:08:33 Test complete: [1]Login
-------------------------------------------------------------------------------------------------------
20.02.2014 15:08:33 Running: [1]Logout
20.02.2014 15:08:33 Running test: [1]Logout, Test id: 939, Test instance id: 1413
Test: [1]Logout, Id: 1413, Execution status: Running
Test: [1]Logout, Id: 1413, Execution status: Error, Message: Access is denied
==============\nJob timed out!\n==============
================================================
Run status: Job failed, total tests: 2, succeeded: 0, failures: 0, errors: 2
Build step 'Execute HP tests from HP ALM' changed build result to FAILURE
Finished: FAILURE

{
    'Build': {
         'TestSet': [
             {
                  'name' .   : 'execution',
                  'id'       : 2457,
                  'startedAt': '20.02.2014 15:08:28',
                  'nbOfTests': 2,
                  'tests' .  : [
                      {
                           'name': "[1]Login",
                           'host': "si0vm839"
                      }
                  ]

             } 
         ]
    }
}

我可以用一个python脚本来实现这一点，该脚本使用正则表达式逐行处理并创建JSON，但我认为它没有经过优化，因为我是python新手

我想知道是否可以使用Logstash将它们存储在Elasticsearch中，并构建看板仪表板。

---

提前感谢您的帮助：）

使用Logstash可以做到这一点，包括摄入时的多行编解码器和一些真正粗糙的grok语句。那不是我期待的项目。事实上，我会用两种方法之一解决这个问题：

使用

ruby

过滤器 这允许内联ruby将这个庞大的字符串传递到事件中所需的字段中。实际上，您正在以这种方式编写自己的

编解码器

使用
exec
输入
这将把转换为可解析格式的过程卸载到外部脚本中，然后外部脚本将其发送到logstash以注入ElasticSearch。由于我认为您不会在一秒钟内运行多个，这可能对您有用。
谢谢您的回复，您能再解释一下吗。。我是新来的麋鹿堆栈。。。谢谢again@steep2000ruby和exec过滤器都是Logstash的专家模式功能，因为它们处理现有过滤器无法处理的
其他情况。要使两者都能很好地发挥作用并相互融合，一个入门课程就是单独进行30分钟的会议演讲。使用
exec
插件，您可以调用已经构建的python脚本。它需要调整输入/输出格式，但这是最快的工作方式。