Python解析web访问日志_Python_Regex

Python解析web访问日志

python regex

Python解析web访问日志,python,regex,Python,Regex,我试图解析访问日志中的特定数据，日志格式各不相同，可能来自nginx或apache。我需要获得以下数据：远程主机IP 请求日期时间请求类型{GET | POST | PUT | etc} 请求路径{/main/index.html etc} HTTP版本{HTTP 1.1 | HTTP 1.0} HTTP响应代码{200 | 400 | 403…etc} 我尝试使用split，但由于日志格式不总是相同，因此它并不总是有效： sample = """ ::1

我试图解析访问日志中的特定数据，日志格式各不相同，可能来自nginx或apache。我需要获得以下数据：

远程主机IP
请求日期时间
请求类型{GET | POST | PUT | etc}
请求路径{/main/index.html etc}
HTTP版本{HTTP 1.1 | HTTP 1.0}
HTTP响应代码{200 | 400 | 403…etc}

我尝试使用split，但由于日志格式不总是相同，因此它并不总是有效：

sample = """
::1 - - [03/Jan/2018:21:28:49 +0100] "GET /moodle/course/view.php?id=19 HTTP/1.1" 200 78325 "http://localhost/moodle/login/index.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
83.198.250.175 - - [22/Mar/2009:07:40:06 +0100] "GET /style.css HTTP/1.1" 200 1692 "http://www.example.org/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Wanadoo 6.7; Orange 8.0)" "-"
212.31.110.34 0.597 - [16/May/2018:12:30:44 +0000] safefin.example.com "GET / HTTP/1.1" 200 18193 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36"
151.227.152.48 - - [02/Jul/2014:14:35:55 +0100] "GET /css/main.css HTTP/1.1" 200 4658 "http://example.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] "POST /administrator/index.php HTTP/1.1" 200 4494 "http://example.net/administrator/" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
80.91.33.133 - - [17/May/2015:08:05:24 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.17)"
217.168.17.5 - - [17/May/2015:08:05:34 +0000] "GET /downloads/product_1 HTTP/1.1" 200 490 "-" "Debian APT-HTTP/1.3 (0.8.10.3)"
192.168.0.11 - - [27/Jun/2016:18:36:14 -0500] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Linux; Android 5.1.1; SM-N910T Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.81 Mobile Safari/537.36"
51.68.152.26 - - [09/Apr/2019:01:37:30 +0400] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
71.169.154.24 - - [01/Mar/2015:20:58:55 -0500] "GET /BarHarborcemeteries/Burns-RichardsonCemeteryimages/general%20view%20(2008).jpg HTTP/1.1" 200 165457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
94.90.115.82 - - [02/Apr/2012:04:56:17 +0900] "GET /manager/html HTTP/1.1" 404 77 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
172.20.32.1 - - [25/Feb/2015:10:42:29 +0300] "PUT /putfile?partNumber=5&uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt HTTP/1.1" 200 - "-" "-"
172.20.32.1 - - [25/Feb/2015:10:42:32 +0300] "POST /putfile?uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt HTTP/1.1" 200 279 "-" "-"
172.20.32.1 - - [25/Feb/2015:10:43:04 +0300] "DELETE /putfile HTTP/1.1" 400 81 "-" "-"
172.20.32.1 - - [25/Feb/2015:10:43:04 +0300] "DELETE /putfile HTTP/1.1" 204 - "-" "-"
172.20.32.1 - - [25/Feb/2015:10:41:02 +0300] "POST /putfile?uploads HTTP/1.1" 200 242 "-" "-"
151.227.152.48 - - [02/Jul/2014:14:35:56 +0100] "GET /img/Customers/Absolute-Steel-Framing.gif HTTP/1.1" 200 10123 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
159.226.202.17 - - [31/Aug/2010:23:45:30 +0100] "GET / HTTP/1.1" 403 323 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; iCafeMedia; .NET CLR 2.0.50727; CIBA)"
65.55.3.169 - - [01/Sep/2010:08:03:47 +0100] "GET /robots.txt HTTP/1.1" 403 272 "-" "msnbot/2.0b (+http://search.example.com/msnbot.htm)._"
66.187.104.20 - - [24/Apr/2009:19:15:52 +1100] "GET /misc/arrow-desc.png HTTP/1.1" 404 217
77.35.168.108 - - [28/Apr/2009:10:38:09 +1100] "GET / HTTP/1.1" 200 85
77.35.172.105 - - [28/Apr/2009:12:49:27 +1100] "GET / HTTP/1.1" 304 -
79.137.201.45 - - [02/May/2009:12:17:26 +1100] "GET /robots.txt HTTP/1.0" 404 208
151.21.4.47 - - [17/Feb/2018:16:06:48 +0100] "GET /noindex/css/open-sans.css HTTP/1.1" 200 5081 "http://94.177.222.96/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"
151.21.4.47 - - [17/Feb/2018:16:06:48 +0100] "GET /images/apache_pb.gif HTTP/1.1" 200 2326 "http://94.177.222.96/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"
"""

lines = sample.split('\n')
structured_data = []
for line in lines:
    if line == '':
        continue

    parts = line.split(' ')

    remote_host = parts[0]    # IP
    time_date = parts[3]      # Log datetime
    request_method = parts[5] # GET OR POST..
    request_path = parts[6]   # Requested resource
    http_version = parts[7]
    response_status_code = parts[8]

    structured_dict = {'remote_host': remote_host,
                       'datetime': time_date,
                       'method': request_method,
                       'path': request_path,
                       'http_version': http_version,
                       'response_code': response_status_code
                       }
    structured_data.append(structured_dict)

for dict in structured_data:
    print(dict)

样本输出：

{'remote_host': '::1', 'datetime': '[03/Jan/2018:21:28:49', 'method': '"GET', 'path': '/moodle/course/view.php?id=19', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '83.198.250.175', 'datetime': '[22/Mar/2009:07:40:06', 'method': '"GET', 'path': '/style.css', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '212.31.110.34', 'datetime': '[16/May/2018:12:30:44', 'method': 'safefin.example.com', 'path': '"GET', 'http_version': '/', 'response_code': 'HTTP/1.1"'}
{'remote_host': '151.227.152.48', 'datetime': '[02/Jul/2014:14:35:55', 'method': '"GET', 'path': '/css/main.css', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '109.169.248.247', 'datetime': '[12/Dec/2015:18:25:11', 'method': '"POST', 'path': '/administrator/index.php', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '80.91.33.133', 'datetime': '[17/May/2015:08:05:24', 'method': '"GET', 'path': '/downloads/product_1', 'http_version': 'HTTP/1.1"', 'response_code': '304'}
{'remote_host': '217.168.17.5', 'datetime': '[17/May/2015:08:05:34', 'method': '"GET', 'path': '/downloads/product_1', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '192.168.0.11', 'datetime': '[27/Jun/2016:18:36:14', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '302'}
{'remote_host': '51.68.152.26', 'datetime': '[09/Apr/2019:01:37:30', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '302'}
{'remote_host': '71.169.154.24', 'datetime': '[01/Mar/2015:20:58:55', 'method': '"GET', 'path': '/BarHarborcemeteries/Burns-RichardsonCemeteryimages/general%20view%20(2008).jpg', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '94.90.115.82', 'datetime': '[02/Apr/2012:04:56:17', 'method': '"GET', 'path': '/manager/html', 'http_version': 'HTTP/1.1"', 'response_code': '404'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:42:29', 'method': '"PUT', 'path': '/putfile?partNumber=5&uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:42:32', 'method': '"POST', 'path': '/putfile?uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:43:04', 'method': '"DELETE', 'path': '/putfile', 'http_version': 'HTTP/1.1"', 'response_code': '400'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:43:04', 'method': '"DELETE', 'path': '/putfile', 'http_version': 'HTTP/1.1"', 'response_code': '204'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:41:02', 'method': '"POST', 'path': '/putfile?uploads', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '151.227.152.48', 'datetime': '[02/Jul/2014:14:35:56', 'method': '"GET', 'path': '/img/Customers/Absolute-Steel-Framing.gif', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '159.226.202.17', 'datetime': '[31/Aug/2010:23:45:30', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '403'}
{'remote_host': '65.55.3.169', 'datetime': '[01/Sep/2010:08:03:47', 'method': '"GET', 'path': '/robots.txt', 'http_version': 'HTTP/1.1"', 'response_code': '403'}
{'remote_host': '66.187.104.20', 'datetime': '[24/Apr/2009:19:15:52', 'method': '"GET', 'path': '/misc/arrow-desc.png', 'http_version': 'HTTP/1.1"', 'response_code': '404'}
{'remote_host': '77.35.168.108', 'datetime': '[28/Apr/2009:10:38:09', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '77.35.172.105', 'datetime': '[28/Apr/2009:12:49:27', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '304'}
{'remote_host': '79.137.201.45', 'datetime': '[02/May/2009:12:17:26', 'method': '"GET', 'path': '/robots.txt', 'http_version': 'HTTP/1.0"', 'response_code': '404'}
{'remote_host': '151.21.4.47', 'datetime': '[17/Feb/2018:16:06:48', 'method': '"GET', 'path': '/noindex/css/open-sans.css', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '151.21.4.47', 'datetime': '[17/Feb/2018:16:06:48', 'method': '"GET', 'path': '/images/apache_pb.gif', 'http_version': 'HTTP/1.1"', 'response_code': '200'}

我从不同的地方收集了样本，大多数格式看起来都与上面的样本相同。

如果您想使用正则表达式解析日志，以下是一些可能有用的方法：

捕获IP地址有点困难。如果要检查它是否是有效的ip地址，请尝试。否则，如果您想要4组最多3位数的数字，它们之间用点隔开：

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

对于datetime，看起来您可以抓取第一个出现在方括号中的字符

\[([^\]]+)\]

对于方法、路径和响应，看起来您可以获取第一次出现的由引号包围的字符，然后是紧跟其后的数字

"([^"]+)"\s+(\d{1,3})

因为这里有多个匹配项，所以您可以使用组来获取单个项。使用这个正则表达式，您将得到第一个组，只需去掉“GET、POST、DELETE等”，剩下的就是路径

使用python的

re

库，将每个正则表达式应用到输入中的一行，看看得到了什么

#!/usr/bin/env python
import re

bad_ip_regex = re.compile("\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}")
datetime_regex = re.compile("\[([^\]]+)\]")
other_regex = re.compile('"([^"]+)"\s+(\d{1,3})')
with open("input.log", "r") as f:
  for line in f:
    item = {}

    # attempt to grab IP
    ip = bad_ip_regex.search(line)
    if ip:
      item["remote_host"] = ip.group(0)
    else:
      # no ip, just skip?
      continue

    # attempt to grab datetime
    datetime = datetime_regex.search(line)
    if datetime:
      item["datetime"] = datetime.group(1)
    else:
      continue

    # attempt to grab other
    other = other_regex.search(line)
    if other:
      item["method"] = other.group(1).split()[0]
      item["path"] = other.group(1).split()[1]
      item["response"] = other.group(2)
    else:
      continue

    print(item)

因为不能保证这些项的顺序，所以尝试使用正则表达式一次获取所有字段是没有意义的。只需在每行上一次尝试一个。

嗯。。。你的说明有点误导，但幸运的是，我不久前做过类似的事情，所以我只是修改了一些你可以使用的脏代码。请记住，在Python中，字典不会默认以任何特定顺序显示。

但是下面的代码应该满足您的需要并使用一个正则表达式

>>> sample = '''
::1 - - [03/Jan/2018:21:28:49 +0100] "GET /moodle/course/view.php?id=19 HTTP/1.1" 200 78325 "http://localhost/moodle/login/index.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
83.198.250.175 - - [22/Mar/2009:07:40:06 +0100] "GET /style.css HTTP/1.1" 200 1692 "http://www.example.org/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Wanadoo 6.7; Orange 8.0)" "-"
212.31.110.34 0.597 - [16/May/2018:12:30:44 +0000] safefin.example.com "GET / HTTP/1.1" 200 18193 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36"
151.227.152.48 - - [02/Jul/2014:14:35:55 +0100] "GET /css/main.css HTTP/1.1" 200 4658 "http://example.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] "POST /administrator/index.php HTTP/1.1" 200 4494 "http://example.net/administrator/" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
80.91.33.133 - - [17/May/2015:08:05:24 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.17)"
217.168.17.5 - - [17/May/2015:08:05:34 +0000] "GET /downloads/product_1 HTTP/1.1" 200 490 "-" "Debian APT-HTTP/1.3 (0.8.10.3)"
192.168.0.11 - - [27/Jun/2016:18:36:14 -0500] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Linux; Android 5.1.1; SM-N910T Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.81 Mobile Safari/537.36"
51.68.152.26 - - [09/Apr/2019:01:37:30 +0400] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
71.169.154.24 - - [01/Mar/2015:20:58:55 -0500] "GET /BarHarborcemeteries/Burns-RichardsonCemeteryimages/general%20view%20(2008).jpg HTTP/1.1" 200 165457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
94.90.115.82 - - [02/Apr/2012:04:56:17 +0900] "GET /manager/html HTTP/1.1" 404 77 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
172.20.32.1 - - [25/Feb/2015:10:42:29 +0300] "PUT /putfile?partNumber=5&uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt HTTP/1.1" 200 - "-" "-"
172.20.32.1 - - [25/Feb/2015:10:42:32 +0300] "POST /putfile?uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt HTTP/1.1" 200 279 "-" "-"
172.20.32.1 - - [25/Feb/2015:10:43:04 +0300] "DELETE /putfile HTTP/1.1" 400 81 "-" "-"
172.20.32.1 - - [25/Feb/2015:10:43:04 +0300] "DELETE /putfile HTTP/1.1" 204 - "-" "-"
172.20.32.1 - - [25/Feb/2015:10:41:02 +0300] "POST /putfile?uploads HTTP/1.1" 200 242 "-" "-"
151.227.152.48 - - [02/Jul/2014:14:35:56 +0100] "GET /img/Customers/Absolute-Steel-Framing.gif HTTP/1.1" 200 10123 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
159.226.202.17 - - [31/Aug/2010:23:45:30 +0100] "GET / HTTP/1.1" 403 323 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; iCafeMedia; .NET CLR 2.0.50727; CIBA)"
65.55.3.169 - - [01/Sep/2010:08:03:47 +0100] "GET /robots.txt HTTP/1.1" 403 272 "-" "msnbot/2.0b (+http://search.example.com/msnbot.htm)._"
66.187.104.20 - - [24/Apr/2009:19:15:52 +1100] "GET /misc/arrow-desc.png HTTP/1.1" 404 217
77.35.168.108 - - [28/Apr/2009:10:38:09 +1100] "GET / HTTP/1.1" 200 85
77.35.172.105 - - [28/Apr/2009:12:49:27 +1100] "GET / HTTP/1.1" 304 -
79.137.201.45 - - [02/May/2009:12:17:26 +1100] "GET /robots.txt HTTP/1.0" 404 208
151.21.4.47 - - [17/Feb/2018:16:06:48 +0100] "GET /noindex/css/open-sans.css HTTP/1.1" 200 5081 "http://94.177.222.96/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"
151.21.4.47 - - [17/Feb/2018:16:06:48 +0100] "GET /images/apache_pb.gif HTTP/1.1" 200 2326 "http://94.177.222.96/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"'''




>>> def text_to_dict(string):
        import re
        dict_array = []
        found_items = re.findall('(?m)^((?:[\d]{1,3}\.){3}[\d]{1,3}|[\d]*[:]*[\d]*)[\S\ ]*?\[([\S\ ]*?)\][\S\ ]*?\"([A-Z]+)[\S\ ]*?(/(?=[\s]+)|/[\s]*[\S]+)[\S\ ]*?(HTTP[\S]*?)\"[\S\ ]*?([\d]{3}(?=\s|$))', string)
        for i in range(len(found_items)):
            try:
                dict = {"remote_host":found_items[i][0], "datetime":found_items[i][1], "method":found_items[i][2], "path":found_items[i][3],"http_version":found_items[i][4], "response_code":found_items[i][5]}
                dict_array.append(dict)
            except:
                print('\n\n================Failed')
                print(found_items[i])
        return dict_array




>>> found_items = text_to_dict(sample)



>>> for elements in found_items:
        print(elements)





 #OUTPUT
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '03/Jan/2018:21:28:49 +0100', 'path': '/moodle/course/view.php?id=19', 'remote_host': '::1', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '22/Mar/2009:07:40:06 +0100', 'path': '/style.css', 'remote_host': '83.198.250.175', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '16/May/2018:12:30:44 +0000', 'path': '/', 'remote_host': '212.31.110.34', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '02/Jul/2014:14:35:55 +0100', 'path': '/css/main.css', 'remote_host': '151.227.152.48', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '12/Dec/2015:18:25:11 +0100', 'path': '/administrator/index.php', 'remote_host': '109.169.248.247', 'method': 'POST'}
 {'http_version': 'HTTP/1.1', 'response_code': '304', 'datetime': '17/May/2015:08:05:24 +0000', 'path': '/downloads/product_1', 'remote_host': '80.91.33.133', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '17/May/2015:08:05:34 +0000', 'path': '/downloads/product_1', 'remote_host': '217.168.17.5', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '302', 'datetime': '27/Jun/2016:18:36:14 -0500', 'path': '/', 'remote_host': '192.168.0.11', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '302', 'datetime': '09/Apr/2019:01:37:30 +0400', 'path': '/', 'remote_host': '51.68.152.26', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '01/Mar/2015:20:58:55 -0500', 'path': '/BarHarborcemeteries/Burns-RichardsonCemeteryimages/general%20view%20(2008).jpg', 'remote_host': '71.169.154.24', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '404', 'datetime': '02/Apr/2012:04:56:17 +0900', 'path': '/manager/html', 'remote_host': '94.90.115.82', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '25/Feb/2015:10:42:29 +0300', 'path': '/putfile?partNumber=5&uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt', 'remote_host': '172.20.32.1', 'method': 'PUT'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '25/Feb/2015:10:42:32 +0300', 'path': '/putfile?uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt', 'remote_host': '172.20.32.1', 'method': 'POST'}
 {'http_version': 'HTTP/1.1', 'response_code': '400', 'datetime': '25/Feb/2015:10:43:04 +0300', 'path': '/putfile', 'remote_host': '172.20.32.1', 'method': 'DELETE'}
 {'http_version': 'HTTP/1.1', 'response_code': '204', 'datetime': '25/Feb/2015:10:43:04 +0300', 'path': '/putfile', 'remote_host': '172.20.32.1', 'method': 'DELETE'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '25/Feb/2015:10:41:02 +0300', 'path': '/putfile?uploads', 'remote_host': '172.20.32.1', 'method': 'POST'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '02/Jul/2014:14:35:56 +0100', 'path': '/img/Customers/Absolute-Steel-Framing.gif', 'remote_host': '151.227.152.48', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '403', 'datetime': '31/Aug/2010:23:45:30 +0100', 'path': '/', 'remote_host': '159.226.202.17', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '403', 'datetime': '01/Sep/2010:08:03:47 +0100', 'path': '/robots.txt', 'remote_host': '65.55.3.169', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '404', 'datetime': '24/Apr/2009:19:15:52 +1100', 'path': '/misc/arrow-desc.png', 'remote_host': '66.187.104.20', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '28/Apr/2009:10:38:09 +1100', 'path': '/', 'remote_host': '77.35.168.108', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '304', 'datetime': '28/Apr/2009:12:49:27 +1100', 'path': '/', 'remote_host': '77.35.172.105', 'method': 'GET'}
 {'http_version': 'HTTP/1.0', 'response_code': '404', 'datetime': '02/May/2009:12:17:26 +1100', 'path': '/robots.txt', 'remote_host': '79.137.201.45', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '17/Feb/2018:16:06:48 +0100', 'path': '/noindex/css/open-sans.css', 'remote_host': '151.21.4.47', 'method': 'GET'}
 {'http_version': 'HTTP/1.1', 'response_code': '200', 'datetime': '17/Feb/2018:16:06:48 +0100', 'path': '/images/apache_pb.gif', 'remote_host': '151.21.4.47', 'method': 'GET'}

“因为日志格式并不总是相同的”。那么日志格式是什么样的呢？如果你不知道它们应该是什么样子，那么很难解析它们。如果您不知道该格式，那么您可以做的最好是上面的内容，但要进行错误检查（即，在读取远程主机后，确保它是有效的IP，确保datetime是有效的，等等）。我从不同的地方收集了样本，大多数格式看起来与上面的样本相同。只要每个源使用相同的格式，您就可以解析来自每个源的日志，并将它们转换成与上面相同的格式，然后将所有数据连接在一起。如果您尝试一次性完成所有操作，这个问题会变得更加困难，因为您必须在尝试解析行时尝试找出行的格式。我认为使用正则表达式解析更好，但我不擅长正则表达式：（正如您可以看到输出中的第三行不正确，因为数据的索引不相同