查找包含字符串的行&；使用shell脚本将值回显到新行

我想在bash脚本中找到解决方案： 我有原始输出日志。每行以日期开头，例如4月10日11:17:35

我想循环遍历每个日志项，并找到包含字符串coderbyte heroku/router的行。对于其中的每一个，将request_id值回显到一个新行，如果fwd键的值为MASKED，则在该行末尾添加一个[M]，并在其前面加一个空格

输出日志

Apr 10 11:17:35 coderbyte app/web.3: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?shareLinkId=69dff0hba0nv HTTP/1.1" 200 148 "https://coderbyte.com" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0
Apr 10 11:17:35 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?key=s2fwad2Es2" host=coderbyte.com request_id=b19a87a1-1bbb-4e67-b207-bd9f23d46afa fwd="108.31.000.000" dyno=web.3 connect=0ms service=92ms status=200 bytes=3194 protocol=https

Apr 10 11:17:35 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q" host=coderbyte.com request_id=910b07d1-3f71-4347-a1a7-bfa20384ef65 fwd="108.31.000.000" dyno=web.2 connect=1ms service=17ms status=200 bytes=4435 protocol=https

Apr 10 11:17:35 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q" host=coderbyte.com request_id=097bf65e-e189-4f9f-9dfb-4758cff411b2 fwd="108.31.000.000" dyno=web.3 connect=1ms service=10ms status=200 bytes=4435 protocol=https

Apr 10 11:17:35 coderbyte app/web.2: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?key=s2fwad2Es2 HTTP/1.1" 200 4263 "https://coderbyte.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Apr 10 11:17:35 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=4eiramcmayu0" host=coderbyte.com request_id=d48278c2-5731-464e-be38-ab9ad84ac4a8 fwd="108.31.000.000" dyno=web.4 connect=1ms service=7ms status=200 bytes=3194 protocol=https

Apr 10 11:17:35 coderbyte app/web.3: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q HTTP/1.1" 200 4263 "https://coderbyte.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Apr 10 11:17:35 coderbyte app/web.3: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q HTTP/1.1" 200 4263 "https://coderbyte.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Apr 10 11:17:36 coderbyte app/web.4: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?shareLinkId=4eiramcmayu0 HTTP/1.1" 200 3023 "https://coderbyte.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Apr 10 11:17:36 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q" host=coderbyte.com request_id=8bb2413c-3c67-4180-8091-000313b8d9ca fwd="MASKED" dyno=web.3 connect=1ms service=32ms status=200 bytes=4435 protocol=https

Apr 10 11:17:36 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q" host=coderbyte.com request_id=10f93da3-2753-48a3-9485-857a93d8a88a fwd="MASKED" dyno=web.3 connect=1ms service=37ms status=200 bytes=4435 protocol=https

这是我的密码

#!/bin/bash
curl -s https://coderbyte.com/api/challenges/logs/web-logs-raw -O > /dev/null
#cat web-logs-raw

grep -F "coderbyte heroku/router" web-logs-raw >> test
cat test

到目前为止，这可以过滤日志并找到包含字符串coderbyte heroku/router的行。但是如何将request_id值回显到新行，如果fwd key的值为MASKED，则在该行末尾添加一个[M]，并在其前面加一个空格

输出应该是这样的

b19a87a1-1bbb-000-00000
b19a87a1-1bbb-000-11111
8bb2413c-3c67-4180-22222 [M]
10f93da3-2753-48a3-33333 [M]

一艘班轮：

 awk '/coderbyte heroku\/router/ { split($10,map,"=");id=map[2];split($11,map1,"\"");print map1[2]=="MASKED"?id" [M]":id }' web-logs-raw

说明：

awk '/coderbyte heroku\/router/ { # Search for lines with required text
          split($10,map,"="); # Split the 10th space delimited field into the array map using "=" as the field separator
          id=map[2]; # Set the variable id to the the second index of the map array
          split($11,map1,"\""); # Split the 11th field into the array map1 using " as the field separator (this is the masked variable)
          print map1[2]=="MASKED"?id" [M]":id # If the masked entry is MASKED, print "[M]" and then the id otherwise just print the id
     }' web-logs-raw  

输出：

b19a87a1-1bbb-4e67-b207-bd9f23d46afa
910b07d1-3f71-4347-a1a7-bfa20384ef65
097bf65e-e189-4f9f-9dfb-4758cff411b2
d48278c2-5731-464e-be38-ab9ad84ac4a8
8bb2413c-3c67-4180-8091-000313b8d9ca [M]
10f93da3-2753-48a3-9485-857a93d8a88a [M]

---

另一种选择是使用模式并匹配

coderbyte-heroku\/router

，并在捕获组中捕获请求id和

fwd=“MASKED”

测试组2是否不为空。如果不是，则写请求id，后跟

[M]

，否则只写请求id

使用带有第三个参数的

gnu awk

来捕获组（如@所述）

---

示例输出的值如

b19a87a1-1bbb-000-00000

不在输出日志中，但其

b19a87a1-1bbb-

的一部分在日志中。它应该是完全相同的值吗？抱歉，这是打字错误。它应该是“b19a87a1-1bbb-4e67-b207-bd9f23d46afa”、“910b07d1-3f71-4347-a1a7-bfa20384ef65”、“097bf65e-e189-4f9f-9dfb-4758CFF41B2”检索某些内容并将其写入

/dev/null

的行有什么意义？@SimplyAnks如果给出的任何答案都能帮助您解决问题，请看。只需提到它需要

gnu awk

@anubhava好的，我会的，我是新来的。POSIX awk中新的

match

函数没有捕获组的第三个参数，不像

gnu awk

感谢Raman，它按预期工作！！伟大的如果你能投票并接受答案，那就太好了。

awk '
match($0, /^.*?\<coderbyte heroku\/router\>.* request_id=(\S+) (fwd="MASKED)?/, m) {
  print m[2]? m[1] " [M]": m[1]
}
' web-logs-raw

b19a87a1-1bbb-4e67-b207-bd9f23d46afa
910b07d1-3f71-4347-a1a7-bfa20384ef65
097bf65e-e189-4f9f-9dfb-4758cff411b2
d48278c2-5731-464e-be38-ab9ad84ac4a8
8bb2413c-3c67-4180-8091-000313b8d9ca [M]
10f93da3-2753-48a3-9485-857a93d8a88a [M]