NGINX记录awk按IP地址查找带宽

我试图在nginx访问日志中查找最常见的发出请求的ip地址所使用的带宽。这就是我从以下几点开始的：

$cat/path/to/access.log | awk'{print$1}'| sort | uniq-c | sort-n | tail

（$1是ip地址，而请求字节是$10）-这将输出：

# of requests | IP Address
1220 xxx.xxx.xxx.xxx
1347 xxx.xxx.xxx.xxx
1420 xxx.xxx.xxx.xxx
2104 xxx.xxx.xxx.xxx
etc...    

我试图完成的是确定每个地址请求的带宽。例如：

# of requests | IP Address | total bytes requested (unique to ip)
1220 xxx.xxx.xxx.xxx 45626026
1347 xxx.xxx.xxx.xxx 49565157
1420 xxx.xxx.xxx.xxx 56689122
2104 xxx.xxx.xxx.xxx 76665299
etc...

我的限制不是太有限。因此，尽管如此，如果可能的解决方案是在最终查询时使用多个命令进行解析（即通过ip查找总带宽），那么就这样吧。谢谢你的帮助

使用单个GNUawk解决方案：

示例

access.log

用于演示：

127.0.0.1 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 111 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 171 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 152 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1502 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.2 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 14111 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.2 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 1414 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.2 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 1522 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.2 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1332 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.3 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) G$
127.0.0.1 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 141 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.1 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 1041 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.3 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 1529 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.1 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1026 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.1 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) G$
127.0.0.3 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 1414 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.1 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 13341 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.3 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 172 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.3 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1502 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.3 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) G$

---

工作：

awk 'BEGIN{ PROCINFO["sorted_in"]="@val_num_desc" }
     { a[$1]++; b[$1]+=$10 }
     END{ 
         for(i in a) { if(++c>10) break; print i,b[i] } 
     }' /path/to/access.log

• PROCINFO[“sorted_in”]=“@val_num_desc”

  -比较数组值，按IP地址频率降序排序

• 如果（++c>10）

  -确保只迭代前10项，这是对tail命令的模拟（获取最后10行）。循环从最频繁的IP地址开始

输出：

127.0.0.1 65437
127.0.0.3 52569
127.0.0.2 18379

使用单个GNUawk解决方案：

示例

access.log

用于演示：

127.0.0.1 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 111 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 171 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 152 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1502 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.2 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 14111 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.2 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 1414 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.2 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 1522 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.2 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1332 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.3 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) G$
127.0.0.1 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 141 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.1 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 1041 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.3 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 1529 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.1 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1026 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.1 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) G$
127.0.0.3 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 1414 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.1 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 13341 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.3 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 172 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.3 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1502 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.3 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) G$

---

工作：

awk 'BEGIN{ PROCINFO["sorted_in"]="@val_num_desc" }
     { a[$1]++; b[$1]+=$10 }
     END{ 
         for(i in a) { if(++c>10) break; print i,b[i] } 
     }' /path/to/access.log

• PROCINFO[“sorted_in”]=“@val_num_desc”

  -比较数组值，按IP地址频率降序排序

• 如果（++c>10）

  -确保只迭代前10项，这是对tail命令的模拟（获取最后10行）。循环从最频繁的IP地址开始

输出：

127.0.0.1 65437
127.0.0.3 52569
127.0.0.2 18379

您的问题包括简明的可测试样本输入和预期输出。使用编辑器

{}

按钮格式化示例和代码。您的问题包括简明的可测试样本输入和预期输出。使用编辑器

{}

按钮格式化您的示例和代码。我可以问一下为什么在那里有if（++c>10）条件吗？@JFS31，是的，

if（++c>10）

确保只迭代前10项，这是模拟

tail

命令（获取最后10行）循环从最频繁的IP地址开始。我可以问一下，为什么会有if（++c>10）条件吗？@JFS31，是的，

if（++c>10）

-确保只迭代前10项，这是模拟

tail

命令（获取最后10行）循环从最频繁的IP地址开始