使用awk对nginx access.log进行统计分析

        nginx可以配置访问日志，如果我们要对日志文件进行统计分析，在linux环境下可以借助awk命令完成。

        日志格式配置如下所示：

log_format access_json '{"@timestamp":"$time_iso8601",'
        '"host":"$server_addr",'
        '"clientip":"$remote_addr",'
        '"size":$body_bytes_sent,'
        '"responsetime":$request_time,'
        '"upstreamtime":"$upstream_response_time",'
        '"upstreamhost":"$upstream_addr",'
        '"login_user":"$cookie_fine_login_users",'
        '"http_host":"$host",'
        '"url":"$uri",'
        '"request":"$request",'
        '"method":"$request_method",'
        '"domain":"$host",'
        '"xff":"$http_x_forwarded_for",'
        '"referer":"$http_referer",'
        '"user_agent":"$http_user_agent",'
        '"status":"$status"}';
    access_log  /usr/local/openresty/nginx/logs/access.log  access_json;

访问日志示例

{"@timestamp":"2024-05-21T19:11:31+08:00","host":"10.1.1.1","clientip":"10.2.6.1","size":10600,"responsetime":0.523,
"upstreamtime":"0.522","upstreamhost":"10.1.11.11:8080","login_user":"-","http_host":"abc.com.cn",
"url":"/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher",
"request":"GET /irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher?BI_COMMAND-BI_COMMAND_TYPE=UPDATE&BI_COMMAND-TARGET_DATA_PROVIDER_REF=DP_1&BI_COMMAND-TARGET_DIALOG_REF=SELECTOR&PAGE_ID=1_T0k1OUFpT0VyQ1d3WUVVa0dfa0g5QS0tWWNJejk2TmU1ZU03SzNiR2NqNnFkUS0t&REQUEST_ID=195 HTTP/1.1",
"method":"GET","domain":"abc.com.cn","xff":"-","referer":"http://abc.com.cn/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher?BI_COMMAND-BI_COMMAND_TYPE=UPDATE&PAGE_ID=1_T0k1OUFpT0VyQ1d3WUVVa0dfa0g5QS0tWWNJejk2TmU1ZU03SzNiR2NqNnFkUS0t&REQUEST_ID=7",
"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0","status":"200"}
 
{"@timestamp":"2024-05-21T19:11:37+08:00","host":"10.1.1.1","clientip":"10.3.9.1","size":405,"responsetime":0.035,
"upstreamtime":"0.034","upstreamhost":"10.1.11.11:8080","login_user":"-","http_host":"abc.com.cn",
"url":"/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher","request":"POST /irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher HTTP/1.1",
"method":"POST","domain":"abc.com.cn","xff":"-","referer":"http://abc.com.cn/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher?BI_COMMAND-BI_COMMAND_TYPE=UPDATE&PAGE_ID=2_a3JxVW1LM09oWWNSOTVnWGpOMmRWdy0tKnZzbDZkQXV4NVdvVFlIQ0dvSXFYdy0t&REQUEST_ID=1",
"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0","status":"200"}

1.统计调用uri次数最多的前20个

awk -F',"' '{s[$10]+=1}END{for(i in s){print substr(i,7),s[i]}}' 
access.log | sort -r -n -k2 | head -n 20

2.统计size 大小最大的前20个

awk -F',"' '{s[substr($4,7)]+=1}END{for(i in s){print i,s[i]}}' 
access.log | sort -r -n -k1 | head -n 20

3.统计特定url size大小最大的前n个

根据1中的命令获取的目标uri，比如：/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher

将指定uri内容导出到临时文件t1.txt

grep '"url":"/irj/servlet/prt/portal/prtroot/com.sap.ip.bi.web.portal.integration.launcher' access.log > t1.txt

统计size>1000的调用次数之和

awk -F',"' '{s[substr($4,7)]+=1}END{for(i in s){if((i+0)>=1000)print i,s[i]}}' t1.txt | sort  -n -r -k1 | awk '{sum+=$2}END{print sum}'