ELK之Logstash解析json报错 JSON parse error：Unrecognized character escape ‘x‘_json parse error, original data now in message fie

一、报错分析

原始报错信息：

JSON parse error, original data now in message field {:message=>"Unrecognized character escape 'x' (code 120)\n at [Source: (String)\"{\"@timestamp\":\"2022-06-29T10:02:00+08:00\",\"@source\":\"172.1.2.13\",\"hostname\":\"nb001\",\"ip\":\"121.8.185.98\",\"xff\":\"-\",\"client\":\"121.8.185.98\",\"remote_user\":\"-\",\"request_method\":\"POST\",\"request_body\": \"{\\x22equipmentId\\x22:1501805378853269505}\",\"scheme\":\"https\",\"domain\":\"gw.xxx.com\",\"referer\":\"-\",\"request\":\"/api/equipment/findWaitPlan\",\"args\":\"-\",\"size\":69,\"status\": 200,\"responsetime\":0.075,\"upstreamtime\":\"0.076\",\"upstreamaddr\":\"172.1.2.12:31149\",\"upstream_stat\"[truncated 205 chars]; line: 1, column: 204]", :exception=>LogStash::Json::ParserError, :data=>"{\"@timestamp\":\"2022-06-29T10:02:00+08:00\",\"@source\":\"172.1.2.13\",\"hostname\":\"nb001\",\"ip\":\"121.8.185.98\",\"xff\":\"-\",\"client\":\"121.8.185.98\",\"remote_user\":\"-\",\"request_method\":\"POST\",\"request_body\": \"{\\x22equipmentId\\x22:1501805378853269505}\",\"scheme\":\"https\",\"domain\":\"gw.xxx.com\",\"referer\":\"-\",\"request\":\"/api/equipment/findWaitPlan\",\"args\":\"-\",\"size\":69,\"status\": 200,\"responsetime\":0.075,\"upstreamtime\":\"0.076\",\"upstreamaddr\":\"172.1.2.12:31149\",\"upstream_status\": \"200\",\"http_user_agent\":\"Apache-HttpClient/4.5.6 (Java/1.8.0_131)\",\"http_host\":\"gw.xxx.com\",\"url\":\"/api/equipment/findWaitPlan\",\"connection_requests\": 4,\"https\":\"on\"}"}

1
2

上述有说解析的nginx日志的第一行204个字符（"[truncated 205 chars]; line: 1, column: 204]",）无法解析：
发现就是json中的\"request_body\": \"{\\x22equipmentId\\x22:1501805378853269505}\" request_body的只在转义的时候变成了\\x22。

二、解决办法

在生成json格式的nginx日志上明确指定escape=json

 log_format json escape=json '{"@timestamp":"$time_iso8601",'
                        '"@source":"$server_addr",'
                        '"hostname":"$hostname",'
                        '"ip":"$remote_addr",'
                        '"xff":"$http_x_forwarded_for",'
                        '"client":"$remote_addr",'
                        '"remote_user":"$remote_user",'
                        '"request_method":"$request_method",'
                        '"request_body": "$request_body",'
                        '"scheme":"$scheme",'
                        '"domain":"$server_name",'
                        '"referer":"$http_referer",'
                        '"request":"$request_uri",'
                        '"args":"$args",'
                        '"size":$body_bytes_sent,'
                        '"status": $status,'
                        '"responsetime":$request_time,'
                        '"upstreamtime":"$upstream_response_time",'
                        '"upstreamaddr":"$upstream_addr",'
                        '"upstream_status": "$upstream_status",'
                        '"http_user_agent":"$http_user_agent",'
                        '"http_host":"$host",'
                        '"url":"$uri",'
                        '"connection_requests": $connection_requests,'
                        '"https":"$https"'
                        '}';

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

三、改后效果

再次查看结果：
发现logstash已经解析成功了：

{
                   "args" => "",
           "upstreamaddr" => "172.1.2.12:31149",
    "connection_requests" => 1,
         "request_method" => "POST",
        "upstream_status" => "200",
                    "log" => {
        "offset" => 349425307,
          "file" => {
            "path" => "/usr/local/nginx/logs/access.log"
        }
    },
                   "tags" => [
        [0] "beats_input_codec_json_applied",
        [1] "_grokparsefailure"
    ],
            "remote_user" => "",
              "container" => {
        "id" => "access.log"
    },
                    "xff" => "",
                  "agent" => {
             "version" => "7.17.4",
        "ephemeral_id" => "4e4e1f17-99bc-4ba1-b431-df953c9594d8",
                "type" => "filebeat",
                  "id" => "3f299217-a27a-4fb0-ac7d-f80c561d6f2a",
                "name" => "nb001",
            "hostname" => "nb001"
    },
               "@version" => "1",
                 "client" => "218.85.130.106",
        "http_user_agent" => "",
                  "geoip" => {
           "city_name" => "Xiamen",
        "country_name" => "China",
            "location" => {
            "lat" => 24.4798,
            "lon" => 118.0819
        },
         "region_name" => "Fujian",
         "coordinates" => [
            [0] 118.0819,
            [1] 24.4798
        ],
                  "ip" => "218.85.130.106"
    },
             "@timestamp" => 2022-06-29T10:36:02.940Z,
                  "https" => "on",
                 "status" => 200,
                "referer" => "",
                "@source" => "172.1.2.13",
           "request_body" => "{\"experimentId\":\"d03ac7b7548a49828d207431a48ef916\",\"subExperimentId\":\"f95dd19af6e34904b32ef3cc76124ba4\",\"resourceId\":\"1413404267903803394\",\"orgId\":\"616d1ccee4b0ab7641c47a2e\",\"userId\":\"9ea996f86eb3459688390789bc151974\"}",
                "request" => "/api/open/platform/heartbeats",
                   "size" => 112,
           "responsetime" => 0.068,
           "upstreamtime" => "0.067",
                    "url" => "/api/open/platform/heartbeats",
                    "ecs" => {
        "version" => "1.12.0"
    },
                 "scheme" => "https",
                 "domain" => "gw.xxx.com",
              "http_host" => "gw.xxx.com",
                     "ip" => "218.85.130.106",
               "hostname" => "nb001",
                  "input" => {
        "type" => "filestream"
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

END