深入理解Istio流量管理的熔断配置_istio reset reason: protocol error

环境准备

httpbin 服务

httpbin 是一个使用 Python + Flask 编写的 HTTP 请求响应服务。

该服务主要用于测试 HTTP 库，你可以向他发送请求，然后他会按照指定的规则将你的请求返回。

部署 httpbin 服务：

kubectl apply -f samples/httpbin/httpbin.yaml
1

fortio 服务

Fortio是一个微服务（http, grpc)负载测试库，命令行工具，高级回显服务器和web UI在go (golang)。Fortio允许指定一组每秒查询负载，并记录延迟直方图和其他有用的统计数据。

部署 fortio 服务：

kubectl apply -f samples/httpbin/sample-client/fortio-deploy.yaml
1

httpbin 服务作为接收请求的服务端， fortio 服务作为发送请求的客户端。

熔断配置

创建目标规则，访问httpbin 服务时应用熔断配置：

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13

在 fortio 服务中向 httpbin 服务的发出并发请求：

export FORTIO_POD=$(kubectl get pod -l app=fortio -n istio-demo -o jsonpath={.items..metadata.name})
kubectl exec -it "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 5 -qps 0 -n 5 -loglevel Warning http://httpbin:8000/get
1
2

返回结果如下：

I0725 11:43:55.895950 4039039 request.go:621] Throttling request took 1.130727121s, request: GET:https://172.24.28.71:6443/apis/ceph.rook.io/v1?timeout=32s
03:43:58 I logger.go:127> Log level is now 3 Warning (was 2 Info)
Fortio 1.17.1 running at 0 queries per second, 104->104 procs, for 5 calls: http://httpbin:8000/get
Starting at max qps with 5 thread(s) [gomax 104] for exactly 5 calls (1 per thread + 0)
03:43:58 W http_client.go:806> [3] Non ok http code 503 (HTTP/1.1 503)
03:43:58 W http_client.go:806> [1] Non ok http code 503 (HTTP/1.1 503)
03:43:58 W http_client.go:806> [2] Non ok http code 503 (HTTP/1.1 503)
Ended after 5.398808ms : 5 calls. qps=926.13
Aggregated Function Time : count 5 avg 0.0035955544 +/- 0.001035 min 0.002619591 max 0.005337051 sum 0.017977772
# range, mid point, percentile, count
>= 0.00261959 <= 0.003 , 0.0028098 , 60.00, 3
> 0.004 <= 0.005 , 0.0045 , 80.00, 1
> 0.005 <= 0.00533705 , 0.00516853 , 100.00, 1
# target 50% 0.0029049
# target 75% 0.00475
# target 90% 0.00516853
# target 99% 0.0053202
# target 99.9% 0.00533537
Sockets used: 5 (for perfect keepalive, would be 5)
Jitter: false
Code 200 : 2 (40.0 %)
Code 503 : 3 (60.0 %)
Response Header Sizes : count 5 avg 92 +/- 112.7 min 0 max 230 sum 460
Response Body/Total Sizes : count 5 avg 419.8 +/- 177.3 min 275 max 637 sum 2099
All done 5 calls (plus 0 warmup) 3.596 ms avg, 926.1 qps
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

可以看到，在发送的5个请求中，有3个请求返回 503 。

查看Envoy日志

执行下面的命令，查看fortio 服务的Envoy日志：

kubectl logs -l app=fortio -c istio-proxy
1

可以看到fortio服务对httpbin服务的调用的日志：

{
     "authority": "httpbin:8000",
     "bytes_received": 0,
     "bytes_sent": 81,
     "connection_termination_details": null,
     "downstream_local_address": "172.24.146.239:8000",
     "downstream_remote_address": "172.24.158.166:55030",
     "duration": 1,
     "method": "GET",
     "path": "/get",
     "protocol": "HTTP/1.1",
     "request_id": "3aa81a10-cddf-9b3b-b704-c1099b05c6fb",
     "requested_server_name": null,
     "response_code": 503,
     "response_code_details": "via_upstream",
     "response_flags": "-",
     "route_name": "allow_any",
     "start_time": "2022-07-22T09:19:04.465Z",
     "upstream_cluster": "PassthroughCluster",
     "upstream_host": "172.24.146.239:8000",
     "upstream_local_address": "172.24.158.166:55042",
     "upstream_service_time": "1",
     "upstream_transport_failure_reason": null,
     "user_agent": "fortio.org/fortio-1.17.1",
     "x_forwarded_for": null
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

其中，response_flags为-，没有任何异常。

执行下面的命令，查看httpbin 服务的Envoy日志：

kubectl logs -l app=httpbin -c istio-proxy
1

可以看到httpbin服务被fortio服务调用的Envoy日志：

{
     "authority": "httpbin:8000",
     "bytes_received": 0,
     "bytes_sent": 81,
     "connection_termination_details": null,
     "downstream_local_address": "172.24.158.96:80",
     "downstream_remote_address": "172.24.158.166:55042",
     "duration": 0,
     "method": "GET",
     "path": "/get",
     "protocol": "HTTP/1.1",
     "request_id": "3aa81a10-cddf-9b3b-b704-c1099b05c6fb",
     "requested_server_name": null,
     "response_code": 503,
     "response_code_details": "upstream_reset_before_response_started{overflow}",
     "response_flags": "UO",
     "route_name": "default",
     "start_time": "2022-07-22T09:19:04.466Z",
     "upstream_cluster": "inbound|80||",
     "upstream_host": null,
     "upstream_local_address": null,
     "upstream_service_time": null,
     "upstream_transport_failure_reason": null,
     "user_agent": "fortio.org/fortio-1.17.1",
     "x_forwarded_for": null
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

其中，response_flags为UO，表示因为熔断产生的上游（upstream）溢出。

深入分析

通过Envoy日志，我们可以做出一些分析和判断：

当httpbin服务的请求正常的时候，调用过程如下图：

当httpbin服务的熔断被触发的时候，调用过程如下图：

当httpbin服务熔断被触发时，返回503状态码的是httpbin服务的Envoy。

清理

kubectl delete destinationrule httpbin
kubectl delete -f samples/httpbin/httpbin.yaml
kubectl delete -f samples/httpbin/sample-client/fortio-deploy.yaml
1
2
3

最后，感谢你这么帅，还给我点赞。