Hystrix相关问题源码分析_熔断hystric后恢复不了

hystrix执行流程：

1. 将请求构造成command对象
2. 执行command
3. 判断command的响应是否被缓存，如果有，直接返回，这边缓存的操作在客户端配置
4. 判断熔断器是否打开，如果打开执行熔断操作，进行fallback处理
5. 判断线程池是否满，默认是线程池隔离，如果线程池满了，进行fallback处理
6. 执行实际的业务逻辑，如果执行异常则进行fallback处理，否则若在预设的执行时间窗口内没有返回结果则执行fallback处理
7. 后台线程会收集相关的指标计算，来设置熔断器是否打开，指标收集来自command执行过程发送的成功、失败事件进行聚合
8. fallback处理，如果没实现fallback方法则抛异常
9. 执行成功返回响应

熔断器状态变化：

关于command耗时计算

当hystrix线程池队列开始堆积的时候，command的耗时时间并未计算队列中等待的时间，hystrix判断一个command是否超时，是在commad执行时加入到一个定时任务中做的判断，源码在如下HystrixObservableTimeoutOperator定时执行的：

Observable execution;
if ((Boolean)this.properties.executionTimeoutEnabled().get()) {
     execution = this.executeCommandWithSpecifiedIsolation(_cmd).lift(new AbstractCommand.HystrixObservableTimeoutOperator(_cmd));
 } else {
     execution = this.executeCommandWithSpecifiedIsolation(_cmd);
 }
1
2
3
4
5
6

关于线程池扩容问题

线程池配置有如下几个相关配置，当已堆积线程池队列长度>queueSizeRejectionThreshold且queueSizeRejectionThreshold<maxQueueSize，这是由于线程池的队列未满，所以线程池不会使用maximumSize的线程扩容，可看到如下代码线程池调度时候的判断，会返回false，如果需要扩容的话，需要设置queueSizeRejectionThreshold>maxQueueSize

hystrix.threadpool.default.coreSize
hystrix.threadpool.default.maximumSize
hystrix.threadpool.default.maxQueueSize
hystrix.threadpool.default.queueSizeRejectionThreshold
1
2
3
4

public boolean isQueueSpaceAvailable() {
   if (this.queueSize <= 0) {
       return true;
   } else {
       return this.threadPool.getQueue().size() < (Integer)this.properties.queueSizeRejectionThreshold().get();
   }
}
1
2
3
4
5
6
7

关于熔断器熔断后不恢复的问题

5.1.12版本有个bug，当OPEN->HALF_OPEN，放一个请求执行时，如果这个请求被取消订阅（unsubscribe），这时候状态没有更新，还是HALF_OPEN，所以后面请求进来判断是否熔断，永远是false，熔断器永远不会close，判断熔断源码如下：

public boolean attemptExecution() {
   if ((Boolean)this.properties.circuitBreakerForceOpen().get()) {
       return false;
   } else if ((Boolean)this.properties.circuitBreakerForceClosed().get()) {
       return true;
   } else if (this.circuitOpened.get() == -1L) {
       return true;
   } else if (this.isAfterSleepWindow()) {
       return this.status.compareAndSet(HystrixCircuitBreaker.HystrixCircuitBreakerImpl.Status.OPEN, HystrixCircuitBreaker.HystrixCircuitBreakerImpl.Status.HALF_OPEN);
   } else {
       return false;
   }
}
1
2
3
4
5
6
7
8
9
10
11
12
13

unsubscribe触发的条件比较隐蔽还没找到，可以手动调用api复现此bug，如下复现的代码，升级版本后bug已修复：

public void startDemo() {
  int i=0;
  runSimulatedRequestOnThread(i++);
}

 public void runSimulatedRequestOnThread(int i) {

     CountDownLatch countDownLatch=new CountDownLatch(1);
     CountDownLatch countDownLatch1=new CountDownLatch(1);

     pool.execute(new Runnable() {

         @Override
         public void run() {
             HystrixRequestContext context = HystrixRequestContext.initializeContext();
             try {
                 //触发熔断
                 for (int j = 0; j < 10; j++) {
                     HystrixCommand<PaymentInformation> faile1 = new GetPaymentInformationCommand(0);
                     faile1.execute();
                     System.out.println("Request1 => " + HystrixRequestLog.getCurrentRequest().getExecutedCommandsAsString());
                 }
                 Thread.sleep(400);
                 countDownLatch.countDown();
             } catch (Exception e) {
                 e.printStackTrace();
             } finally {
                 context.shutdown();
             }
         }

     });

     try {
         countDownLatch.await();
     } catch (InterruptedException e) {
         e.printStackTrace();
     }


     pool.execute(new Runnable() {

         @Override
         public void run() {
             HystrixRequestContext context = HystrixRequestContext.initializeContext();
             try {

                 //模拟取消订阅
                 HystrixCommand<PaymentInformation> success = new GetPaymentInformationCommand(1);
//                    success.execute();
//                    System.out.println("Request2 => " + HystrixRequestLog.getCurrentRequest().getExecutedCommandsAsString());
                 Observable<PaymentInformation> observe = success.observe();
                 Subscription s = observe.subscribe();
                 s.unsubscribe();
                 System.out.println("Request2 => " + HystrixRequestLog.getCurrentRequest().getExecutedCommandsAsString());
                 Thread.sleep(3000);
                 countDownLatch1.countDown();
             } catch (Exception e) {
                 e.printStackTrace();
             } finally {
                 context.shutdown();
             }
         }
     });

     try {
         countDownLatch1.await();
     } catch (InterruptedException e) {
         e.printStackTrace();
     }

     pool.execute(new Runnable() {
         @Override
         public void run() {
             HystrixRequestContext context = HystrixRequestContext.initializeContext();
             try {
                 for (int i=0;i<100;i++){
                     PaymentInformation paymentInformation = new GetPaymentInformationCommand(2).execute();
                     System.out.println("Request3 => " + HystrixRequestLog.getCurrentRequest().getExecutedCommandsAsString());
                 }
             } catch (Exception e) {
                 e.printStackTrace();
             } finally {
                 context.shutdown();
             }
         }
     });
 }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88