＜httpasyncclient＞高并发场景报错分析与解决_connection lease request time out

本文总结讲解httpasyncclient使用过程中遇到的各种异常（java.net.ConnectException、java.net.SocketTimeoutException、java.util.concurrent.TimeoutException: Connection lease request time out），原因、分析、解决方方案及思考。

背景：

httpasyncclient 在负载高场景下经常遇到以下报错：接下来我们进项以下综合分析解决

java.util.concurrent.TimeoutException: Connection lease request time out
  at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:411)
  at org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest(AbstractNIOConnPool.java:391)
  at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:355)
  at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseConnection(PoolingNHttpClientConnectionManager.java:391)
  at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.discardConnection(AbstractClientExchangeHandler.java:276)
  at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:428)
  at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
  at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
  at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
  at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
  at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
  at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
  at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
  at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
  at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
  at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
  at java.lang.Thread.run(Thread.java:852)收起

首先解释一下 Connection lease request time out，字面意思为连接契约请求超时，即为getConnectionRequestTimeout 获取连接超时。

httpasyncclient 的三种超时：

java.net.ConnectException

异常栈：

java.net.ConnectException: Timeout connecting to [/120.77.77.52:80]
	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169)
	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:632)
	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:898)
	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:198)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:213)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.lang.Thread.run(Thread.java:748)

分析：

连接超时，建立http连接超时。通常发生原因与服务器抖动，服务方系统性能关联，解决方案亦为排查以上两个问题。

java.net.SocketTimeoutException

异常栈：

java.net.SocketTimeoutException: XXX milliseconds timeout on connection http-outgoing-133 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(Thread.java:748)

分析：

通信超时。通常发生原因与服务器抖动，服务方系统性能关联，解决方案亦为排查以上两个问题。当然也有例外对于java应用本机的问题也容易导致这个异常，后续我会专门写一篇文章来介绍怎样排查，欢迎大家关注。

java.util.concurrent.TimeoutException: Connection lease request time out

异常栈：

java.util.concurrent.TimeoutException: Connection lease request time out
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:411)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:92)
	at com.amap.aos.http.client.core.support.DefaultHttpClient.asyncExecute(DefaultHttpClient.java:650)
	at com.amap.aos.http.client.core.support.DefaultHttpClient.post(DefaultHttpClient.java:615)
	at com.amap.aos.http.client.core.support.DefaultHttpClient.post(DefaultHttpClient.java:638)

分析：

Connection lease request time out是一个比较复杂且难解决的异常，本文是由该异常的排查解决及实验产生的，如背景中介绍Connection lease request time out，字面意思为连接契约请求超时，即为getConnectionRequestTimeout 获取连接超时。lease request time 的计时从请求开始请求连接开始，因此只要连接池满，排队时间也计入lease request time。所以这个异常通常是本地http连接负载高时发生。那么解决这个问题就与在http连接高时进行阻塞式单机限流，提供背压反馈关联。

后续我也会专门下一篇文章介绍我的解决方案，欢迎大家关注