【Redis连接超时】io.lettuce.core.RedisConnectionException: Unable to connect to 192.168.x.x:7000_caused by: io.lettuce.core.redisconnectionexceptio

「扫码关注我，面试、各种技术（mysql、zookeeper、微服务、redis、jvm）持续更新中～」

今天临近下班了，线上开始频繁报警，各种Redis连接超时，顿时慌的一批，因为最近在优化系统高频查询时用到了Redis作为缓存，难道要出生产事故，额～～～ 一首凉凉送给自己。。。。。。
于是马上联系下运维看下什么情况，运维看了下监控情况，OPS（operation per second）确实增加了不少，见下图：

于是乎发现确实是自己的锅，二话不说，先回复线上优化查询前的版本，保证线上能够正确运行，减少事故造成的影响，发完版本后，报警顿时停止，好了，开始找BUG吧～
先去ELK楼下日志，发现日志如下：

org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is io.lettuce.core.RedisException: io.lettuce.core.RedisConnectionException: Unable to connect to 192.168.x.x:7000
	at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:74)
	at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
	at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:257)
	at org.springframework.data.redis.connection.lettuce.LettuceHashCommands.convertLettuceAccessException(LettuceHashCommands.java:445)
	at org.springframework.data.redis.connection.lettuce.LettuceHashCommands.hGet(LettuceHashCommands.java:171)
	at org.springframework.data.redis.connection.DefaultedRedisConnection.hGet(DefaultedRedisConnection.java:855)
	at org.springframework.data.redis.connection.DefaultStringRedisConnection.hGet(DefaultStringRedisConnection.java:429)
	at org.springframework.data.redis.core.DefaultHashOperations.lambda$get$0(DefaultHashOperations.java:52)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184)
	at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:95)
	at org.springframework.data.redis.core.DefaultHashOperations.get(DefaultHashOperations.java:52)
	at com.xes.merchant.service.impl.InvoiceServiceImpl.getCacheByRegIdAndStagesId(InvoiceServiceImpl.java:336)
	at com.xes.merchant.service.impl.InvoiceServiceImpl$$FastClassBySpringCGLIB$$d92d6c6c.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
	at com.xes.merchant.service.impl.InvoiceServiceImpl$$EnhancerBySpringCGLIB$$cf8e4582.getCacheByRegIdAndStagesId(<generated>)
	at com.xes.merchant.service.impl.ForOrdersServiceImpl.queryFinanceListByRegistId(ForOrdersServiceImpl.java:269)
	at com.xes.merchant.controller.ForOrdersController.queryFinanceListByRegistId(ForOrdersController.java:182)
	at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.xes.payment.safe.filter.XssFilter.doFilter(XssFilter.java:64)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:105)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:123)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: io.lettuce.core.RedisException: io.lettuce.core.RedisConnectionException: Unable to connect to 192.168.x.x:7000
	at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:125)
	at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:118)
	at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
	at com.sun.proxy.$Proxy156.hget(Unknown Source)
	at org.springframework.data.redis.connection.lettuce.LettuceHashCommands.hGet(LettuceHashCommands.java:169)
	... 76 common frames omitted
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

异常已经很明显了，那么接下来就是如何去解决问题了，但是自己也好奇，Redis的性能应该非常高啊，但是为什么会出现连接超时呢？
首先描述下业务背景，该接口是查询财务交易的接口，该接口中包含财务支付核心信息还有是否开过发票，并且该接口还是个批量查询接口，就是一次性可以传递不超过50个报名ID，根据报名ID集合去查询数据库中的财务信息和发票信息。

由于是以传过来报名ID集合拼接作为Redis的key，所以当发票状态改变时无法找到对应的报名ID的缓存，因此，基于以上情况，设置为2个缓存来分别存放财务信息和发票信息，财务记录存取采用Redis的String结构来存放，发票信息财务Hash结构存放。

当第一次查询财务发票接口时，首先在缓存中获取，如果缓存中没有，则直接查询数据库，然后将查询当财务信息存取到缓存中，同时将发票信息也存储到对应的缓存中（单条操作），这样如果一次传过来多个报名ID，同样需要设置多次发票缓存。

情况已经比较明了了，经过分析发现存在以下几方面的问题：
1.Redis缓存key是报名ID拼接的，每一个报名ID是32位，那么如果传过来是批量报名ID，那么一次行传50个的话，拼接起来作为key，那么key就会有1600字符，如果缓存数量过多，那么在缓存Get时，就不利于缓存命中，严重影响OPS，所以将key进行MD5加密后进行存储，减小key的长度；
2.排查SpringBoot整合Redis时，发现配置文件中并没有重新设置连接池大小，而是使用的默认值，同时连接超时时间（连接断开等待时间）也没有进行设置，当请求量上来也会影响Redis的OPS，所以重新配置了下，如下：

# 连接池最大连接数（使用负值表示没有限制）
spring.redis.lettuce.pool.max-active=300
# 连接池中的最大空闲连接
spring.redis.lettuce.pool.max-idle=100
# 连接池中的最小空闲连接
spring.redis.lettuce.pool.min-idle=8
# 连接池最大阻塞等待时间（使用负值表示没有限制）
spring.redis.lettuce.pool.max-wait=-1ms
# 连接超时时间（毫秒）
spring.redis.timeout=12000ms
1
2
3
4
5
6
7
8
9
10

3.采用管道方式（executePipelined）批量设置发票缓存，同时获取也采用批量的方式（multiGet），代码如下：

//批量设置
redisTemplate.executePipelined(new RedisCallback<Object>() {
            @Override
            public Object doInRedis(RedisConnection connection) throws DataAccessException {
                for(FinanceVO financeVO : financeVOList){
                    String key = StringUtils.join(RedisConstant.INVOICE_BY_REGIDANDSTAGESID,financeVO.geRegtId(),financeVO.getStagesId());
                    //获取序列化器
                    RedisSerializer<String> keySerializer = (RedisSerializer<String>) redisTemplate.getKeySerializer();
                    RedisSerializer<String> hashKeySerializer = (RedisSerializer<String>) redisTemplate.getHashKeySerializer();
                    RedisSerializer<String> hashValueSerializer = (RedisSerializer<String>) redisTemplate.getHashValueSerializer();

                    byte[] rawKey = keySerializer.serialize(key);
                    byte[] invoiceIdField = hashKeySerializer.serialize("id");
                    byte[] invoiceStatusField = hashKeySerializer.serialize("status");
                    byte[] invoiceIdValue = hashValueSerializer.serialize(financeVO.getId());
                    byte[] invoiceStatusValue = hashValueSerializer.serialize(financeVO.getStatus());

                    Map<byte[],byte[]> hashes = new HashMap<>();
                    hashes.put(invoiceIdField,invoiceIdValue);
                    hashes.put(invoiceStatusField,invoiceStatusValue);
                    connection.hMSet(rawKey,hashes);
                    connection.expire(rawKey,invoiceTimeout*24*60*60);
                }
                return null;
            }
        });
 //批量获取
 List<String> values = redisTemplate.opsForHash().multiGet(key,hashKeys);
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

经过改造之后，重新上线，OPS值正常了，说明以上优化是起作用的，同时，也在以后的开发工作中积累了经验，但是缓存数据也会出现各种个样的问题，主要是数据库缓存不一致等问题，所以，缓存数据应该是基本保持不变的数据，不然可能导致很多问题。