解决Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large问题

一、背景

公司有一批8万的数据存储在Mysql中，然后我使用多线程的方式调用Elasticsearch的bulk()方法推送到ES，但是在推送过程中出现了该问题，这属于插入数据时产生的问题

二、异常

EVERE: Servlet.service() for servlet [default] in context with path [appBoot] threw exception [http://192.168.3.83:10014/api/kms-wiki/ElasticsearchService/createIndex] with root cause
ElasticsearchStatusException[Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [511812774/488.1mb], which is larger than the limit of [510027366/486.3mb], real usage: [510461080/486.8mb], new bytes reserved: [1351694/1.2mb], usages [request=0/0b, fielddata=10341/10kb, in_flight_requests=18268042/17.4mb, accounting=2110340/2mb]]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1793)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1770)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1527)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
	at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:616)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1629)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/_bulk?timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [511812774/488.1mb], which is larger than the limit of [510027366/486.3mb], real usage: [510461080/486.8mb], new bytes reserved: [1351694/1.2mb], usages [request=0/0b, fielddata=10341/10kb, in_flight_requests=18268042/17.4mb, accounting=2110340/2mb]","bytes_wanted":511812774,"bytes_limit":510027366,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [511812774/488.1mb], which is larger than the limit of [510027366/486.3mb], real usage: [510461080/486.8mb], new bytes reserved: [1351694/1.2mb], usages [request=0/0b, fielddata=10341/10kb, in_flight_requests=18268042/17.4mb, accounting=2110340/2mb]","bytes_wanted":511812774,"bytes_limit":510027366,"durability":"TRANSIENT"},"status":429}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514)
		... 37 more
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

三、解决办法

加大-Xms和-Xmx的值，比如docker-compose.yaml文件中可以这样设置：

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    environment:
     - ES_JAVA_OPTS=-Xms1g -Xmx2g
1
2
3
4
5

四、解释

1、异常分析

从上面错误日志可以看出，现在是父熔断器（parent）直接熔断了，现在我们来介绍一下这几种熔断器作用

• parent circuit breaker（父熔断器）：在所有断路器上使用的总内存量
• Field data circuit breaker（列数据熔断器）
• Request circuit breaker(请求熔断器)
• Accounting requests circuit breaker
• Script compilation circuit breaker（脚本编译熔断器）

2、查看父熔断器占据JVM最大堆内存的比例值

默认值： 95%

要求： 大点好

查看方式：

请求方式：
GET

请求链接（ip和port都是Elasticsearch的；查看该接口返回值中的indices.breaker.total.limit参数值就是上面提到的比例值；如果没有该值，那就是使用默认值95%）：
http://ip:port/_cluster/settings​
1
2
3
4
5

设置方式：

请求方式：
PUT 

请求链接（其中ip和port都是Elasticsearch的）：
http://ip:port/_cluster/settings

请求体：
{
  "persistent": {
    "indices.breaker.total.limit": "95%"
  }
}
1
2
3
4
5
6
7
8
9
10
11
12

3、查看父熔断器占据JVM堆的具体值

请求方式：
GET

请求链接（ip和port都是Elasticsearch的；查看该接口返回值中的parent的limit_size参数值，该值受到上面比例值的限制）：
http://ip:port/_nodes/stats/breaker
1
2
3
4
5

比如现在设置- ES_JAVA_OPTS=-Xms1g -Xmx2g，然后具体值是1.8g，我现在的父熔断器占比是95%，所以1.8g是差不多的，如下：

4、解决查询时导入的熔断问题

• ES调优2（bulk报错）