SpringBoot系列之ES详细讲解_springboot es

                     SpringBoot系列之ES基本项目搭建

I. 项目搭建

1. 项目依赖

本项目借助SpringBoot 2.2.1.RELEASE + maven 3.5.3 + IDEA进行开发

开一个web服务用于测试

<dependencies>    <dependency>        <groupId>org.elasticsearch.client</groupId>        <artifactId>elasticsearch-rest-high-level-client</artifactId>    </dependency></dependencies>

2. 配置信息

配置文件application.yml，注意下面的配置信息，下面采用的是由我们自己来解析配置的方式

elasticsearch:  host: localhost  port: 9200  user: elastic  pwd: test123  connTimeout: 3000  socketTimeout: 5000  connectionRequestTimeout: 500

说明

上面配置介绍的是一种偏基础的es文档操作姿势，相比较于封装得更好的spring-boot-starter-data-elasticsearch，使用更加灵活

II. SpringBoot结合ES使用

1. RestHighLevelClient 初始化

接下来我们基于RestHighLevelClient来操作es，首先第一步就是需要初始化这实例

@Getter@Configurationpublic class ElasticsearchConfiguration {    @Value("${elasticsearch.host}")    private String host;    @Value("${elasticsearch.port}")    private int port;    @Value("${elasticsearch.connTimeout}")    private int connTimeout;    @Value("${elasticsearch.socketTimeout}")    private int socketTimeout;    @Value("${elasticsearch.connectionRequestTimeout}")    private int connectionRequestTimeout;    @Value("${elasticsearch.user}")    private String user;    @Value("${elasticsearch.pwd}")    private String pwd;    @Bean(destroyMethod = "close", name = "client")    public RestHighLevelClient initRestClient() {        RestClientBuilder builder = RestClient.builder(new HttpHost(host, port))                .setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder                        .setConnectTimeout(connTimeout)                        .setSocketTimeout(socketTimeout)                        .setConnectionRequestTimeout(connectionRequestTimeout));        return new RestHighLevelClient(builder);    }}

注意上面的实现，用户名 + 密码并没有使用，当es设置了用户名、密码之后，是通过每次请求时，在请求头基于Basic Auth方式进行身份验证的；后面会介绍到

2. 基本使用

我们在本机搭建了一个es用于模拟测试，在上面的配置完之后，就可以直接与es进行交互了

下面是一个简单的使用姿势

@Servicepublic class EsTest {    @Autowired    private RestHighLevelClient client;    private static String auth;    public EsTest(ElasticsearchConfiguration elasticsearchConfiguration) {        auth = Base64Utils.encodeToString((elasticsearchConfiguration.getUser() + ":" + elasticsearchConfiguration.getPwd()).getBytes());        auth = "Basic " + auth;    }    public void testGet() throws Exception {        // 文档查询        GetRequest getRequest = new GetRequest("first-index", "_doc", "gvarh3gBF9fSFsHNuO49");        RequestOptions.Builder requestOptions = RequestOptions.DEFAULT.toBuilder();        requestOptions.addHeader("Authorization", auth);        GetResponse getResponse = client.get(getRequest, requestOptions.build());        if (getResponse.isExists()) {            String sourceAsString = getResponse.getSourceAsString();            System.out.println(sourceAsString);        } else {            System.out.println("no string!");        }    }}

注意上面的实现，有下面几个重要知识点

身份验证

采用Basic Auth方式进行身份校验，简单来说就是在请求头中添加一个

•key = Authorization•value = "Basic " + base64(user + ":" + pwd)

访问姿势

上面是一个根据id查询文档的实例，简单可以理解为三步

•创建：XxRequest•添加请求头：RequestOptions.Builder.addHeader•执行: client.get(xxRequest, RequestOptions)

                   SpringBoot系列之ES查询常用实例演示

I. 项目搭建

1. 项目依赖

本项目借助SpringBoot 2.2.1.RELEASE + maven 3.5.3 + IDEA进行开发

开一个web服务用于测试

<dependencies>    <dependency>        <groupId>org.elasticsearch.client</groupId>        <artifactId>elasticsearch-rest-high-level-client</artifactId>    </dependency></dependencies>

2. 配置信息

配置文件application.yml，注意下面的配置信息，下面采用的是由我们自己来解析配置的方式

elasticsearch:  host: localhost  port: 9200  user: elastic  pwd: test123  connTimeout: 3000  socketTimeout: 5000  connectionRequestTimeout: 500

II. 实例演示

在开始之前就准备两条数据​​​​​​​

@Componentpublic class TermQueryDemo {    private BasicCurdDemo basicCurdDemo;    @Autowired    private RestHighLevelClient client;    @Autowired    private RequestOptions requestOptions;    private String TEST_ID = "11123-33345-66543-55231";    private String TEST_ID_2 = "11123-33345-66543-55232";    private String index = "term-demo";    public TermQueryDemo(BasicCurdDemo basicCurdDemo) throws IOException {        this.basicCurdDemo = basicCurdDemo;        Map<String, Object> doc = newMap("name", "一灰灰", "age", 10, "skills", Arrays.asList("java", "python"), "site", "blog.hhui.top");        basicCurdDemo.addDoc(index, doc, TEST_ID);        doc = newMap("name", "二灰灰", "age", 16, "skills", Arrays.asList("js", "html"));        basicCurdDemo.addDoc(index, doc, TEST_ID_2);    }    @PreDestroy    public void remove() throws IOException {        basicCurdDemo.delete(index, TEST_ID);        basicCurdDemo.delete(index, TEST_ID_2);    }}

1. 全量查询

即查询所有的文档，如借助kibanan的控制台，发起的请求形如​​​​​​​

GET index/_search{  "query": {    "match_all": {}  }}

于此对应的java实现如下​​​​​​​

/** * 全量查询 * * @throws IOException */private void queryAll() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    // 查询所有的文档    searchSourceBuilder.query(QueryBuilders.matchAllQuery());    searchRequest.source(searchSourceBuilder);    SearchResponse searchResponse = client.search(searchRequest, requestOptions);    System.out.println("mathAll: " + searchResponse.toString());}

注意上面的实现：

•初始化SearchRequest实例，用于构建请求相关数据•SearchSourceBuilder 来填充查询条件•client.search(searchRequest, requestOptions) 执行查询请求，第二个参数为请求参数，这里主要是设置请求时的权限验证信息

通常来说，实际的业务场景中，不太可能出现上面这种没有任何限制的查全量数据，即便真的有查全量数据的case，更常见的是分页查询，如下​​​​​​​

private void queryAll() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    int page = 1;    //每页记录数    int size = 2;    //计算出记录起始下标    int from = (page - 1) * size;    //起始记录下标，从0开始    searchSourceBuilder.from(from);    //每页显示的记录数    searchSourceBuilder.size(size);    // 根据age字段进行倒排    searchSourceBuilder.sort(new FieldSortBuilder("age").order(SortOrder.DESC));    // 查询所有的文档    searchSourceBuilder.query(QueryBuilders.matchAllQuery());    searchRequest.source(searchSourceBuilder);    SearchResponse searchResponse = client.search(searchRequest, requestOptions);    System.out.println("mathAll: " + searchResponse.toString());}

2. 根据Field值精确查询

即es中常说的term查询，具体实现如下​​​​​​​

/** * term精确查询 * * @throws IOException */private void term() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    // termQuery: 精确查询    // SpanTermQuery: 词距查询    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    searchSourceBuilder.query(QueryBuilders.termQuery("site", "blog.hhui.top"));    searchRequest.source(searchSourceBuilder);    SearchResponse response = client.search(searchRequest, requestOptions);    System.out.println("term: " + response.toString());}

从上面的实现也可以看出，查询的套路没啥区别，无非就是SearchSourceBuilder中的参数构造不一样；上面主要通过

•QueryBuilders.termQuery("site", "blog.hhui.top") 来构建 term的查询条件，表明查询 site=blog.hhui.top 的文档

中文查询不到问题

在我们实际使用过程中，如果value为中文，在查询时，可能会遇到命名有对应的数据，但是就查不到，主要原因就在于分词，如对于中文的查询，可以考虑下面这种方式​​​​​​​

private void term2() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    // 对于中文查询，需要注意分词的场景, 如果直接使用 "name : 张三" 的方式进行查询，则啥也不会返回    // elasticsearch 里默认的IK分词器是会将每一个中文都进行了分词的切割，所以你直接想查一整个词，或者一整句话是无返回结果的。    // 在此种情况下，我们可以通过指定 keyword 的方式来处理, 设置关键词搜索（不进行分词）    searchSourceBuilder.query(QueryBuilders.termQuery("name.keyword", "张三"));    searchRequest.source(searchSourceBuilder);    SearchResponse response = client.search(searchRequest, requestOptions);    System.out.println("term2: " + response.toString());}

3. Field值in查询

另外一个常见的就是多值查询，也就是我们常说的 field in (val1, val2...)，这个对应的就是es中的terms查询​​​​​​​

/** * 相当于in查询 * {"terms": { "name": ["张三", "李四] }} * * @throws IOException */private void multTerm() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    searchSourceBuilder.query(QueryBuilders.termsQuery("name.keyword", "张三", "李四"));    searchRequest.source(searchSourceBuilder);    SearchResponse response = client.search(searchRequest, requestOptions);    System.out.println("term: " + response.toString());}

4. 范围查询

对于数值类型的Field，同样是支持比较、范围查询的，对应的是es中 range​​​​​​​

/** * 范围查询 * { "range": { "age": { "gt":8, "lt": 12 } }} * * @throws IOException */private void range() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    searchSourceBuilder.query(QueryBuilders.rangeQuery("age").gt(8).lt(12));    searchRequest.source(searchSourceBuilder);    SearchResponse response = client.search(searchRequest, requestOptions);    System.out.println("range: " + response.toString());}

注意上面的查询有条件

•QueryBuilders.rangeQuery("age").gt(8).lt(12)•表示查询 age > 8 && age < 12•gte: 表示 >=•lte: 表示 <=

5. Field是否存在查询

es不同于mysql的在于它的field可以动态新增，当我们希望查询包含某个字段的文档时，可以考虑 exists​​​​​​​

/** * 根据字段是否存在查询 * * @throws IOException */private void exists() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    searchSourceBuilder.query(QueryBuilders.existsQuery("site"));    searchRequest.source(searchSourceBuilder);    SearchResponse response = client.search(searchRequest, requestOptions);    System.out.println("exists: " + response.toString());}

6. 模糊查询

es作为搜索引擎，更常见的是模糊匹配，比如match查询​​​​​​​

/**   * 根据字段匹配查询   *   * @throws IOException   */  private void match() throws IOException {      SearchRequest searchRequest = new SearchRequest(index);      searchRequest.types("_doc");      SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();      searchSourceBuilder.query(QueryBuilders.matchQuery("name", "灰"));      searchRequest.source(searchSourceBuilder);      SearchResponse response = client.search(searchRequest, requestOptions);      System.out.println("matchQuery: " + response.toString());  }

多Field中进行查询​​​​​​​

/** * 多字段中查询 * * @throws IOException */private void multiMatch() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    searchSourceBuilder.query(QueryBuilders.multiMatchQuery("灰", "name", "site"));    searchRequest.source(searchSourceBuilder);    SearchResponse response = client.search(searchRequest, requestOptions);    System.out.println("multiMatchQuery: " + response.toString());}

在es的语法支持中，除了match，还有一个wildcard，可以使用?来代指单字符，*来代指0..n字符​​​​​​​

/** * 模糊查询 ? 单字符  * 0..n字符 * * @throws IOException */private void wild() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    searchSourceBuilder.query(QueryBuilders.wildcardQuery("site", "*top"));    searchRequest.source(searchSourceBuilder);    SearchResponse response = client.search(searchRequest, requestOptions);    System.out.println("wildcard: " + response.toString());}

7. 正则匹配​​​​​​​

private void regexp() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    searchSourceBuilder.query(QueryBuilders.regexpQuery("site", ".*hhui.*"));    searchRequest.source(searchSourceBuilder);    SearchResponse response = client.search(searchRequest, requestOptions);    System.out.println("regexpQuery: " + response.toString());}

8. 前缀查询​​​​​​​

private void prefix() throws IOException {    SearchRequest searchRequest = new SearchRequest(index);    searchRequest.types("_doc");    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();    searchSourceBuilder.query(QueryBuilders.prefixQuery("site", "blog"));    searchRequest.source(searchSourceBuilder);    SearchResponse response = client.search(searchRequest, requestOptions);    System.out.println("prefixQuery: " + response.toString());}

             SpringBoot之ES文档基本操作CURD实例演示 

I. 项目搭建

1. 项目依赖

本项目借助SpringBoot 2.2.1.RELEASE + maven 3.5.3 + IDEA进行开发

开一个web服务用于测试​​​​​​​

<dependencies>    <dependency>        <groupId>org.elasticsearch.client</groupId>        <artifactId>elasticsearch-rest-high-level-client</artifactId>    </dependency></dependencies>

2. 配置信息

配置文件application.yml，注意下面的配置信息，下面采用的是由我们自己来解析配置的方式​​​​​​​

elasticsearch:  host: localhost  port: 9200  user: elastic  pwd: test123  connTimeout: 3000  socketTimeout: 5000  connectionRequestTimeout: 500

II. CURD实例

1. 配置

注意，本文介绍的es是添加了权限验证，因此我们在于es进行交互时，需要在请求头中携带验证信息，注意下面的实现姿势

读取配置，初始化RestHighLevelClient，和前文介绍的差不多​​​​​​​

@Getter@Configurationpublic class ElasticsearchConfiguration {    @Value("${elasticsearch.host}")    private String host;    @Value("${elasticsearch.port}")    private int port;    @Value("${elasticsearch.connTimeout}")    private int connTimeout;    @Value("${elasticsearch.socketTimeout}")    private int socketTimeout;    @Value("${elasticsearch.connectionRequestTimeout}")    private int connectionRequestTimeout;    @Value("${elasticsearch.user}")    private String user;    @Value("${elasticsearch.pwd}")    private String pwd;    @Bean(destroyMethod = "close", name = "client")    public RestHighLevelClient initRestClient() {        RestClientBuilder builder = RestClient.builder(new HttpHost(host, port))                .setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder                        .setConnectTimeout(connTimeout)                        .setSocketTimeout(socketTimeout)                        .setConnectionRequestTimeout(connectionRequestTimeout));        return new RestHighLevelClient(builder);    }    @Bean    public RequestOptions requestOptions() {        String auth = "Basic " + Base64Utils.encodeToString((user + ":" + pwd).getBytes());        RequestOptions.Builder build = RequestOptions.DEFAULT.toBuilder();        build.addHeader("Authorization", auth);        return build.build();    }}

2. 添加数据​​​​​​​

@Componentpublic class BasicCurdDemo {    @Autowired    private RestHighLevelClient client;    @Autowired    private RequestOptions requestOptions;    private String TEST_ID = "11123-33345-66543-55231";    /**     * 新增数据     */    public void addDoc(String indexName, Object obj, String id) throws IOException {        // 指定索引        IndexRequest request = new IndexRequest(indexName);        request.type("_doc");        // 文档内容，source传参，第一种时按照 fieldName, fieldValue 成对的方式传入；下面是采用json串 + 指定ContentType的方式传入        request.source(JSON.toJSONString(obj), XContentType.JSON);        // 指定特殊的id，不指定时自动生成id        request.id(id);        IndexResponse response = client.index(request, requestOptions);        System.out.println("添加数据返回结果: " + response.toString());    }}

添加数据，注意是利用 IndexRequest 来构建请求对象，添加文档时有几个注意事项

•request.source() : 具体需要上传的文档，就是通过它挂上去的，我们这里采用的是json方式•request.id(): 如果上传的文档需要指定id，则可以使用它；若未指定，则表明自动生成id

发起请求: client.index()

3. 查询数据

这里先介绍一个基础的根据id进行查询的实例case，更多的查询姿势后面会详细介绍​​​​​​​

/** * 查询结果 * * @param indexName * @param id * @throws Exception */public void get(String indexName, String id) throws IOException {    GetRequest getRequest = new GetRequest(indexName, "_doc", id);    GetResponse response = client.get(getRequest, requestOptions);    System.out.println("查询结果:" + response.toString());}

3. 增量更新数据

根据主键进行更新文档，如下​​​​​​​

/** * 更新文档，根据id进行更新，增量更新，存在的字段，覆盖；新增的字段，插入；旧字段，保留 * * @param indexName * @param docId * @param obj * @throws IOException */public void updateDoc(String indexName, String docId, Object obj) throws IOException {    UpdateRequest updateRequest = new UpdateRequest();    updateRequest.index(indexName);    updateRequest.type("_doc");    updateRequest.id(docId);    // 设置数据    updateRequest.doc(JSON.toJSONString(obj), XContentType.JSON);    UpdateResponse response = client.update(updateRequest, requestOptions);    System.out.println("更新数据返回：" + response.toString());}

注意

•上面的实现属于增量更新策略•即：新传的文档，若key之前已经存在，则覆盖更新；若之前不存在，则插入；之前文档中未被覆盖的数据依然保留

4. 全量更新

另外一个根据条件进行更新的使用case如下​​​​​​​

/** * 条件更新 * * @param indexName * @param query * @param data * @throws IOException */public void updateByCondition(String indexName, Map<String, String> query, Map<String, Object> data) throws IOException {    UpdateByQueryRequest updateRequest = new UpdateByQueryRequest(indexName);    for (Map.Entry<String, String> entry : query.entrySet()) {        QueryBuilder queryBuilder = new TermQueryBuilder(entry.getKey(), entry.getValue());        updateRequest.setQuery(queryBuilder);    }    // 更新值脚本，精确的更新方式    // ctx._source['xx'].add('新增字段')    // 条件判定 if(ctx._source.addr == 'hubei') { ctx._source.addr = 'wuhan';}    String source = "ctx._source.name='1hui';";    Script script = new Script(source);    updateRequest.setScript(script);    BulkByScrollResponse response = client.updateByQuery(updateRequest, requestOptions);    System.out.println("条件更新返回: " + response.toString());    get(indexName, TEST_ID);    System.out.println("0---------------------0");    // 采用全量覆盖式更新，直接使用data中的数据，覆盖之前的文档内容    source = "ctx._source=params";    script = new Script(ScriptType.INLINE, "painless", source, data);    updateRequest.setScript(script);    response = client.updateByQuery(updateRequest, requestOptions);    System.out.println("条件更新返回: " + response.toString());    get(indexName, TEST_ID);}

5. 删除数据

直接根据id进行删除​​​​​​​

/** * 根据id进行删除 * * @param indexName * @param id * @throws IOException */public void delete(String indexName, String id) throws IOException {    DeleteRequest deleteRequest = new DeleteRequest(indexName);    deleteRequest.type("_doc");    deleteRequest.id(id);    DeleteResponse response = client.delete(deleteRequest, requestOptions);    System.out.println("删除后返回" + response.toString());}

6. 条件删除数据

根据条件进行匹配删除​​​​​​​

/** * 条件删除 * * @param indexName * @param query * @throws IOException */public void deleteByQuery(String indexName, Map<String, String> query) throws IOException {    DeleteByQueryRequest request = new DeleteByQueryRequest(indexName);    request.types("_doc");    for (Map.Entry<String, String> entry : query.entrySet()) {        QueryBuilder queryBuilder = new TermQueryBuilder(entry.getKey(), entry.getValue());        request.setQuery(queryBuilder);    }    BulkByScrollResponse response = client.deleteByQuery(request, requestOptions);    System.out.println("条件删除:" + response.toString());    get(indexName, TEST_ID);}

7. 测试case

写一个测试demo，将上面的case都跑一遍​​​​​​​

public void testOperate() throws IOException {    String index = "basic_demo";    Map<String, Object> doc = newMap("name", "一灰灰", "age", 10, "skills", Arrays.asList("java", "python"));    // 新增    addDoc(index, doc, TEST_ID);    // 查询    get(index, TEST_ID);    // 更新    doc.clear();    doc.put("name", "一灰灰blog");    doc.put("addr", "hubei");    updateDoc(index, TEST_ID, doc);    get(index, TEST_ID);    updateByCondition(index, newMap("addr", "hubei"), newMap("name", "yihuihui", "site", "https://hhui.top"));    get(index, TEST_ID);    // 删除文档    delete(index, TEST_ID);}public <K, V> Map<K, V> newMap(K k, V v, Object... kv) {    Map<K, V> map = new HashMap<>();    map.put(k, v);    for (int i = 0; i < kv.length; i += 2) {        map.put((K) kv[i], (V) kv[i + 1]);    }    return map;}

输出如下​​​​​​​

# 1. 添加数据添加数据返回结果: IndexResponse[index=basic_demo,type=_doc,id=11123-33345-66543-55231,version=1,result=created,seqNo=34,primaryTerm=4,shards={"total":2,"successful":1,"failed":0}]# 2. 查询数据查询结果:{"_index":"basic_demo","_type":"_doc","_id":"11123-33345-66543-55231","_version":1,"_seq_no":34,"_primary_term":4,"found":true,"_source":{"skills":["java","python"],"name":"一灰灰","age":10}}# 3. 增量更新2022-03-28 19:56:08.781  WARN 18332 --- [/O dispatcher 1] org.elasticsearch.client.RestClient      : request [POST http://localhost:9200/basic_demo/_doc/11123-33345-66543-55231/_update?timeout=1m] returned 1 warnings: [299 Elasticsearch-7.12.0-78722783c38caa25a70982b5b042074cde5d3b3a "[types removal] Specifying types in document update requests is deprecated, use the endpoint /{index}/_update/{id} instead."]更新数据返回：UpdateResponse[index=basic_demo,type=_doc,id=11123-33345-66543-55231,version=2,seqNo=35,primaryTerm=4,result=updated,shards=ShardInfo{total=2, successful=1, failures=[]}]查询结果:{"_index":"basic_demo","_type":"_doc","_id":"11123-33345-66543-55231","_version":2,"_seq_no":35,"_primary_term":4,"found":true,"_source":{"skills":["java","python"],"name":"一灰灰blog","age":10,"addr":"hubei"}}# 4. 全量条件更新条件更新返回: BulkByScrollResponse[took=970ms,timed_out=false,sliceId=null,updated=1,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]查询结果:{"_index":"basic_demo","_type":"_doc","_id":"11123-33345-66543-55231","_version":3,"_seq_no":36,"_primary_term":4,"found":true,"_source":{"skills":["java","python"],"name":"1hui","addr":"hubei","age":10}}​​​​​​​​​​​​​​