ElasticSearch搜索_elasticsearch 获取索引

ElasticSearch

安装

环境要求：JDK1.8，最低要求。ElasticSearch 客户端，界面工具！Node.js npm

官网：欢迎来到 Elastic — Elasticsearch 和 Kibana 的开发者 | Elastic

1、使用

解压即可使用

2、目录解析

bin 目录文件

config 配置文件

​ log4j2 日志文件

​ jvm.options java 虚拟机相关的配置

​ elasticsearch.yml elasticsearch 的配置文件！默认端口9200 跨域！

lib 相关jar包

logs 日志！

modules 功能模块

plugins 插件！ ik

3、启动

直接双击bin目录下的elasticsearch.bat启动文件即可。

访问9200

4、访问测试

5、安装可视化工具 elasticsearch-head-master

github地址：mobz/elasticsearch-head：弹性搜索集群的 Web 前端 (github.com)

需安装node.js环境

启动

npm install 安装依赖包

npm run start 启动工具

6、问题

http://localhost:9100通过可视化工具访问ElasticSerach时出现跨域问题

7、解决

进入elasticsearch.yml文件修改配置，然后重启elasticsearch.bat

成功访问

Kibana安装

Kibana是一个针对Elasticsearch的开源分析及可视化平台，用来搜索、查看交互存储在Elasticsearch索引中的数据。使用Kibana，可以通过各种图表高级数据分析及展示。Kibana让海量数据更容易理解。它操作简单，基于浏览器的用户界面可以快速创建仪表板实时显示Elasticsearch查询动态。设置Kibana非常简单。无需编码或者额外的基础架构，几分钟内就可以完成Kibana安装并启动Elasticsearch索引监测。

官网：免费下载 Kibana |立即开始 |弹性的 (elastic.co)

Kibana 版本要和Elasticsearch一样。

1、启动

启动bin目录下的kibana.bat文件

2、测试访问

3、操作

之后所有的请求测试操作都在这里进行

4、可汉化

找到文件目录D:\Elasticsearch\kibana-7.6.1-windows-x86_64\x-pack\plugins\translations\translations 有zh-CN.json

找到配置文件修改配置文件，然后重启Kibana.bat文件

效果

核心

1、索引

2、字段类型（mapping）

3、文档

与数据库对比

IK分词器

将ik分词器放到es的插件目录中，然后重启es

IK提供了两个分词算法：ik_smart 和 ik_max_word,其中，ik_smart 为最少切分，ik_max_word为最细粒度划分

1、ik_smart演示

{
  "tokens" : [
    {
      "token" : "中国共产党",
      "start_offset" : 1,
      "end_offset" : 6,
      "type" : "CN_WORD",
      "position" : 0
    }
  ]
}
1
2
3
4
5
6
7
8
9
10
11

2、ik_max_word演示

{
  "tokens" : [
    {
      "token" : "中国共产党",
      "start_offset" : 1,
      "end_offset" : 6,
      "type" : "CN_WORD",
      "position" : 0
    },
    {
      "token" : "中国",
      "start_offset" : 1,
      "end_offset" : 3,
      "type" : "CN_WORD",
      "position" : 1
    },
    {
      "token" : "国共",
      "start_offset" : 2,
      "end_offset" : 4,
      "type" : "CN_WORD",
      "position" : 2
    },
    {
      "token" : "共产党",
      "start_offset" : 3,
      "end_offset" : 6,
      "type" : "CN_WORD",
      "position" : 3
    },
    {
      "token" : "共产",
      "start_offset" : 3,
      "end_offset" : 5,
      "type" : "CN_WORD",
      "position" : 4
    },
    {
      "token" : "党",
      "start_offset" : 5,
      "end_offset" : 6,
      "type" : "CN_CHAR",
      "position" : 5
    }
  ]
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

3、发现问题

狂深说被拆开了，自己需要的此需要独立加到的自己的字典

{
  "tokens" : [
    {
      "token" : "超级",
      "start_offset" : 1,
      "end_offset" : 3,
      "type" : "CN_WORD",
      "position" : 0
    },
    {
      "token" : "喜欢",
      "start_offset" : 3,
      "end_offset" : 5,
      "type" : "CN_WORD",
      "position" : 1
    },
    {
      "token" : "狂",
      "start_offset" : 5,
      "end_offset" : 6,
      "type" : "CN_CHAR",
      "position" : 2
    },
    {
      "token" : "深",
      "start_offset" : 6,
      "end_offset" : 7,
      "type" : "CN_CHAR",
      "position" : 3
    },
    {
      "token" : "说",
      "start_offset" : 7,
      "end_offset" : 8,
      "type" : "CN_CHAR",
      "position" : 4
    }
  ]
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

添加字典

修改Ik分词器里面的配置文件即可

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
	<comment>IK Analyzer 扩展配置</comment>
	<!--用户可以在这里配置自己的扩展字典 -->
	<entry key="ext_dict">study.dic</entry>
	 <!--用户可以在这里配置自己的扩展停止词字典-->
	<entry key="ext_stopwords"></entry>
	<!--用户可以在这里配置远程扩展字典 -->
	<!-- <entry key="remote_ext_dict">words_location</entry> -->
	<!--用户可以在这里配置远程扩展停止词字典-->
	<!-- <entry key="remote_ext_stopwords">words_location</entry> -->
</properties>
1
2
3
4
5
6
7
8
9
10
11
12
13

重启es

添加自己的字典后

{
  "tokens" : [
    {
      "token" : "超级",
      "start_offset" : 1,
      "end_offset" : 3,
      "type" : "CN_WORD",
      "position" : 0
    },
    {
      "token" : "喜欢",
      "start_offset" : 3,
      "end_offset" : 5,
      "type" : "CN_WORD",
      "position" : 1
    },
    {
      "token" : "狂深说",
      "start_offset" : 5,
      "end_offset" : 8,
      "type" : "CN_WORD",
      "position" : 2
    }
  ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

REST风格操作

基础测试

1、插入一条数据：PUT

PUT /索引名称/类型名称/文档id

2、定义一个无数据的索引：PUT

PUT /索引名称/类型名称

---

3、GET数据

4、查看索引库

5、复杂查询

高亮查询

SpringBoot基础es

1、创建项目

问题：es版本要和自己使用的版本一致。

导入依赖

<parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.7.12</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.es</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>elasticsearch</name>
    <description>elasticsearch</description>
    <properties>
        <java.version>1.8</java.version>
        <!--自定义es版本-->
        <elasticsearch.version>7.6.1</elasticsearch.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

ES配置类

/**
 * Es配置
 */
@Configuration //注入到Spring中管理
public class EsConfig {

   @Bean
   public RestHighLevelClient restHighLevelClient(){
       RestHighLevelClient client = new RestHighLevelClient(
               RestClient.builder(
                       new HttpHost("127.0.0.1",9200,"http")
               ));
       return client;
   }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

2、API操作

创建索引

/**
     * ES高级客户端对象
     */
    @Autowired
    private RestHighLevelClient restHighLevelClient;

    /**
     * 测试创建索引
     */
    @Test
    void testCreateIndex() throws IOException {
        //创建索引对象
        CreateIndexRequest indexRequest = new CreateIndexRequest("sixkey_index1");
        //通过客户端对象发起创建索引操作
        CreateIndexResponse indexResponse = restHighLevelClient.indices().create(indexRequest, RequestOptions.DEFAULT);
        //打印索引对象
        log.info("索引对象：{}",indexResponse);
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

获取索引是否存在

/**
     * 测试获取索引，只能判断是否存在
     */
    @Test
    public void testIndexExists() throws IOException {
        GetIndexRequest indexRequest = new GetIndexRequest("sixkey_index1");
        boolean exists = restHighLevelClient.indices().exists(indexRequest, RequestOptions.DEFAULT);
        log.info("索引是否存在：{}",exists);
    }
1
2
3
4
5
6
7
8
9

删除索引

 /**
     * 测试删除索引
     */
    @Test
    public void testDeleteIndex() throws IOException {
        DeleteIndexRequest indexRequest = new DeleteIndexRequest("sixkey_index1");
        AcknowledgedResponse delete = restHighLevelClient.indices().delete(indexRequest, RequestOptions.DEFAULT);
        log.info("索引是否删除成功：{}",delete);
    }
1
2
3
4
5
6
7
8
9

3、文档操作

添加数据

/**
     * 测试向索引中添加文档（数据）
     */
    @Test
    public void testAddDocuments() throws IOException {
        //添加一个User对象
        User user = new User("test1", 1);
        //获取索引请求
        IndexRequest request = new IndexRequest("sixkey_index1");
        //规则 PUT sixkey_index1/_doc/1
        request.id("1");
        request.timeout("1s");
        //将User数据放入请求,需要以JSON格式发送
        request.source(JSON.toJSONString(user), XContentType.JSON);

        //客户端发送请求，获取响应结果
        IndexResponse response = restHighLevelClient.index(request, RequestOptions.DEFAULT);
        log.info("响应结果：{}",response.toString());
        //响应状态：CREATED
        log.info("响应状态：{}",response.status());
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

测试数据是否存在

/**
     * 测试判断数据是否存在
     */
    @Test
    public void testIsExists() throws IOException {
        //获取数据 GET sixkey_index1/_doc/1
        GetRequest request = new GetRequest("sixkey_index1", "1");
        //判断是否存在
        boolean exists = restHighLevelClient.exists(request, RequestOptions.DEFAULT);
        log.info("数据是否存在：{}",exists);
    }
1
2
3
4
5
6
7
8
9
10
11

测试修改文档

/**
     * 测试修改文档（数据）
     */
    @Test
    public void testUpdateDocuments() throws IOException {
        //添加一个User对象
        User user = new User("test2", 2);
        //获取索引请求
        UpdateRequest request = new UpdateRequest("sixkey_index1","1");
        //规则 PUT sixkey_index1/_doc/1
        request.timeout("1s");
        //将User数据放入请求,需要以JSON格式发送
        request.doc(JSON.toJSONString(user), XContentType.JSON);

        //客户端发送请求，获取响应结果
        UpdateResponse update = restHighLevelClient.update(request, RequestOptions.DEFAULT);
        log.info("响应结果：{}",update.toString());
        //响应状态：CREATED
        log.info("响应状态：{}",update.status());
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

删除文档记录

/**
     * 删除文档记录
     */
    @Test
    public void testDeleteDocument() throws IOException {
        DeleteRequest request = new DeleteRequest("sixkey_index1", "1");
        request.timeout("1s");
        //执行删除操作
        DeleteResponse response = restHighLevelClient.delete(request, RequestOptions.DEFAULT);
        log.info("是否删除成功：{}",response.status());
    }
1
2
3
4
5
6
7
8
9
10
11

批量添加操作

/**
     * 批量添加数据
     * @throws IOException
     */
    @Test
    public void testBulkAddDocuments() throws IOException {
        //批量操作对象
        BulkRequest request = new BulkRequest();
        request.timeout("5s");
        List<User> userList = new ArrayList<>();
        userList.add(new User("test1",1));
        userList.add(new User("test2",2));
        userList.add(new User("test3",3));
        userList.add(new User("test4",4));
        userList.add(new User("test5",5));
        for (int i = 0; i < userList.size(); i++) {
            request.add(new IndexRequest("sixkey_index1")
                    .id((i+1) + "")
                    .source(JSON.toJSONString(userList.get(i)),XContentType.JSON));
        }
        BulkResponse bulk = restHighLevelClient.bulk(request, RequestOptions.DEFAULT);
        log.info("响应状态：{}",bulk.status());
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

查询操作

/**
     * 查询操作
     * // SearchRequest 搜索请求
     * // SearchSourceBuilder 条件构造
     * // HighlightBuilder 构建高亮
     * // TermQueryBuilder 精确查询
     * // MatchAllQueryBuilder 
     * @throws IOException
     */
    @Test
    public void testSearch() throws IOException {
        //获取查询请求
        SearchRequest searchRequest = new SearchRequest("sixkey_index1");
        //构建搜索条件
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        //查询条件，可以使用QueryBuilders工具来实现
        //QueryBuilders：termQuery 精确查询
        //QueryBuilders：matchAllQuery() 匹配所有
        TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("name", "test1");
        sourceBuilder.query(termQueryBuilder);
        searchRequest.source(sourceBuilder);

        //执行查询操作
        SearchResponse search = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
        for (SearchHit hit : search.getHits().getHits()) {
            System.out.println(hit.getSourceAsMap());
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

4、爬取数据jsoup

导入依赖

<!--解析网页-->
        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.10.2</version>
        </dependency>
1
2
3
4
5
6

工具类

/**
 * 网页解析工具类
 */
public class HtmlParseUtils {

    public static List<Content> getContent(String keyword) throws IOException {
        //获取url : https://search.jd.com/Search?keyword=javascript
        String url = "https://search.jd.com/Search?keyword=" + keyword;
        //获取浏览器document对象
        Document document = Jsoup.parse(new URL(url),30000);
        Element element = document.getElementById("J_goodsList");
        Elements elements = element.getElementsByTag("li");
        List<Content> list = new ArrayList<>();
        for (Element el : elements) {
            String src = el.getElementsByTag("img").get(0).attr("data-lazy-img");
            String price = el.getElementsByClass("p-price").get(0).text();
            String title = el.getElementsByClass("p-name").get(0).text();

            Content content = new Content();
            content.setSrc(src);
            content.setPrice(price);
            content.setTitle(title);
            list.add(content);
        }
        return list;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

5、爬京东实战

导入依赖

<dependencies>
        <!--解析网页-->
        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.10.2</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>2.0.21</version>
        </dependency>
    </dependencies>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

配置文件

server:
  port: 9090
1
2

ES配置，连接ES层

/**
 * Es配置
 */
@Configuration //注入到Spring中管理
public class EsConfig {

   @Bean
   public RestHighLevelClient restHighLevelClient(){
       RestHighLevelClient client = new RestHighLevelClient(
               RestClient.builder(
                       new HttpHost("127.0.0.1",9200,"http")
               ));
       return client;
   }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

常量

/**
 * 索引名称
 */
public class EsContest {

    public static final String INDEX_NAME = "jd_goods";
}
1
2
3
4
5
6
7

实体类

@Data
@NoArgsConstructor
@AllArgsConstructor
@Component
public class Content {

    private String src;
    private String price;
    private String title;
}
1
2
3
4
5
6
7
8
9
10

网页解析工具类

/**
 * 网页解析工具类
 */
public class HtmlParseUtils {

    public static List<Content> getContent(String keyword) throws IOException {
        //获取url : https://search.jd.com/Search?keyword=javascript
        String url = "https://search.jd.com/Search?keyword=" + keyword;
        //获取浏览器document对象
        Document document = Jsoup.parse(new URL(url),30000);
        Element element = document.getElementById("J_goodsList");
        Elements elements = element.getElementsByTag("li");
        List<Content> list = new ArrayList<>();
        for (Element el : elements) {
            String src = el.getElementsByTag("img").get(0).attr("data-lazy-img");
            String price = el.getElementsByClass("p-price").get(0).text();
            String title = el.getElementsByClass("p-name").get(0).text();

            Content content = new Content();
            content.setSrc(src);
            content.setPrice(price);
            content.setTitle(title);
            list.add(content);
        }
        return list;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

controller层

@RestController
public class ContentController {

    @Autowired
    private ContentService contentService;

    /**
     * 解析数据并插入到ES中
     * @param keyword
     * @return
     * @throws IOException
     */
    @GetMapping("/parse/{keyword}")
    public boolean putContent(@PathVariable("keyword") String keyword) throws IOException {
        return contentService.putContent(keyword);
    }

    @GetMapping("/search/{keyword}/{pageNo}/{pageSize}")
    public List<Map<String,Object>> searchPage(@PathVariable("keyword") String keyword,
                                               @PathVariable("pageNo") int pageNo,
                                               @PathVariable("pageSize") int pageSize) throws IOException {
        List<Map<String,Object>> contentList = contentService.searchPage(keyword,pageNo,pageSize);
        return contentList;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

实现类

@Service
public class ContentServiceImpl implements ContentService {

    @Autowired
    private RestHighLevelClient restHighLevelClient;

    /**
     * 解析数据并插入到ES中
     * @param keyword
     * @return
     * @throws IOException
     */
    @Override
    public boolean putContent(String keyword) throws IOException {
        //爬取数据
        List<Content> contentList = HtmlParseUtils.getContent(keyword);

        //向ES中批量插入数据
        BulkRequest bulkRequest = new BulkRequest();
        bulkRequest.timeout("2m");

        for (int i = 0; i < contentList.size(); i++) {
            bulkRequest.add(new IndexRequest(INDEX_NAME)
                    .source(JSON.toJSONString(contentList.get(i)), XContentType.JSON));
        }
        BulkResponse response = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
        return !response.hasFailures();
    }

    /**
     * 根据关键字分页查询ES中的数据
     * @param keyword
     * @param pageNo
     * @param pageSize
     * @return
     */
    @Override
    public List<Map<String, Object>> searchPage(String keyword, int pageNo, int pageSize) throws IOException {
        if(pageNo <= 1){
            pageNo = 1;
        }
        //条件搜索
        SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        //分页
        sourceBuilder.from(pageNo);
        sourceBuilder.size(pageSize);
        //构造查询条件
        TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("title", keyword);
        //查询
        sourceBuilder.query(termQueryBuilder);
        //高亮查询
        HighlightBuilder highlightBuilder = new HighlightBuilder();
        highlightBuilder.field("title");
        highlightBuilder.requireFieldMatch(false); //关闭多个高亮显示
        highlightBuilder.preTags("<span style='color: red'>");
        highlightBuilder.postTags("</span>");
        sourceBuilder.highlighter(highlightBuilder);

        //执行查询请求
        searchRequest.source(sourceBuilder);
        SearchResponse search = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
        //封装数据返回
        List<Map<String,Object>> searchList = new ArrayList<>();
        //解析结果
        for (SearchHit hit : search.getHits().getHits()) {
            //解析高亮字段,将原来的字段换成高亮字段即可
            Map<String, HighlightField> highlightFields = hit.getHighlightFields();
            //获取高亮字段
            HighlightField highlightTitle = highlightFields.get("title");
            //原来的结果
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            if(highlightTitle != null){
                Text[] fragments = highlightTitle.fragments();
                String new_title = "";
                for (Text text : fragments) {
                    new_title += text;
                }
                //将原来的字段换成高亮字段
                sourceAsMap.put("title",new_title);
            }
            searchList.add(sourceAsMap);
        }
        return searchList;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86