赞
踩
场景:在bug关联固件的时候将bug的数据放到固件的数据下,可以根据固件数据下是否包含bug数据查询出已关联和未关联的数据。
ES文档结构
目录
- {
- "query":{
- "nested":{
- "path":"s25_kernel_check",
- "query":{
- "bool":{
- "must":[
- {
- "match":{
- "s25_kernel_check.cve_id":"value"
- }
- },
- {
- "match":{
- "s25_kernel_check.exploit-db":"value"
- }
- }
- ]
- }
- }
- }
- }
- }
同时满足must下的条件才可以被返回
- BoolQueryBuilder must = QueryBuilders.boolQuery();
- if (cvesById.getCveId()!=null){
- must.must(QueryBuilders.matchQuery("s25_kernel_check.cve_id", cvesById.getCveId()));
- }
- if (cvesById.getExploitDb()!=null){
- must.must(QueryBuilders.matchQuery("s25_kernel_check.exploit-db", cvesById.getExploitDb()));
- }
-
- HashMap<String, Object> map2 = new HashMap<>();
- NestedQueryBuilder s25_kernel_check = QueryBuilders.nestedQuery("s25_kernel_check", must,ScoreMode.Max);
- SearchQuery queryBuilder = new NativeSearchQueryBuilder()
- .withQuery(s25_kernel_check)
- .withFields("firmware_id")
- .build();
- AggregatedPage<Cvesdd> page = template.queryForPage(queryBuilder, Cvesdd.class, new SearchResultMapper() {
- @Override
- public <T> AggregatedPage<T> mapResults(SearchResponse searchResponse, Class<T> aClass, Pageable pageable) {
- map2.put("total", searchResponse.getHits().totalHits);
- List<T> list = new ArrayList<>();
- return new AggregatedPageImpl<T>(list);
- }
- });
- Integer total= new Integer(String.valueOf((Long) map2.get("total")));
- if (total == 0){
- total = 10;
- }
- NestedQueryBuilder new_s25_kernel_check = QueryBuilders.nestedQuery("s25_kernel_check", must,ScoreMode.Max);
- SearchQuery new_queryBuilder = new NativeSearchQueryBuilder()
- .withQuery(new_s25_kernel_check)
- .withFields("firmware_id")
- .withPageable(PageRequest.of(0,total))
- .build();
- List<Cvesdd> list = template.queryForList(new_queryBuilder, Cvesdd.class);
- {
- "size": 200,
- "query": {
- "bool": {
- "must_not": [
- {
- "nested": {
- "path": "s25_kernel_check",
- "query": {
- "bool": {
- "must": [
- {
- "match": {
- "s25_kernel_check.cve_id": "value"
- }
- },
- {
- "match": {
- "s25_kernel_check.exploit-db": "value"
- }
- }
- ]
- }
- }
- }
- }
- ]
- }
- }
- }
排除同时满足must下条件的数据并返回
- HashMap<String, Object> totalMap = new HashMap<>();
- BoolQueryBuilder must = QueryBuilders.boolQuery();
- if (cvesById.getCveId()!=null){
- must.must(QueryBuilders.matchQuery("s25_kernel_check.cve_id", cvesById.getCveId()));
- }
- if (cvesById.getExploitDb()!=null){
- must.must(QueryBuilders.matchQuery("s25_kernel_check.exploit-db", cvesById.getExploitDb()));
- }
- NestedQueryBuilder nestedQuery = QueryBuilders.nestedQuery("s25_kernel_check", must,ScoreMode.Max);
- BoolQueryBuilder newBool = QueryBuilders.boolQuery();
- newBool.mustNot(nestedQuery);
- SearchQuery queryBuilder = new NativeSearchQueryBuilder()
- .withQuery(newBool)
- .withFields("firmware_id")
- .build();
- AggregatedPage<Cvesdd> page = template.queryForPage(queryBuilder, Cvesdd.class, new SearchResultMapper() {
- @Override
- public <T> AggregatedPage<T> mapResults(SearchResponse searchResponse, Class<T> aClass, Pageable pageable) {
- totalMap.put("total", searchResponse.getHits().totalHits);
- List<T> list = new ArrayList<>();
- return new AggregatedPageImpl<T>(list);
- }
- });
- Integer total= new Integer(String.valueOf((Long) totalMap.get("total")));
- SearchQuery new_queryBuilder = new NativeSearchQueryBuilder()
- .withQuery(newBool)
- .withFields("firmware_id")
- .withPageable(PageRequest.of(0,total))
- .build();
- List<Cvesdd> list = template.queryForList(new_queryBuilder, Cvesdd.class);
查询语句跟代码的对应关系
OK,上面是满足了我想要的效果,但是发现在关联后,在es中再增加一条不同任务但是固件已经被某条bug关联过的数据,就会发现某条bug的未关联固件和已关联固件都会出现这个固件,原因是es中其实是存放的任务跟固件的关联关系,又因为业务关系,会存在多条相同固件但是不同任务的数据,关联之后创建的新任务已有固件数据,其中并未包含某条bug的数据,所以才会导致这样,那么ok,在查询未关联的逻辑之前,先查询一下已经关联的数据,不管它在哪个任务,只要所有的固件数据,有一条满足包含此条bug数据,那么我们就认为此条固件已经关联了这个bug了
还尝试了排除嵌套查询满足或者id存在集合中的数据,gpt说这俩是and的关系,但是怎么测都是or的关系,现在es有三条id等于137的数据,其中有两条是满足嵌套查询的条件,如果是and那么应该是给我返回只是id等于137不满足嵌套查询的数据,但是并没有。。
但是如果将must_not改成must下面的这个两个就成了and的关系,就会返回同时满足这两个条件的数据
就很奇怪!离谱它妈给离谱开门,离谱到家了~
那么我们来看看gpt对疑问的回答
- {
- "size": 200,
- "query": {
- "bool": {
- "must_not": [
- {
- "terms": {
- "firmware_id": [
- 137
- ]
- }
- },
- {
- "nested": {
- "path": "s25_kernel_check",
- "query": {
- "bool": {
- "must": [
- {
- "match": {
- "s25_kernel_check.cve_id": "CVE-2010-3848,CVE-2010-3850,CVE-2010-4073"
- }
- },
- {
- "match": {
- "s25_kernel_check.exploit-db": "17787"
- }
- }
- ]
- }
- }
- }
- }
- ]
- }
- }
- }
最上面所提到的关联操作
- //查询es中要关联的固件数据
- BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
- boolQuery.must(QueryBuilders.matchQuery("firmware_id",ids[i]));
- SearchQuery searchQuery = new NativeSearchQueryBuilder()
- .withQuery(boolQuery)
- //指定索引
- .withIndices("information_result")
- //指定type
- .withTypes("fulldata")
- .build();
- List<?> results = template.query(searchQuery, response -> {
- List<Object> list = new ArrayList<>();
- for (SearchHit hit : response.getHits()) {
- Map<String, Object> sourceAsMap = hit.getSourceAsMap();
- sourceAsMap.put("id",hit.getId());
- list.add(sourceAsMap);
- }
- return list;
- });
-
- //构建批量插入的集合
- List<IndexQuery> queries = new ArrayList<>();
- for (int i1 = 0; i1 < results.size(); i1++) {
- Map oldData = (HashMap) results.get(i1);
- ArrayList s25_kernel_check1 =new ArrayList();
- boolean key1 = oldData.containsKey("s25_kernel_check");
- if (key1){
- s25_kernel_check1 = (ArrayList) oldData.get("s25_kernel_check");
- }
- //添加的bug信息
- HashMap<String, Object> stringObjectHashMap = new HashMap<>();
- //.....添加本条新bug数据
- s25_kernel_check1.add(stringObjectHashMap);
- oldData.put("s25_kernel_check",s25_kernel_check1);
- //构建插入数据
- IndexQuery indexQuery = new IndexQuery();
- //指定id,覆盖原有数据
- indexQuery.setId(oldData.get("id").toString());
- oldData.remove("id");
- JSONObject newData=new JSONObject(oldData);
- //插入数据
- indexQuery.setSource(newData.toString());
- //索引
- indexQuery.setIndexName("information_result");
- //类型
- indexQuery.setType("fulldata");
- //3.添加到queries
- queries.add(indexQuery);
- }
- //4.添加数据
- template.bulkIndex(queries);
- queries.clear();
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。