devbox
花生_TL007
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

apdplat.word.WordSegmenter分词功能使用自有词库，实现过滤功能，可是实际上，导致的结果差强人意，没办法只能使用JDK的自带过滤的功能_org.apdplat.word

作者：花生_TL007 | 2024-05-22 21:41:09

赞

踩

org.apdplat.word

WOrd的分词功能，自定义的词库，可以使用自定义的，可是实际上自带的词库实在是无法删除，导致的分词的效果很差劲

  1.  
  2. import com.alibaba.fastjson.JSON;
  3. import org.apache.commons.lang3.StringUtils;
  4. import org.apdplat.word.WordSegmenter;
  5. import org.apdplat.word.dictionary.DictionaryFactory;
  6. import org.apdplat.word.segmentation.Word;
  7. import org.apdplat.word.util.WordConfTools;
  8.  
  9. import java.util.ArrayList;
  10. import java.util.List;
  11.  
  12. /********************************************
  13.  * 模块名称: 主要功能是做标题分词的操作，工具类
  14.  * 功能说明: 
  15.  * 开发人员:  
  16.  * 开发时间:2020/8/29 12:21   
  17.  * v1.0.0.0 2020/8/29-01    
  18.  *******************************************/
  19.  
  20. public class WordPartitionUtils {
  21.  
  22.  
  23.     public static void main(String[] args) {
  24.         //分词效果加载词库
  25.         DictionaryFactory.getDictionary().clear();
  26.         List<String> parameterList = new ArrayList<>();
  27.         parameterList.add("对决");
  28.         DictionaryFactory.getDictionary().addAll(parameterList);
  29.         //词典
  30.        WordConfTools.set("dic.path", "classpath:word/custom.txt");
  31.        //词性标注数据
  32.        WordConfTools.set("part.of.speech.dic.path", "classpath:word/part_of_speech.txt");
  33.        //词性说明数据
  34.        WordConfTools.set("part.of.speech.des.path", "classpath:word/part_of_speech_des.txt");
  35.        //二元模型
  36.        WordConfTools.set("bigram.path", "classpath:word/bigram.txt");
  37.        //三元模型
  38.        WordConfTools.set("trigram.path", "classpath:word/trigram.txt");
  39.        //停用词词典
  40.        WordConfTools.set("stopwords.path", "classpath:word/stopwords.txt");
  41.        //用于分割词的标点符号
  42.        WordConfTools.set("punctuation.path", "classpath:word/punctuation.txt");
  43.        //百家姓
  44.        WordConfTools.set("surname.path", "classpath:word/surname.txt");
  45.        //数量词
  46.        WordConfTools.set("quantifier.path", "classpath:word/quantifier.txt");
  47.         //     WordConfTools.forceOverride("classpath:custom.txt");
  48. //        WordConfTools.set("dic.path", "classpath:dic.txt，classpath:custom.txt");
  49.         DictionaryFactory.reload();
  50.         String title = "<刺猬索尼克>曝正片片段,音速小子上演高萌对决";
  51.         List<Word> list = WordSegmenter.seg(title);
  52.         String value = WordConfTools.get("dic.path");
  53.  
  54.         System.out.println(JSON.toJSONString(list));
  55.         System.out.println("value =" + value);
  56.  
  57.     }
  58.  
  59.  
  60.     /**
  61.      * 针对【标题不含QYJC（企业简称） 且 标题不含负面关键词 且 标题不含重要关键词 且 dsCode为转化率低于50%的栏目】进行过滤
  62.      *
  63.      * @param title  入参 标题
  64.      * @param dsCode 资讯的编码
  65.      * @return false 不满足条件，true满足条件
  66.      */
  67.     public Boolean isContionWord(String title, String dsCode, List<String> parameterDsCodeList) {
  68.  
  69.         Boolean wordFlag = false;
  70.         List<Word> list = WordSegmenter.seg(title);
  71.         for (Word word : list) {
  72.             if (word.getPartOfSpeech() != null && word.getPartOfSpeech().getPos().equals("i")) {
  73.                 if (StringUtils.isNotBlank(word.getText())) { //匹配上的关键字
  74.                     wordFlag = true;
  75. //                    log.error("【Word分词标题为】：{}，【匹配上关键字】:{}", title, word.getText());
  76.                 } else {
  77. //                    log.error("【Word分词标题为】：{}，【匹配关键字-无】", title);
  78.                 }
  79.                 break;
  80.             }
  81.         }
  82.         if (wordFlag && parameterDsCodeList.contains(dsCode)) {
  83.             return true;
  84.         }
  85.         return false;
  86.     }

运行结果：

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
[{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"des":"未知","pos":"i"},"synonym":[],"text":"刺"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"猬"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"des":"","pos":"nr"},"synonym":[],"text":"索尼克"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"曝"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"正"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"片"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"片段"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"音"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"速"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"小"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"子"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"上演"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"des":"","pos":"nr"},"synonym":[],"text":"高萌对"},{"acronymPinYin":"","antonym":[],"frequency":0,"fullPinYin":"","partOfSpeech":{"$ref":"$[0].partOfSpeech"},"synonym":[],"text":"决"}]
value =classpath:word/custom.txt

使用Word分词来实现文本的过滤，效果耗时是单位数；

 

使用JDK的过滤stream流式来实现文本的过滤，效果耗时是单位数；差异不大

  1. SELECT
  2.     t.keyword AS '标题',
  3.     t.tag_count AS '耗时（毫秒）',
  4.     t.tags AS '过滤方式',
  5.     t.remark AS '返回匹配结果',
  6.     t.is_add AS '结果0 false 1 true',
  7.     t.xwbt AS '返回结果',
  8.     t.mtcc AS '数据编码',
  9.     t.update_time AS '操作时间'
  10. FROM
  11.     tbm_news_log t where  t.tags='WORD'
  12. ORDER BY
  13.     t.id DESC   limit 1000;
  14. SELECT
  15.     t.keyword AS '标题',
  16.     t.tag_count AS '耗时（毫秒）',
  17.     t.tags AS '过滤方式',
  18.     t.remark AS '返回匹配结果',
  19.     t.is_add AS '结果0 false 1 true',
  20.     t.xwbt AS '返回结果',
  21.     t.mtcc AS '数据编码',
  22.     t.update_time AS '操作时间'
  23. FROM
  24.     tbm_news_log t where  t.tags='JDKCONTAINS'
  25. ORDER BY
  26.     t.id DESC  limit 1000;

 

综上是redis先缓存8万条数据，然后进行过滤，

测试1000条数据的标题过滤效果如截图，差异不明显。

 

 

依赖pom.xml

 

  1. <!-- https://mvnrepository.com/artifact/com.janeluo/ikanalyzer -->
  2. <dependency>
  3.     <groupId>com.janeluo</groupId>
  4.     <artifactId>ikanalyzer</artifactId>
  5.     <version>2012_u6</version>
  6.     <exclusions>
  7.         <exclusion>
  8.             <artifactId>lucene-queryparser</artifactId>
  9.             <groupId>org.apache.lucene</groupId>
  10.         </exclusion>
  11.         <exclusion>
  12.             <artifactId>lucene-core</artifactId>
  13.             <groupId>org.apache.lucene</groupId>
  14.         </exclusion>
  15.         <exclusion>
  16.         <groupId>org.slf4j</groupId>
  17.         <artifactId>slf4j-api</artifactId>
  18.         </exclusion>
  19.     </exclusions>
  20. </dependency>
  21.  
  22.  
  23. <dependency>
  24.     <groupId>org.apdplat</groupId>
  25.     <artifactId>word</artifactId>
  26.     <version>${apdplat.word.version}</version>
  27.     <exclusions>
  28.         <exclusion>
  29.             <artifactId>lucene-queryparser</artifactId>
  30.             <groupId>org.apache.lucene</groupId>
  31.         </exclusion>
  32.         <exclusion>
  33.             <artifactId>lucene-core</artifactId>
  34.             <groupId>org.apache.lucene</groupId>
  35.         </exclusion>
  36.         <exclusion>
  37.             <groupId>org.slf4j</groupId>
  38.             <artifactId>slf4j-api</artifactId>
  39.         </exclusion>
  40.         <exclusion>
  41.             <groupId>org.apache.lucene</groupId>
  42.             <artifactId>lucene-analyzers-common</artifactId>
  43.         </exclusion>
  44.     </exclusions>
  45. </dependency>

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/花生_TL007/article/detail/610018
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号