devbox
小惠珠哦
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Hanlp之文本分类

作者：小惠珠哦 | 2024-08-01 14:00:19

赞

踩

Hanlp之文本分类

1、语料库格式

  1. 分类语料的根目录.目录必须满足如下结构:<br>
  2.                      根目录<br>
  3.                      ├── 分类A<br>
  4.                      │   └── 1.txt<br>
  5.                      │   └── 2.txt<br>
  6.                      │   └── 3.txt<br>
  7.                      ├── 分类B<br>
  8.                      │   └── 1.txt<br>
  9.                      │   └── ...<br>
  10.                      └── ...<br>
  11.  文件不一定需要用数字命名,也不需要以txt作为后缀名,但一定需要是文本文件.

2、项目格式

训练分类语料库要放到data/test/ 目录下

3、代码

(1)TestUtility类

  1. /*
  2.  * <author>Han He</author>
  3.  * <email>me@hankcs.com</email>
  4.  * <create-date>2018-06-23 11:05 PM</create-date>
  5.  *
  6.  * <copyright file="TestUtility.java">
  7.  * Copyright (c) 2018, Han He. All Rights Reserved, http://www.hankcs.com/
  8.  * This source is subject to Han He. Please contact Han He for more information.
  9.  * </copyright>
  10.  */
  11. package com.cn.test.TextClassification;
  12.  
  13. import com.hankcs.hanlp.HanLP;
  14.  
  15. import java.io.*;
  16. import java.net.HttpURLConnection;
  17. import java.net.URL;
  18. import java.util.zip.ZipEntry;
  19. import java.util.zip.ZipInputStream;
  20.  
  21. /**
  22.  * @author hankcs
  23.  */
  24. public class TestUtility
  25. {
  26.     static
  27.     {
  28.         ensureFullData();
  29.     }
  30.  
  31.     public static void ensureFullData()
  32.     {
  33.         ensureData(HanLP.Config.PerceptronCWSModelPath, "http://nlp.hankcs.com/download.php?file=data", HanLP.Config.PerceptronCWSModelPath.split("data")[0], false);
  34.     }
  35.  
  36.     /**
  37.      * 保证 name 存在，不存在时自动下载解压
  38.      *
  39.      * @param name 路径
  40.      * @param url  下载地址
  41.      * @return name的绝对路径
  42.      */
  43.     public static String ensureData(String name, String url)
  44.     {
  45.         return ensureData(name, url, null, true);
  46.     }
  47.  
  48.     /**
  49.      * 保证 name 存在，不存在时自动下载解压
  50.      *
  51.      * @param name 路径
  52.      * @param url  下载地址
  53.      * @return name的绝对路径
  54.      */
  55.     public static String ensureData(String name, String url, String parentPath, boolean overwrite)
  56.     {
  57.         File target = new File(name);
  58.         if (target.exists()) return target.getAbsolutePath();
  59.         try
  60.         {
  61.             File parentFile = parentPath == null ? new File(name).getParentFile() : new File(parentPath);
  62.             if (!parentFile.exists()) parentFile.mkdirs();
  63.             String filePath = downloadFile(url, parentFile.getAbsolutePath());
  64.             if (filePath.endsWith(".zip"))
  65.             {
  66.                 unzip(filePath, parentFile.getAbsolutePath(), overwrite);
  67.             }
  68.             return target.getAbsolutePath();
  69.         }
  70.         catch (Exception e)
  71.         {
  72.             System.err.printf("数据下载失败，请尝试手动下载 %s 到 %s 。原因如下：\n", url, target.getAbsolutePath());
  73.             e.printStackTrace();
  74.             System.exit(1);
  75.             return null;
  76.         }
  77.     }
  78.  
  79.     /**
  80.      * 保证 data/test/name 存在
  81.      *
  82.      * @param name
  83.      * @param url
  84.      * @return
  85.      */
  86.     public static String ensureTestData(String name, String url)
  87.     {
  88.         return ensureData(String.format("data/test/%s", name), url);
  89.     }
  90.  
  91.     /**
  92.      * Downloads a file from a URL
  93.      *
  94.      * @param fileURL  HTTP URL of the file to be downloaded
  95.      * @param savePath path of the directory to save the file
  96.      * @throws IOException
  97.      * @author www.codejava.net
  98.      */
  99.     public static String downloadFile(String fileURL, String savePath)
  100.         throws IOException
  101.     {
  102.         System.err.printf("Downloading %s to %s\n", fileURL, savePath);
  103.         HttpURLConnection httpConn = request(fileURL);
  104.         while (httpConn.getResponseCode() == HttpURLConnection.HTTP_MOVED_PERM || httpConn.getResponseCode() == HttpURLConnection.HTTP_MOVED_TEMP)
  105.         {
  106.             httpConn = request(httpConn.getHeaderField("Location"));
  107.         }
  108.  
  109.         // always check HTTP response code first
  110.         if (httpConn.getResponseCode() == HttpURLConnection.HTTP_OK)
  111.         {
  112.             String fileName = "";
  113.             String disposition = httpConn.getHeaderField("Content-Disposition");
  114.             String contentType = httpConn.getContentType();
  115.             int contentLength = httpConn.getContentLength();
  116.  
  117.             if (disposition != null)
  118.             {
  119.                 // extracts file name from header field
  120.                 int index = disposition.indexOf("filename=");
  121.                 if (index > 0)
  122.                 {
  123.                     fileName = disposition.substring(index + 10,
  124.                                                      disposition.length() - 1);
  125.                 }
  126.             }
  127.             else
  128.             {
  129.                 // extracts file name from URL
  130.                 fileName = new File(httpConn.getURL().getPath()).getName();
  131.             }
  132.  
  133. //            System.out.println("Content-Type = " + contentType);
  134. //            System.out.println("Content-Disposition = " + disposition);
  135. //            System.out.println("Content-Length = " + contentLength);
  136. //            System.out.println("fileName = " + fileName);
  137.  
  138.             // opens input stream from the HTTP connection
  139.             InputStream inputStream = httpConn.getInputStream();
  140.             String saveFilePath = savePath;
  141.             if (new File(savePath).isDirectory())
  142.                 saveFilePath = savePath + File.separator + fileName;
  143.             String realPath;
  144.             if (new File(saveFilePath).isFile())
  145.             {
  146.                 System.err.printf("Use cached %s instead.\n", fileName);
  147.                 realPath = saveFilePath;
  148.             }
  149.             else
  150.             {
  151.                 saveFilePath += ".downloading";
  152.  
  153.                 // opens an output stream to save into file
  154.                 FileOutputStream outputStream = new FileOutputStream(saveFilePath);
  155.  
  156.                 int bytesRead;
  157.                 byte[] buffer = new byte[4096];
  158.                 long start = System.currentTimeMillis();
  159.                 int progress_size = 0;
  160.                 while ((bytesRead = inputStream.read(buffer)) != -1)
  161.                 {
  162.                     outputStream.write(buffer, 0, bytesRead);
  163.                     long duration = (System.currentTimeMillis() - start) / 1000;
  164.                     duration = Math.max(duration, 1);
  165.                     progress_size += bytesRead;
  166.                     int speed = (int) (progress_size / (1024 * duration));
  167.                     float ratio = progress_size / (float) contentLength;
  168.                     float percent = ratio * 100;
  169.                     int eta = (int) (duration / ratio * (1 - ratio));
  170.                     int minutes = eta / 60;
  171.                     int seconds = eta % 60;
  172.  
  173.                     System.err.printf("\r%.2f%%, %d MB, %d KB/s, ETA %d min %d s", percent, progress_size / (1024 * 1024), speed, minutes, seconds);
  174.                 }
  175.                 System.err.println();
  176.                 outputStream.close();
  177.                 realPath = saveFilePath.substring(0, saveFilePath.length() - ".downloading".length());
  178.                 if (!new File(saveFilePath).renameTo(new File(realPath)))
  179.                     throw new IOException("Failed to move file");
  180.             }
  181.             inputStream.close();
  182.             httpConn.disconnect();
  183.  
  184.             return realPath;
  185.         }
  186.         else
  187.         {
  188.             httpConn.disconnect();
  189.             throw new IOException("No file to download. Server replied HTTP code: " + httpConn.getResponseCode());
  190.         }
  191.     }
  192.  
  193.     private static HttpURLConnection request(String url) throws IOException
  194.     {
  195.         HttpURLConnection httpConn = (HttpURLConnection) new URL(url).openConnection();
  196.         httpConn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");
  197.         return httpConn;
  198.     }
  199.  
  200.     private static void unzip(String zipFilePath, String destDir, boolean overwrite)
  201.     {
  202.         System.err.println("Unzipping to " + destDir);
  203.         File dir = new File(destDir);
  204.         // create output directory if it doesn't exist
  205.         if (!dir.exists()) dir.mkdirs();
  206.         FileInputStream fis;
  207.         //buffer for read and write data to file
  208.         byte[] buffer = new byte[4096];
  209.         try
  210.         {
  211.             fis = new FileInputStream(zipFilePath);
  212.             ZipInputStream zis = new ZipInputStream(fis);
  213.             ZipEntry ze = zis.getNextEntry();
  214.             while (ze != null)
  215.             {
  216.                 String fileName = ze.getName();
  217.                 File newFile = new File(destDir + File.separator + fileName);
  218.                 if (overwrite || !newFile.exists())
  219.                 {
  220.                     if (ze.isDirectory())
  221.                     {
  222.                         //create directories for sub directories in zip
  223.                         newFile.mkdirs();
  224.                     }
  225.                     else
  226.                     {
  227.                         new File(newFile.getParent()).mkdirs();
  228.                         FileOutputStream fos = new FileOutputStream(newFile);
  229.                         int len;
  230.                         while ((len = zis.read(buffer)) > 0)
  231.                         {
  232.                             fos.write(buffer, 0, len);
  233.                         }
  234.                         fos.close();
  235.                         //close this ZipEntry
  236.                         zis.closeEntry();
  237.                     }
  238.                 }
  239.                 ze = zis.getNextEntry();
  240.             }
  241.             //close last ZipEntry
  242.             zis.closeEntry();
  243.             zis.close();
  244.             fis.close();
  245.             new File(zipFilePath).delete();
  246.         }
  247.         catch (IOException e)
  248.         {
  249.             e.printStackTrace();
  250.         }
  251.     }
  252. }

(2)ModelTrain类

  1. package com.cn.test.TextClassification;
  2.  
  3. import com.hankcs.hanlp.classification.classifiers.IClassifier;
  4. import com.hankcs.hanlp.classification.classifiers.NaiveBayesClassifier;
  5. import com.hankcs.hanlp.classification.models.NaiveBayesModel;
  6. import com.hankcs.hanlp.corpus.io.IOUtil;
  7.  
  8. import java.io.File;
  9.  
  10.  
  11. public class ModelTrain {
  12.     /**
  13.      * 搜狗文本分类语料库5个类目，每个类目下1000篇文章，共计5000篇文章
  14.      */
  15.     public static final String CORPUS_FOLDER = TestUtility.ensureTestData("搜狗文本分类语料库迷你版", "");
  16.  
  17.     /**
  18.      * 模型保存路径
  19.      */
  20.     public static final String MODEL_PATH = "data/test/classification-model.ser";
  21.  
  22.  
  23.     public static NaiveBayesModel trainOrLoadModel()
  24.     {
  25.         NaiveBayesModel model = (NaiveBayesModel) IOUtil.readObjectFrom(MODEL_PATH);
  26.         if (model != null) return model;
  27.  
  28.         File corpusFolder = new File(CORPUS_FOLDER);
  29.         if (!corpusFolder.exists() || !corpusFolder.isDirectory())
  30.         {
  31.             System.err.println("没有文本分类语料!");
  32.             System.exit(1);
  33.         }
  34.         try{
  35.             IClassifier classifier = new NaiveBayesClassifier(); // 创建分类器，更高级的功能请参考IClassifier的接口定义
  36.             classifier.train(CORPUS_FOLDER);                     // 训练后的模型支持持久化，下次就不必训练了
  37.             model = (NaiveBayesModel) classifier.getModel();
  38.             IOUtil.saveObjectTo(model, MODEL_PATH);
  39.         }catch (Exception e){
  40.             e.printStackTrace();
  41.         }
  42.         return model;
  43.     }
  44. }

(3)InitOneObject类

  1. package com.cn.test.TextClassification;
  2.  
  3. import com.hankcs.hanlp.classification.classifiers.IClassifier;
  4. import com.hankcs.hanlp.classification.classifiers.NaiveBayesClassifier;
  5.  
  6. public class InitOneObject {
  7.     public static final InitOneObject instance= new InitOneObject();
  8.     //获取classifier对象,训练后的模型支持持久化，下次就不必训练了.
  9.     public IClassifier classifier = new NaiveBayesClassifier(ModelTrain.trainOrLoadModel());
  10. }

(4)TrainMain类

  1. package com.cn.test.TextClassification;
  2.  
  3. import com.hankcs.hanlp.classification.classifiers.IClassifier;
  4.  
  5. import java.io.IOException;
  6.  
  7. public class TrainMain {
  8.     public static void main(String[] args) throws IOException
  9.     {
  10.         //IClassifier classifier = new NaiveBayesClassifier(trainOrLoadModel());
  11.         IClassifier classifier= InitOneObject.instance.classifier;
  12.         predict(classifier, "C罗获2018环球足球奖最佳球员 德尚荣膺最佳教练");
  13.         predict(classifier, "英国造航母耗时8年仍未服役 被中国速度远远甩在身后");
  14.         predict(classifier, "研究生考录模式亟待进一步专业化");
  15.         predict(classifier, "如果真想用食物解压,建议可以食用燕麦");
  16.         predict(classifier, "锄禾日当午，汗滴禾下土");
  17.     }
  18.  
  19.     private static void predict(IClassifier classifier, String text)
  20.     {
  21.         System.out.printf("《%s》 属于分类 【%s】\n", text, classifier.classify(text));
  22.     }
  23. }

4、运行结果

  1. 模式:训练集
  2. 文本编码:UTF-8
  3. 根目录:C:\MyselfApplication\MyProject\HanLp\data\test\搜狗文本分类语料库迷你版
  4. 加载中...
  5. [体育]...100.00% 1000 篇文档
  6. [健康]...100.00% 1000 篇文档
  7. [军事]...100.00% 1000 篇文档
  8. [教育]...100.00% 1000 篇文档
  9. [汽车]...100.00% 1000 篇文档
  10. 耗时 15868 ms 加载了 5 个类目,共 5000 篇文档
  11. 原始数据集大小:5000
  12. 使用卡方检测选择特征中...耗时 189 ms,选中特征数:18156 / 80986 = 22.42%
  13. 贝叶斯统计结束
  14. 《C罗获2018环球足球奖最佳球员 德尚荣膺最佳教练》 属于分类 【体育】
  15. 《英国造航母耗时8年仍未服役 被中国速度远远甩在身后》 属于分类 【军事】
  16. 《研究生考录模式亟待进一步专业化》 属于分类 【教育】
  17. 《如果真想用食物解压,建议可以食用燕麦》 属于分类 【健康】
  18. 《锄禾日当午，汗滴禾下土》 属于分类 【健康】

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/小惠珠哦/article/detail/914415
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号