devbox
我家自动化
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

TF-IDF计算比较compare（gensim、jieba、sklearn、手工的异同）_结巴 tf-idf合并

作者：我家自动化 | 2024-04-06 02:15:14

赞

踩

结巴 tf-idf合并

一.概述

        TF-IDF（英文名: term frequency-inverse document frequency），引用百度百科的说法: TF-IDF是一种用于信息检索与数据挖掘的常用加权技术。

        TF意思是词频(Term Frequency)，用在句子构成的语料中，就是字或者词在文本中出现的频率。

                 一般计算是: TF = 字或词在句子中出现的次数 / 字或词在所有语料中出现的次数

        IDF意思是逆文本频率指数(Inverse Document Frequency)，就是出现该字或者词的句子条数。

                 一般计算是: IDF = Log ( 语料中句子总数 / (包含该词或字的句子数+1) )

        TF-IDF = TF * IDF

         这是前文介绍TF-IDF时候的说法，正巧面试也手撸了一次这个算法，真实环境是不是这样呢? 我们一探究竟。

          github地址: 

                           https://github.com/yongzhuo/Tookit-Sihui/tree/master/tookit_sample/tf_idf_compare

         本文主要介绍4中方案实现tf-idf，它们各有优点：

  1. 1.gensim
  2. 2.jieba
  3. 3.sklearn
  4. 4.by_hand(手动)

二.优缺点(推荐 sklearn)

            a. gensim： corpora生成token，doc2bow生成词袋模型，tfidf_model计算tf-idf，idfs可给出。

                                未出现的词语idfs等不计算在内，中规中矩的一个模型,可输入list或者文件地址等。

            b. jieba：     有idf.txt，即计算好的idf，未出现的词语使用平均idf(但对于句子来说,尤其是对单个单词的句子很不友好)。

            c. sklearn:    CountVectorizer统计词频，TfidfTransformer计算tfidf，csr_matrix数据格式压缩，

                                 可选择n-gram特征，可平滑处理，可选features，选择很多，还是推荐这个吧。

            d. by_hand:  手工版可配置, 批量计算词频字典与合并，可在小内存下计算大样本，比如说wikicorpus。

 

三.实现与最后代码说明

    2.1 gensim

  1. # -*- coding: UTF-8 -*-
  2. # !/usr/bin/python
  3. # @time     :2019/7/31 21:20
  4. # @author   :Mo
  5. # @function :
  6.  
  7. from gensim import corpora, models
  8. import jieba
  9.  
  10.  
  11.  
  12. def tfidf_from_questions(corpora_documents):
  13.     """
  14.         从文件读取并计算tf-idf
  15.     :param sources_path: 
  16.     :return: 
  17.     """
  18.     dictionary = corpora.Dictionary(corpora_documents)
  19.     corpus = [dictionary.doc2bow(text) for text in corpora_documents]
  20.     tfidf_model = models.TfidfModel(corpus)
  21.     return dictionary, tfidf_model
  22.  
  23.  
  24. def tfidf_from_corpora(sources_path):
  25.     """
  26.         从文件读取并计算tf-idf
  27.     :param sources_path: 
  28.     :return: 
  29.     """
  30.     from tookit_sihui.utils.file_utils import txt_read, txt_write
  31.     questions = txt_read(sources_path)
  32.     corpora_documents = []
  33.     for item_text in questions:
  34.         item_seg = list(jieba.cut(str(item_text).strip()))
  35.         corpora_documents.append(item_seg)
  36.  
  37.     dictionary = corpora.Dictionary(corpora_documents)
  38.     corpus = [dictionary.doc2bow(text) for text in corpora_documents]
  39.     tfidf_model = models.TfidfModel(corpus)
  40.     return dictionary, tfidf_model
  41.  
  42.  
  43. if __name__ == '__main__':
  44.     # test 1 from questions
  45.     corpora_documents = [['大漠', '帝国'],['紫色', 'Angle'],['花落', '惊', '飞羽'],
  46.                          ['我', 'm', 'o'], ['你', 'the', 'a', 'it', 'this']]
  47.     dictionary, tfidf_model = tfidf_from_questions(corpora_documents)
  48.     sentence = '大漠 大漠 大漠'
  49.     seg = list(jieba.cut(sentence))
  50.     bow = dictionary.doc2bow(seg)
  51.     tfidf_vec = tfidf_model[bow]
  52.     print(bow)
  53.     print(tfidf_vec)
  54.     bow = dictionary.doc2bow(['i', 'i', '大漠', '大漠', '大漠'])
  55.     tfidf_vec = tfidf_model[bow]
  56.     print(bow)
  57.     print(tfidf_vec)
  58.  
  59.     # test 2 from file of text
  60.     from tookit_sihui.conf.path_config import path_tf_idf_corpus
  61.     dictionary, tfidf_model = tfidf_from_corpora(path_tf_idf_corpus)
  62.     sentence = '大漠帝国'
  63.     seg = list(jieba.cut(sentence))
  64.     bow = dictionary.doc2bow(seg)
  65.     tfidf_vec = tfidf_model[bow]
  66.     print(bow)
  67.     print(tfidf_vec)
  68.     bow = dictionary.doc2bow(['sihui'])
  69.     tfidf_vec = tfidf_model[bow]
  70.     print(bow)
  71.     print(tfidf_vec)
  72.     gg = 0
  73.     # 结果
  74.     # [(12, 1)]
  75.     # [(12, 1.0)]
  76.     # []
  77.     # []
  78.     # [(172, 1), (173, 1)]
  79.     # [(172, 0.7071067811865475), (173, 0.7071067811865475)]
  80.     # []
  81.     # []
  82.  
  83.  
  84.  
  85. # # 说明:
  86. # 1.左边的是字典id,右边是词的tfidf,
  87. # 2.中文版停用词(如the)、单个字母(如i)等，不会去掉
  88. # 3.去除没有被训练到的词,如'sihui',没有出现就不会计算
  89. # 4.计算细节
  90. #   4.1 idf = add + log_{log\_base} \frac{totaldocs}{docfreq}, 如下:
  91.     # eps = 1e-12, idf只取大于eps的数字
  92.     def df2idf(docfreq, totaldocs, log_base=2.0, add=0.0):
  93.         import numpy as np
  94.         # np.log()什么都不写就以e为低, 由公式log(a)(b)=log(c)(b)/log(c)(a),
  95.         # 可得函数中为log(2)(totaldocs / docfreq)
  96.         # debug进去可以发现, 没有进行平滑处理, 即log(2)(文本数 / 词出现在文本中的个数),
  97.         # 这也很好理解, 因为如果输入为[],则不会给出模型,出现的文本中的至少出现一次,也没有必要加1了
  98.         return add + np.log(float(totaldocs) / docfreq) / np.log(log_base)
  99.         # 注意self.initialize(corpus)函数
  100. #   4.2 tf 从下面以及debug结果可以发现, gensim的tf取值是词频,
  101. #          也就是说出现几次就取几次,如句子'大漠 大漠 大漠', '大漠'的tf就取3
  102. #         termid_array, tf_array = [], []
  103. #         for termid, tf in bow:
  104. #             termid_array.append(termid)
  105. #             tf_array.append(tf)
  106. #
  107. #         tf_array = self.wlocal(np.array(tf_array))
  108. #
  109. #         vector = [
  110. #             (termid, tf * self.idfs.get(termid))
  111. #             for termid, tf in zip(termid_array, tf_array)
  112. #             if abs(self.idfs.get(termid, 0.0)) > self.eps
  113. #         ]
  114.  
  115.

 

    2.2 jieba

  1. # -*- coding: UTF-8 -*-
  2. # !/usr/bin/python
  3. # @time     :2019/7/31 21:21
  4. # @author   :Mo
  5. # @function :
  6.  
  7.  
  8. import jieba.analyse
  9. import jieba
  10.  
  11. sentence = '大漠 帝国 和 紫色 Angle'
  12. seg = jieba.cut(sentence)
  13. print(seg)
  14. tf_idf = jieba.analyse.extract_tags(sentence, withWeight=True)
  15. print(tf_idf)
  16.  
  17. # 结果
  18. # [('Angle', 2.988691875725), ('大漠', 2.36158258893), ('紫色', 2.10190405216), ('帝国', 1.605909794915)]
  19.  
  20.  
  21. # 说明,
  22. # 1.1 idf  jieba中的idf来自默认文件idf.txt,
  23. #          idf默认一段话来作为一个docunment,
  24. #          没出现过的词语的idf默认为所有idf的平均值,即为11.多
  25. #
  26. # 1.2 tf   tf只统计当前句子出现的频率除以所有词语数,
  27. #          例如'大漠 帝国 和 紫色 Angle'这句话, '大漠'的tf为1/5
  28. #          tfidf的停用词"和"去掉了
  29. #     tf计算代码
  30. #         freq[w] = freq.get(w, 0.0) + 1.0
  31. #         total = sum(freq.values())
  32. #         for k in freq:
  33. #             kw = k.word if allowPOS and withFlag else k
  34. #             freq[k] *= self.idf_freq.get(kw, self.median_idf) / total

 

    2.3 sklearn

  1. # -*- coding: UTF-8 -*-
  2. # !/usr/bin/python
  3. # @time     :2019/7/31 21:21
  4. # @author   :Mo
  5. # @function :
  6.  
  7.  
  8. from sklearn.feature_extraction.text import TfidfTransformer
  9. from sklearn.feature_extraction.text import CountVectorizer
  10.  
  11.  
  12. def tfidf_from_ngram(questions):
  13.     """
  14.         使用TfidfVectorizer计算n-gram
  15.     :param questions:list, like ['孩子气', '大漠帝国'] 
  16.     :return: 
  17.     """
  18.     from sklearn.feature_extraction.text import TfidfVectorizer
  19.     import jieba
  20.     def jieba_cut(x):
  21.         x = list(jieba.cut(x))
  22.         return ' '.join(x)
  23.     questions = [jieba_cut(''.join(ques)) for ques in questions]
  24.     tfidf_model = TfidfVectorizer(ngram_range=(1, 2), # n-gram特征, 默认(1,1)
  25.                                   max_features=10000,
  26.                                   token_pattern=r"(?u)\b\w+\b", # 过滤停用词
  27.                                   min_df=1,
  28.                                   max_df=0.9,
  29.                                   use_idf=1,
  30.                                   smooth_idf=1,
  31.                                   sublinear_tf=1)
  32.     tfidf_model.fit(questions)
  33.     print(tfidf_model.transform(['紫色 ANGEL 是 虾米 回事']))
  34.     return tfidf_model
  35.  
  36.  
  37. if __name__ == "__main__":
  38.     # test 1
  39.     corpora_documents = [['大漠', '帝国'], ['紫色', 'Angle'], ['花落', '惊', '飞羽'],
  40.                          ['我', 'm', 'o'], ['你', 'the', 'a', 'it', 'this'], ['大漠', '大漠']]
  41.     corpora_documents = [''.join(ques) for ques in corpora_documents]
  42.     # 统计词频
  43.     vectorizer = CountVectorizer()
  44.     # 初始化,fit和transformer   tf-idf
  45.     transformer = TfidfTransformer()
  46.     # 第一个fit_transform是计算tf-idf, 第二个是将文本转为词频矩阵
  47.     tfidf = transformer.fit_transform(vectorizer.fit_transform(corpora_documents))
  48.     print(tfidf)
  49.     # 模型所有词语
  50.     word = vectorizer.get_feature_names()
  51.     print(word)
  52.     weight = tfidf.toarray()
  53.     print(weight)
  54.     
  55.     
  56.     # test 2 from file of text
  57.     tf_idf_model = tfidf_from_ngram(corpora_documents)
  58.     print(tf_idf_model.transform(['你 谁 呀, 小老弟']))
  59.  
  60.  
  61.     #  sklearn的tfidf模型,可以采用TfidfVectorizer,提取n-gram特征,直接用于特征计算
  62.     #  和gensim一样, 都有TfidfVectorizer, 继承的是CountVectorizer
  63.     #             df += int(self.smooth_idf)        # 平滑处理
  64.     #             n_samples += int(self.smooth_idf) # 平滑处理
  65.     #             idf = np.log(n_samples / df) + 1  # 加了个1

    2.4 byhand

  1. # -*- coding: UTF-8 -*-
  2. # !/usr/bin/python
  3. # @time     :2019/6/19 21:32
  4. # @author   :Mo
  5. # @function :tf-idf
  6.  
  7.  
  8. from tookit_sihui.utils.file_utils import save_json
  9. from tookit_sihui.utils.file_utils import load_json
  10. from tookit_sihui.utils.file_utils import txt_write
  11. from tookit_sihui.utils.file_utils import txt_read
  12. import jieba
  13. import json
  14. import math
  15. import os
  16.  
  17.  
  18. from tookit_sihui.conf.logger_config import get_logger_root
  19. logger = get_logger_root()
  20.  
  21.  
  22. def count_tf(questions):
  23.     """
  24.       统计字频,或者词频tf
  25.     :param questions: list, 输入语料, 字级别的例子:[['我', '爱', '你'], ['爱', '护', '士']]
  26.     :return: dict, 返回字频,或者词频, 形式:{'我':1, '爱':2} 
  27.     """
  28.     tf_char = {}
  29.     for question in questions:
  30.         for char in question:
  31.             if char.strip():
  32.                 if char not in tf_char:
  33.                     tf_char[str(char).encode('utf-8', 'ignore').decode('utf-8')] = 1
  34.                 else:
  35.                     tf_char[str(char).encode('utf-8', 'ignore').decode('utf-8')] = tf_char[char] + 1
  36.     tf_char['[LENS]'] = sum([v for k,v in tf_char.items()])
  37.     return tf_char
  38.  
  39.  
  40. def count_idf(questions):
  41.     """
  42.       统计逆文档频率idf
  43.     :param questions: list, 输入语料, 字级别的例子:[['我', '爱', '你'], ['爱', '护', '士']]
  44.     :return: dict, 返回逆文档频率, 形式:{'我':1, '爱':2}
  45.     """
  46.     idf_char = {}
  47.     for question in questions:
  48.         question_set = set(question) # 在句子中，重复的只计数一次
  49.         for char in question_set:
  50.             if char.strip(): # ''不统计
  51.                 if char not in idf_char: # 第一次计数为1
  52.                     idf_char[char] = 1
  53.                 else:
  54.                     idf_char[char] = idf_char[char] + 1
  55.     idf_char['[LENS]'] = len(questions) # 保存一个所有的句子长度
  56.     return idf_char
  57.  
  58.  
  59. def count_tf_idf(freq_char, freq_document, ndigits=12, smooth =0):
  60.     """
  61.         统计tf-idf
  62.     :param freq_char: dict, tf
  63.     :param freq_document: dict, idf
  64.     :return: dict, tf-idf
  65.     """
  66.  
  67.     len_tf = freq_char['[LENS]']
  68.     len_tf_mid = int(len(freq_char)/2)
  69.     len_idf = freq_document['[LENS]']
  70.     len_idf_mid = int(len(freq_document) / 2)
  71.     # tf
  72.     tf_char = {}
  73.     for k2, v2 in freq_char.items():
  74.         tf_char[k2] = round((v2 + smooth)/(len_tf + smooth), ndigits)
  75.     # idf
  76.     idf_char = {}
  77.     for ki, vi in freq_document.items():
  78.         idf_char[ki] = round(math.log((len_idf + smooth) / (vi + smooth), 2), ndigits)
  79.     # tf-idf
  80.     tf_idf_char = {}
  81.     for kti, vti in freq_char.items():
  82.         tf_idf_char[kti] = round(tf_char[kti] * idf_char[kti], ndigits)
  83.  
  84.     # 删去文档数统计
  85.     tf_char.pop('[LENS]')
  86.     idf_char.pop('[LENS]')
  87.     tf_idf_char.pop('[LENS]')
  88.  
  89.     # 计算平均/最大/中位数
  90.     tf_char_values = tf_char.values()
  91.     idf_char_values = idf_char.values()
  92.     tf_idf_char_values = tf_idf_char.values()
  93.  
  94.     tf_char['[AVG]'] = round(sum(tf_char_values) / len_tf, ndigits)
  95.     idf_char['[AVG]'] = round(sum(idf_char_values) / len_idf, ndigits)
  96.     tf_idf_char['[AVG]'] = round(sum(tf_idf_char_values) / len_idf, ndigits)
  97.     tf_char['[MAX]'] = max(tf_char_values)
  98.     idf_char['[MAX]'] = max(idf_char_values)
  99.     tf_idf_char['[MAX]'] = max(tf_idf_char_values)
  100.     tf_char['[MIN]'] = min(tf_char_values)
  101.     idf_char['[MIN]'] = min(idf_char_values)
  102.     tf_idf_char['[MIN]'] = min(tf_idf_char_values)
  103.     tf_char['[MID]'] = sorted(tf_char_values)[len_tf_mid]
  104.     idf_char['[MID]'] = sorted(idf_char_values)[len_idf_mid]
  105.     tf_idf_char['[MID]'] = sorted(tf_idf_char_values)[len_idf_mid]
  106.  
  107.     return tf_char, idf_char, tf_idf_char
  108.  
  109.  
  110. def save_tf_idf_dict(path_dir, tf_char, idf_char, tf_idf_char):
  111.     """
  112.         排序和保存
  113.     :param path_dir:str, 保存文件目录 
  114.     :param tf_char: dict, tf
  115.     :param idf_char: dict, idf
  116.     :param tf_idf_char: dict, tf-idf
  117.     :return: None
  118.     """
  119.     if not os.path.exists(path_dir):
  120.         os.mkdir(path_dir)
  121.     # store and save
  122.     tf_char_sorted = sorted(tf_char.items(), key=lambda d: d[1], reverse=True)
  123.     tf_char_sorted = [tf[0] + '\t' + str(tf[1]) + '\n' for tf in tf_char_sorted]
  124.     txt_write(tf_char_sorted, path_dir + 'tf.txt')
  125.  
  126.     idf_char_sorted = sorted(idf_char.items(), key=lambda d: d[1], reverse=True)
  127.     idf_char_sorted = [idf[0] + '\t' + str(idf[1]) + '\n' for idf in idf_char_sorted]
  128.     txt_write(idf_char_sorted, path_dir + 'idf.txt')
  129.  
  130.     tf_idf_char_sorted = sorted(tf_idf_char.items(), key=lambda d: d[1], reverse=True)
  131.     tf_idf_char_sorted = [tf_idf[0] + '\t' + str(tf_idf[1]) + '\n' for tf_idf in tf_idf_char_sorted]
  132.     txt_write(tf_idf_char_sorted, path_dir + 'tf_idf.txt')
  133.  
  134.  
  135. def save_tf_idf_json(path_dir, tf_freq, idf_freq, tf_char, idf_char, tf_idf_char):
  136.     """
  137.         json排序和保存
  138.     :param path_dir:str, 保存文件目录 
  139.     :param tf_char: dict, tf
  140.     :param idf_char: dict, idf
  141.     :param tf_idf_char: dict, tf-idf
  142.     :return: None
  143.     """
  144.     if not os.path.exists(path_dir):
  145.         os.mkdir(path_dir)
  146.     # freq
  147.     save_json([tf_freq], path_dir + '/tf_freq.json')
  148.     save_json([idf_freq], path_dir + '/idf_freq.json')
  149.     # json_tf = json.dumps([tf_char])
  150.     save_json([tf_char], path_dir + '/tf.json')
  151.     # json_idf = json.dumps([idf_char])
  152.     save_json([idf_char], path_dir + '/idf.json')
  153.     # json_tf_idf = json.dumps([tf_idf_char])
  154.     save_json([tf_idf_char], path_dir + '/tf_idf.json')
  155.  
  156.  
  157. def load_tf_idf_json(path_tf_freq=None, path_idf_freq=None, path_tf=None, path_idf=None, path_tf_idf=None):
  158.     """
  159.         从json文件下载tf, idf, tf_idf
  160.     :param path_tf: 
  161.     :param path_idf: 
  162.     :param path_tf_idf: 
  163.     :return: 
  164.     """
  165.     json_tf_freq = load_json(path_tf_freq)
  166.     json_idf_freq = load_json(path_idf_freq)
  167.     json_tf = load_json(path_tf)
  168.     json_idf = load_json(path_idf)
  169.     json_tf_idf = load_json(path_tf_idf)
  170.     return json_tf_freq[0], json_idf_freq[0], json_tf[0], json_idf[0], json_tf_idf[0]
  171.  
  172.  
  173. def dict_add(dict1, dict2):
  174.     """
  175.       两个字典合并
  176.     :param dict1: 
  177.     :param dict2: 
  178.     :return: 
  179.     """
  180.     for i,j in dict2.items():
  181.         if i in dict1.keys():
  182.             dict1[i] += j
  183.         else:
  184.             dict1.update({f'{i}' : dict2[i]})
  185.     return dict1
  186.  
  187.  
  188. class TFIDF:
  189.     def __init__(self, questions=None, path_tf=None, 
  190.                  path_idf=None, path_tf_idf=None, 
  191.                  path_tf_freq=None, path_idf_freq=None,
  192.                  ndigits=12, smooth=0):
  193.         """
  194.             统计字频,或者词频tf
  195.         :param questions: list, 输入语料, 字级别的例子:[['我', '爱', '你'], ['爱', '护', '士']]
  196.         """
  197.         self.esplion = 1e-16
  198.         self.questions = questions
  199.         self.path_tf_freq = path_tf_freq
  200.         self.path_idf_freq = path_idf_freq
  201.         self.path_tf=path_tf
  202.         self.path_idf=path_idf
  203.         self.path_tf_idf=path_tf_idf
  204.         self.ndigits=ndigits
  205.         self.smooth=smooth
  206.         self.create_tfidf()
  207.  
  208.     def create_tfidf(self):
  209.         if self.questions != None: # 输入questions list, 即corpus语料
  210.             self.tf_freq = count_tf(self.questions)
  211.             self.idf_freq = count_idf(self.questions)
  212.             self.tf, self.idf, self.tfidf = count_tf_idf(self.tf_freq, 
  213.                                                          self.idf_freq, 
  214.                                                          ndigits=self.ndigits, 
  215.                                                          smooth =self.smooth)
  216.         else: # 输入训练好的
  217.             self.tf_freq, self.idf_freq, \
  218.             self.tf, self.idf, self.tfidf = load_tf_idf_json(path_tf_freq = self.path_tf_freq,
  219.                                                              path_idf_freq = self.path_idf_freq,
  220.                                                              path_tf=self.path_tf,
  221.                                                              path_idf=self.path_idf,
  222.                                                              path_tf_idf=self.path_tf_idf)
  223.         self.chars = [idf for idf in self.idf.keys()]
  224.  
  225.     def extract_tfidf_of_sentence(self, ques):
  226.         """
  227.             获取tf-idf
  228.         :param ques: str
  229.         :return: float
  230.         """
  231.         assert type(ques)==str
  232.         if not ques.strip():
  233.             return None
  234.         ques_list = list(jieba.cut(ques.replace(' ', '').strip()))
  235.         logger.info(ques_list)
  236.         score = 0.0
  237.         score_list = {}
  238.         for char in ques_list:
  239.             if char in self.chars:
  240.                 score = score + self.tfidf[char]
  241.                 score_list[char] = self.tfidf[char]
  242.             else: #
  243.                 score = score + self.esplion
  244.                 score_list[char] = self.esplion
  245.         score = score/len(ques_list)# 求平均避免句子长度不一的影响
  246.         logger.info(score_list)
  247.         logger.info({ques:score})
  248.         return score
  249.  
  250.     def extract_tf_of_sentence(self, ques):
  251.         """
  252.             获取idf
  253.         :param ques: str
  254.         :return: float
  255.         """
  256.         assert type(ques)==str
  257.         if not ques.strip():
  258.             return None
  259.         ques_list = list(jieba.cut(ques.replace(' ', '').strip()))
  260.         logger.info(ques_list)
  261.         score = 0.0
  262.         score_list = {}
  263.         for char in ques_list:
  264.             if char in self.chars:
  265.                 score = score + self.tf[char]
  266.                 score_list[char] = self.tf[char]
  267.             else: #
  268.                 score = score + self.esplion
  269.                 score_list[char] = self.esplion
  270.         score = score/len(ques_list)# 求平均避免句子长度不一的影响
  271.         logger.info(score_list)
  272.         logger.info({ques:score})
  273.         return score
  274.  
  275.     def extract_idf_of_sentence(self, ques):
  276.         """
  277.            获取idf
  278.         :param ques: str
  279.         :return: float
  280.         """
  281.         assert type(ques)==str
  282.         if not ques.strip():
  283.             return None
  284.         ques_list = list(jieba.cut(ques.replace(' ', '').strip()))
  285.         logger.info(ques_list)
  286.         score = 0.0
  287.         score_list = {}
  288.         for char in ques_list:
  289.             if char in self.chars:
  290.                 score = score + self.idf[char]
  291.                 score_list[char] = self.idf[char]
  292.             else: #
  293.                 score = score + self.esplion
  294.                 score_list[char] = self.esplion
  295.         score = score/len(ques_list) # 求平均避免句子长度不一的影响
  296.         logger.info(score_list)
  297.         logger.info({ques:score})
  298.         return score
  299.  
  300.  
  301. def create_TFIDF(path):
  302.     # 测试1,根据corpus生成
  303.     import time
  304.     time_start = time.time()
  305.     # 首先输入全部文本构建tf-idf,然后再拿去用
  306.     from tookit_sihui.conf.path_config import path_tf_idf_corpus
  307.     from tookit_sihui.utils.file_utils import txt_write, txt_read
  308.  
  309.     path_wiki = path if path else path_tf_idf_corpus
  310.     #  测试1, tf-idf, 调用
  311.     path_dir = 'tf_idf_freq/'
  312.     # ques = ['大漠帝国最强', '花落惊飞羽最漂亮', '紫色Angle最有气质', '孩子气最活泼', '口袋巧克力和过路蜻蜓最好最可爱啦', '历历在目最烦恼']
  313.     # questions = [list(q.strip()) for q in ques]
  314.     # questions = [list(jieba.cut(que)) for que in ques]
  315.     questions = txt_read(path_wiki)
  316.     len_questions = len(questions)
  317.     batch_size = 1000000
  318.     size_trade = len_questions // batch_size
  319.     print(size_trade)
  320.     size_end = size_trade * batch_size
  321.     # 计算tf-freq, idf-freq
  322.     ques_tf_all, ques_idf_all = {}, {}
  323.     for i, (start, end) in enumerate(zip(range(0, size_end, batch_size),
  324.                         range(batch_size, size_end, batch_size))):
  325.         print("第{}次".format(i))
  326.         question = questions[start: end]
  327.         questionss = [ques.strip().split(' ') for ques in question]
  328.         ques_idf = count_idf(questionss)
  329.         ques_tf = count_tf(questionss)
  330.         print('tf_idf_{}: '.format(i) + str(time.time() - time_start))
  331.         # 字典合并 values相加
  332.         ques_tf_all = dict_add(ques_tf_all, ques_tf)
  333.         ques_idf_all = dict_add(ques_idf_all, ques_idf)
  334.         print('dict_add_{}: '.format(i) + str(time.time() - time_start))
  335.         print('的tf:{}'.format(ques_tf_all['的']))
  336.         print('的idf:{}'.format(ques_idf_all['的']))
  337.     # 不足batch-size部分
  338.     if len_questions - size_end >0:
  339.         print("第{}次".format('last'))
  340.         question = questions[size_end: len_questions]
  341.         questionss = [ques.strip().split(' ') for ques in question]
  342.         ques_tf = count_idf(questionss)
  343.         ques_idf = count_tf(questionss)
  344.         # tf_char, idf_char, tf_idf_char = count_tf_idf(ques_tf, ques_idf)
  345.         ques_tf_all = dict_add(ques_tf_all, ques_tf)
  346.         ques_idf_all = dict_add(ques_idf_all, ques_idf)
  347.         print('{}: '.format('last') + str(time.time() - time_start))
  348.         print('的tf:{}'.format(ques_tf_all['的']))
  349.         print('的idf:{}'.format(ques_idf_all['的']))
  350.     # 计算tf-idf
  351.     tf_char, idf_char, tf_idf_char = count_tf_idf(ques_tf_all, ques_idf_all)
  352.     print(len(tf_char))
  353.     print('tf-idf ' + str(time.time()-time_start))
  354.     print('tf-idf ok!')
  355.     # 保存, tf,idf,tf-idf
  356.     save_tf_idf_json(path_dir, ques_tf_all, ques_idf_all, tf_char, idf_char, tf_idf_char)
  357.     gg=0
  358.  
  359.  
  360. if __name__=="__main__":
  361.     # 测试1
  362.     path = None # 语料地址, 格式为切分后的句子, 例如'孩子 气 和 紫色 angle'
  363.     create_TFIDF(path)
  364.  
  365.     # # 测试2, 调用class、json, input预测
  366.     # path_dir = 'tf_idf_freq/'
  367.     # path_tf = path_dir + '/tf.json'
  368.     # path_idf = path_dir + '/idf.json'
  369.     # path_tf_idf = path_dir + '/tf_idf.json'
  370.     #
  371.     # tfidf = TFIDF(path_tf=path_tf, path_idf=path_idf, path_tf_idf=path_tf_idf)
  372.     # score1 = tfidf.extract_tf_of_sentence('大漠帝国')
  373.     # score2 = tfidf.extract_idf_of_sentence('大漠帝国')
  374.     # score3 = tfidf.extract_tfidf_of_sentence('大漠帝国')
  375.     # print('tf: ' + str(score1))
  376.     # print('idf: ' + str(score2))
  377.     # print('tfidf: ' + str(score3))
  378.     # while True:
  379.     #     print("请输入: ")
  380.     #     ques = input()
  381.     #     tfidf_score = tfidf.extract_tfidf_of_sentence(ques)
  382.     #     print('tfidf:' + str(tfidf_score))
  383.

 

希望对你有所帮助!


 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/我家自动化/article/detail/369203
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号