NLP实验六：TF-IDF_nlpttf

实验目的

学习华为云的自然语言处理课程，了解并实现TF-IDF的代码

实验内容

• 学习华为云自然语言处理教程。
• 理解并学习TF-IDF，用代码实现TF-IDF的计算过程
• 按照视频内容来写代码就行，主要涉及的是jieba分词和TF-IDF的学习
• 需要自己找数据集和间断词

代码

import jieba
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer

with open("dataset.txt", 'r', encoding='utf-8') as fenci:
    text = fenci.read()

new_text = jieba.cut(text, cut_all=False)

str_out = ' '.join(new_text).replace('，', '').replace('。', '').replace('？', '').replace('！', '').replace('《', '') \
    .replace('》', '').replace('‘', '').replace('“', '').replace('”', '').replace('：', '').replace('；', '').replace('【','')\
    .replace('】','').replace('、', '').replace('（', '').replace('）', '').replace('···', '').replace('——', '').replace('-', '')\
    .replace('(', '').replace(')', '').replace('/', '').replace('\n', '').replace('[', '').replace(']', '').replace('"', '')

fo = open("dataset_cut.txt", 'w', encoding='utf-8')
fo.write(str_out)
fo.close()

# open files
with open("dataset_cut.txt", 'r', encoding='utf-8') as f:
    dataset = list(f.readlines())
    print(len(dataset))
    print(dataset)

# open stopwords text
with open("stopwords.txt", 'r', encoding='utf-8') as f:
    stopwords = list(f.read().replace('\n', ' ').split())
    print(stopwords)

vectorizer = CountVectorizer(stop_words=stopwords, min_df=0)
transformer = TfidfTransformer()
tfidf = transformer.fit_transform(vectorizer.fit_transform(dataset))
word = vectorizer.get_feature_names_out()
print("word:", word)
print(vectorizer.vocabulary_)

weight = tfidf.toarray()
print("weight:", weight)

word_weight = list()
for i in range(len(word)):
    print("————这里输出第%d类文本的词语tf-idf权重" % i)
    print("     ", word[i], weight[0][i])

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

结果展示

数据集和间断词：

分词和权重：

实验数据（含数据集和间断词）

点击此处获取所有代码