TF-IDF 概念+代码实现_tf-idf程序

TF-IDF

TF-IDF stands for “Term Frequency — Inverse Data Frequency”.

To put it in more formal mathematical terms, the TF-IDF score for the word t in the document d from the document set D is calculated as follows:

$$tfidf(t,d,D)=tf(t,d).idf(t,D)$$

Term Frequency (tf): gives us the frequency of the word in each document in the corpus. It is the ratio of number of times the word appears in a document compared to the total number of words in that document. It increases as the number of occurrences of that word within the document increases. Each document has its own tf.

$$tf(t,d)=\log (1+\mathrm{freq}(t,d))$$

Inverse Data Frequency (idf): used to calculate the weight of rare words across all documents in the corpus. The words that occur rarely in the corpus have a high IDF score. It is given by the equation below.(N is total number of documents.)

$$\mathrm{idf}(t,D)=\log \left(\frac{N}{\mathrm{count}(d\in D:t\in d)}\right)$$

Let’s take an example to get a clearer understanding.

Sentence 1 : The car is driven on the road.
Sentence 2: The truck is driven on the highway.

In this example, each sentence is a separate document.We will now calculate the TF-IDF for the above two documents, which represent our corpus.

From the above table, we can see that TF-IDF of common words was zero, which shows they are not significant. On the other hand, the TF-IDF of “car” , “truck”, “road”, and “highway” are non-zero. These words have more significance.

Implementing TF-IDF in Python

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

train = ['The sky is blue.','The sun is bright.']

tfidfvectorizer = TfidfVectorizer(analyzer='word',stop_words= 'english')
countvectorizer = CountVectorizer(analyzer= 'word', stop_words='english')
tfidf_wm = tfidfvectorizer.fit_transform(train)
tfidf_tokens = tfidfvectorizer.get_feature_names()
df_tfidfvect = pd.DataFrame(data = tfidf_wm.toarray(),index = ['Doc1','Doc2'],columns = tfidf_tokens)
weight=tfidf_wm.toarray()

for i in range(len(weight)):#打印每类文本的tf-idf词语权重，第一个for遍历所有文本，第二个for便利某一类文本下的词语权重  
    print ("-------这里输出第",i,u"类文本的词语tf-idf权重------" )
    for j in range(len(tfidf_tokens)):  
        print(tfidf_tokens[j],weight[i][j])
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# output
-------这里输出第 0 类文本的词语tf-idf权重------
blue 0.7071067811865476
bright 0.0
sky 0.7071067811865476
sun 0.0
-------这里输出第 1 类文本的词语tf-idf权重------
blue 0.0
bright 0.7071067811865476
sky 0.0
sun 0.7071067811865476
1
2
3
4
5
6
7
8
9
10
11

References

1. How to process textual data using TF-IDF in Python
2. TF-IDF Vectorizer scikit-learn