热门标签
热门文章
- 1汽车统一诊断服务UDS协议参考文章和视频
- 2《MATLAB图像处理实例详解》：学习第四天（2022.09.29）
- 3DataScience：数据科学家必掌握知识之从数据结构→数据库→数据仓库→数据湖→数据中台→数据ETL技术的各个知识点的区别与总结_数据中台数据科学
- 4机器学习深度学习毕业设计选题合集
- 5Linux本地搭建FastDFS系统_linux搭建文件上传系统
- 6rasa spacyNLP使用方式
- 7SpringCloud学习笔记(黑马)(五)——gateway网关组件_spring.cloud.gateway.routes[1].id=service-edu spri
- 82020-12-07_english bruce lin博客
- 9故障诊断——迁移学习整理_迁移学习故障诊断
- 10人工智能-简介_人工智能介绍csdn
当前位置: article > 正文
三种文本特征提取（TF-IDF/Word2Vec/CountVectorizer）及Spark MLlib调用实例（Scala/Java/python）_python word2vec tfidf 提取特征
作者：不正经 | 2024-04-02 19:43:24
赞
踩
python word2vec tfidf 提取特征
Spark MLlib 提供三种文本特征提取方法，分别为TF-IDF、Word2Vec以及CountVectorizer其各自原理与调用代码整理如下：
TF-IDF
算法介绍：
词频－逆向文件频率（TF-IDF）是一种在文本挖掘中广泛使用的特征向量化方法，它可以体现一个文档中词语在语料库中的重要程度。
词语由t表示，文档由
d表示，语料库由
D表示。词频TF(t,,d)
是词语
t在文档
d中出现的次数。文件频率DF(t,D)是包含词语
的文档的个数。如果我们只使用词频来衡量重要性，很容易过度强调在文档中经常出现而并没有包含太多与文档有关的信息的词语，比如“a”，“the”以及“of”。如果一个词语经常出现在语料库中，它意味着它并没有携带特定的文档的特殊信息。逆向文档频率数值化衡量词语提供多少信息：
其中，|D|是语料库中的文档总数。由于采用了对数，如果一个词出现在所有的文件，其
IDF值变为0。
在下面的代码段中，我们以一组句子开始。首先使用分解器Tokenizer把句子划分为单个词语。对每一个句子（词袋），我们使用HashingTF将句子转换为特征向量，最后使用IDF重新调整特征向量。这种转换通常可以提高使用文本特征的性能。然后，我们的特征向量可以在算法学习中[plain]view plaincopy
- import org.apache.spark.ml.feature.{HashingTF, IDF, Tokenizer}
- val sentenceData = spark.createDataFrame(Seq(
- (0, “Hi I heard about Spark”),
- (0, “I wish Java could use case classes”),
- (1, “Logistic regression models are neat”)
- )).toDF(“label”, “sentence”)
- val tokenizer = new Tokenizer().setInputCol(“sentence”).setOutputCol(“words”)
- val wordsData = tokenizer.transform(sentenceData)
- val hashingTF = new HashingTF()
- .setInputCol(“words”).setOutputCol(“rawFeatures”).setNumFeatures(20)
- val featurizedData = hashingTF.transform(wordsData)
- // CountVectorizer也可获取词频向量
- val idf = new IDF().setInputCol(“rawFeatures”).setOutputCol(“features”)
- val idfModel = idf.fit(featurizedData)
- val rescaledData = idfModel.transform(featurizedData)
- rescaledData.select(“features”, “label”).take(3).foreach(println)
- import java.util.Arrays;
- import java.util.List;
- import org.apache.spark.ml.feature.HashingTF;
- import org.apache.spark.ml.feature.IDF;
- import org.apache.spark.ml.feature.IDFModel;
- import org.apache.spark.ml.feature.Tokenizer;
- import org.apache.spark.ml.linalg.Vector;
- import org.apache.spark.sql.Dataset;
- import org.apache.spark.sql.Row;
- import org.apache.spark.sql.RowFactory;
- import org.apache.spark.sql.SparkSession;
- import org.apache.spark.sql.types.DataTypes;
- import org.apache.spark.sql.types.Metadata;
- import org.apache.spark.sql.types.StructField;
- import org.apache.spark.sql.types.StructType;
- List<Row> data = Arrays.asList(
- RowFactory.create(0.0, “Hi I heard about Spark”),
- RowFactory.create(0.0, “I wish Java could use case classes”),
- RowFactory.create(1.0, “Logistic regression models are neat”)
- );
- StructType schema = new StructType(new StructField[]{
- new StructField(“label”, DataTypes.DoubleType, false, Metadata.empty()),
- new StructField(“sentence”, DataTypes.StringType, false, Metadata.empty())
- });
- Dataset<Row> sentenceData = spark.createDataFrame(data, schema);
- Tokenizer tokenizer = new Tokenizer().setInputCol(“sentence”).setOutputCol(“words”);
- Dataset<Row> wordsData = tokenizer.transform(sentenceData);
- int numFeatures = 20;
- HashingTF hashingTF = new HashingTF()
- .setInputCol(”words”)
- .setOutputCol(”rawFeatures”)
- .setNumFeatures(numFeatures);
- Dataset<Row> featurizedData = hashingTF.transform(wordsData);
- // CountVectorizer也可获取词频向量
- IDF idf = new IDF().setInputCol(“rawFeatures”).setOutputCol(“features”);
- IDFModel idfModel = idf.fit(featurizedData);
- Dataset<Row> rescaledData = idfModel.transform(featurizedData);
- for (Row r : rescaledData.select(“features”, “label”).takeAsList(3)) {
- Vector features = r.getAs(0);
- Double label = r.getDouble(1);
- System.out.println(features);
- System.out.println(label);
- }
- from pyspark.ml.feature import HashingTF, IDF, Tokenizer
- sentenceData = spark.createDataFrame([
- (0, “Hi I heard about Spark”),
- (0, “I wish Java could use case classes”),
- (1, “Logistic regression models are neat”)
- ], [”label”, “sentence”])
- tokenizer = Tokenizer(inputCol=”sentence”, outputCol=“words”)
- wordsData = tokenizer.transform(sentenceData)
- hashingTF = HashingTF(inputCol=”words”, outputCol=“rawFeatures”, numFeatures=20)
- featurizedData = hashingTF.transform(wordsData)
- # CountVectorizer也可获取词频向量
- idf = IDF(inputCol=”rawFeatures”, outputCol=“features”)
- idfModel = idf.fit(featurizedData)
- rescaledData = idfModel.transform(featurizedData)
- for features_label in rescaledData.select(“features”, “label”).take(3):
- print(features_label)
Word2Vec
算法介绍：
Word2vec是一个Estimator，它采用一系列代表文档的词语来训练word2vecmodel。该模型将每个词语映射到一个固定大小的向量。word2vecmodel使用文档中每个词语的平均数来将文档转换为向量，然后这个向量可以作为预测的特征，来计算文档相似度计算等等。
在下面的代码段中，我们首先用一组文档，其中每一个文档代表一个词语序列。对于每一个文档，我们将其转换为一个特征向量。此特征向量可以被传递到一个学习算法。
调用：
Scala:
-
<span style=
"font-family:SimSun;font-size:14px;">
import org.apache.spark.ml.feature.Word2Vec
-
-
// Input data: Each row is a bag of words from a sentence or document.
-
val documentDF = spark.createDataFrame(
Seq(
-
"Hi I heard about Spark".
split(
" "),
-
"I wish Java could use case classes".
split(
" "),
-
"Logistic regression models are neat".
split(
" ")
-
).
map(
Tuple1.apply)).toDF(
"text")
-
-
// Learn a mapping from words to Vectors.
-
val word2Vec = new
Word2Vec()
-
.setInputCol(
"text")
-
.setOutputCol(
"result")
-
.setVectorSize(
3)
-
.setMinCount(
0)
-
val model = word2Vec.fit(documentDF)
-
val result = model.transform(documentDF)
-
result.select(
"result").take(
3).foreach(
println)</span>
- 1
Java:
-
import java.util.Arrays;
-
import java.util.List;
-
-
import org.apache.spark.ml.feature.Word2Vec;
-
import org.apache.spark.ml.feature.Word2VecModel;
-
import org.apache.spark.sql.Dataset;
-
import org.apache.spark.sql.Row;
-
import org.apache.spark.sql.RowFactory;
-
import org.apache.spark.sql.SparkSession;
-
import org.apache.spark.sql.types.*;
-
-
// Input data: Each row is a bag of words from a sentence or document.
-
List<Row> data = Arrays.asList(
-
RowFactory.create(Arrays.asList(
"Hi I heard about Spark".split(
" "))),
-
RowFactory.create(Arrays.asList(
"I wish Java could use case classes".split(
" "))),
-
RowFactory.create(Arrays.asList(
"Logistic regression models are neat".split(
" ")))
-
);
-
StructType schema =
new StructType(
new StructField[]{
-
new StructField(
"text",
new ArrayType(DataTypes.StringType,
true),
false, Metadata.empty())
-
});
-
Dataset<Row> documentDF = spark.createDataFrame(data, schema);
-
-
// Learn a mapping from words to Vectors.
-
Word2Vec word2Vec =
new Word2Vec()
-
.setInputCol(
"text")
-
.setOutputCol(
"result")
-
.setVectorSize(
3)
-
.setMinCount(
0);
-
Word2VecModel model = word2Vec.fit(documentDF);
-
Dataset<Row> result = model.transform(documentDF);
-
for (Row r : result.select(
"result").takeAsList(
3)) {
-
System.out.println(r);
-
}
- 1
Python:
-
from pyspark.ml.feature import Word2Vec
-
-
# Input data: Each row is a bag of words from a sentence or document.
-
documentDF = spark.createDataFrame([
-
(
"Hi I heard about Spark".split(
" "), ),
-
(
"I wish Java could use case classes".split(
" "), ),
-
(
"Logistic regression models are neat".split(
" "), )
-
], [
"text"])
-
# Learn a mapping from words to Vectors.
-
word2Vec = Word2Vec(vectorSize=
3, minCount=
0, inputCol=
"text", outputCol=
"result")
-
model = word2Vec.fit(documentDF)
-
result = model.transform(documentDF)
-
for feature
in result.select(
"result").take(
3):
-
print(feature)
- 1
Countvectorizer
算法介绍：
Countvectorizer和Countvectorizermodel旨在通过计数来将一个文档转换为向量。当不存在先验字典时，Countvectorizer可作为Estimator来提取词汇，并生成一个Countvectorizermodel。该模型产生文档关于词语的稀疏表示，其表示可以传递给其他算法如LDA。
在fitting过程中，countvectorizer将根据语料库中的词频排序选出前vocabsize个词。一个可选的参数minDF也影响fitting过程中，它指定词汇表中的词语在文档中最少出现的次数。另一个可选的二值参数控制输出向量，如果设置为真那么所有非零的计数为1。这对于二值型离散概率模型非常有用。
示例：
假设我们有如下的DataFrame包含id和texts两列：
id | texts
—-|———-
0 |Array(“a”, “b”, “c”)
1 |Array(“a”, “b”, “b”, “c”,”a”)
文本中的每一行都是一个文档类型的数组(字符串)。调用的CountVectorizer产生词汇(a,b,c)的CountVectorizerModel，转换后的输出向量如下：
id | texts | vector
—-|———————————|—————
0 |Array(“a”, “b”, “c”) | (3,[0,1,2],[1.0,1.0,1.0])
1 |Array(“a”, “b”, “b”, “c”,”a”) |(3,[0,1,2],[2.0,2.0,1.0])
每个向量代表文档的词汇表中每个词语出现的次数。
调用：
Scala：
-
<span style=
"font-family:SimSun;font-size:14px;">
import org.apache.spark.ml.feature.{CountVectorizer, CountVectorizerModel}
-
-
val df = spark.createDataFrame(Seq(
-
(
0,
Array(
"a",
"b",
"c")),
-
(
1,
Array(
"a",
"b",
"b",
"c",
"a"))
-
)).toDF(
"id",
"words")
-
-
// fit a CountVectorizerModel from the corpus
-
val cvModel: CountVectorizerModel =
new CountVectorizer()
-
.setInputCol(
"words")
-
.setOutputCol(
"features")
-
.setVocabSize(
3)
-
.setMinDF(
2)
-
.fit(df)
-
-
// alternatively, define CountVectorizerModel with a-priori vocabulary
-
val cvm =
new CountVectorizerModel(
Array(
"a",
"b",
"c"))
-
.setInputCol(
"words")
-
.setOutputCol(
"features")
-
-
cvModel.transform(df).select(
"features").show()<
/span>
- 1
Java:
-
import java.util.Arrays;
-
import java.util.List;
-
-
import org.apache.spark.ml.feature.CountVectorizer;
-
import org.apache.spark.ml.feature.CountVectorizerModel;
-
import org.apache.spark.sql.Dataset;
-
import org.apache.spark.sql.Row;
-
import org.apache.spark.sql.RowFactory;
-
import org.apache.spark.sql.SparkSession;
-
import org.apache.spark.sql.types.*;
-
-
// Input data: Each row is a bag of words from a sentence or document.
-
List<Row> data = Arrays.asList(
-
RowFactory.create(Arrays.asList(
"a",
"b",
"c")),
-
RowFactory.create(Arrays.asList(
"a",
"b",
"b",
"c",
"a"))
-
);
-
StructType schema =
new StructType(
new StructField [] {
-
new StructField(
"text",
new ArrayType(DataTypes.StringType,
true),
false, Metadata.empty())
-
});
-
Dataset<Row> df = spark.createDataFrame(data, schema);
-
-
// fit a CountVectorizerModel from the corpus
-
CountVectorizerModel cvModel =
new CountVectorizer()
-
.setInputCol(
"text")
-
.setOutputCol(
"feature")
-
.setVocabSize(
3)
-
.setMinDF(
2)
-
.fit(df);
-
-
// alternatively, define CountVectorizerModel with a-priori vocabulary
-
CountVectorizerModel cvm =
new CountVectorizerModel(
new String[]{
"a",
"b",
"c"})
-
.setInputCol(
"text")
-
.setOutputCol(
"feature");
-
-
cvModel.transform(df).show();
- 1
Python:
-
from pyspark.ml.feature import CountVectorizer
-
-
# Input data: Each row is a bag of words with a ID.
-
df = spark.createDataFrame([
-
(
0,
"a b c".split(
" ")),
-
(
1,
"a b b c a".split(
" "))
-
], [
"id",
"words"])
-
-
# fit a CountVectorizerModel from the corpus.
-
cv = CountVectorizer(inputCol=
"words", outputCol=
"features", vocabSize=
3, minDF=
2.0)
-
model = cv.fit(df)
-
result = model.transform(df)
-
result.show()
- 1
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/不正经/article/detail/353518
推荐阅读
相关标签
