赞
踩
sklearn.feature_selection模块中的类可用于样本集的特征选择/降维,以提高估计器的准确性得分或提高其在非常高维的数据集上的性能。
VarianceThreshold是一种简单的特征选择基线方法。它删除方差不满足某个阈值的所有特征。默认情况下,它会删除所有零方差特征,即在所有样本中具有相同值的特征。注解:其实需要预先判断数据分布,计算方差阈值
示例:
例如,假设我们有一个具有布尔特征的数据集,并且我们希望在80%以上的样本中删除所有为1或为0(开或关)的特征。布尔特征是伯努利随机变量,这类变量的方差由
V
a
r
[
X
]
=
p
(
1
−
p
)
Var[X]=p(1-p)
Var[X]=p(1−p)
所以我们可以选择使用阈值
0.8
∗
(
1
−
0.8
)
0.8* (1 - 0.8)
0.8∗(1−0.8):
#代码:
from sklearn.feature_selection import VarianceThreshold
feature = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]
sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
feature_1 = sel.fit_transform(feature)
print(feature_1)
#[[0 1], [1 0], [0 0],[1 1],[1 0],[1 1]]
单变量特征选择的工作原理是根据单变量统计检验选择最佳特征。它可以被看作是一个预处理步骤估计。Scikit-learn将特性选择例程公开为实现转换方法的对象:
1、SelectKBest删除除k个得分最高的特征以外的所有特征
2、SelectPercentile删除除用户指定的得分百分比最高的特征外的所有特征
对每个特征使用常见的单变量统计检验:假阳性率SelectFpr、假发现率SelectFdr或簇错误SelectFwe。
3、GenericUnivariateSelect允许使用可配置策略执行单变量特征选择。这允许选择最佳的单变量选择策略与超参数搜索估计。
例如,我们可以对样本进行χ2检验,仅检索两个最佳特征,如下所示
from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
x_data, y = load_iris(return_X_y=True)
x_data_selected = SelectKBest(chi2, k=2).fit_transform(x_data, y)
这些对象以一个评分函数作为输入,该函数返回单变量分数和p值(或仅用于SelectKBest和SelectPercentile):注意不能混着用。同样可以用于稀疏数据,返回的也是稀疏数据:Feature selection with sparse data
1、回归
f_regression, mutual_info_regression
2、分类
chi2, f_classif, mutual_info_classif
示例:
使用单变量的特征选择,以提高分类精度的噪声数据集。
在本例中,一些嘈杂(非信息性)特征被添加到iris数据集。采用支持向量机(SVM)对单变量特征选择前后的数据集进行分类。对于每个特征,我们绘制单变量特征选择的p值和相应的SVM权重。有了这个,我们将比较模型的准确性,并研究单变量特征选择模型权重的影响。
# Generate sample data import numpy as np from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split # The iris dataset X, y = load_iris(return_X_y=True) # 添加噪声数据 E = np.random.RandomState(42).uniform(0, 0.1, size=(X.shape[0], 20)) X = np.hstack((X, E)) # 划分数据集 X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0) # 特征选择 from sklearn.feature_selection import SelectKBest, f_classif selector = SelectKBest(f_classif, k=4) selector.fit(X_train, y_train) scores = -np.log10(selector.pvalues_) scores /= scores.max() # %% import matplotlib.pyplot as plt X_indices = np.arange(X.shape[-1]) plt.figure(1) plt.clf() plt.bar(X_indices - 0.05, scores, width=0.2) plt.title("Feature univariate score") plt.xlabel("Feature number") plt.ylabel(r"Univariate score ($-Log(p_{value})$)") plt.show() #没有选择数据 from sklearn.pipeline import make_pipeline from sklearn.preprocessing import MinMaxScaler from sklearn.svm import LinearSVC clf = make_pipeline(MinMaxScaler(), LinearSVC()) clf.fit(X_train, y_train) print( "Classification accuracy without selecting features: {:.3f}".format( clf.score(X_test, y_test) ) ) svm_weights = np.abs(clf[-1].coef_).sum(axis=0) svm_weights /= svm_weights.sum() # 淑君选择后 clf_selected = make_pipeline(SelectKBest(f_classif, k=4), MinMaxScaler(), LinearSVC()) clf_selected.fit(X_train, y_train) print( "Classification accuracy after univariate feature selection: {:.3f}".format( clf_selected.score(X_test, y_test) ) ) svm_weights_selected = np.abs(clf_selected[-1].coef_).sum(axis=0) svm_weights_selected /= svm_weights_selected.sum() plt.bar( X_indices - 0.45, scores, width=0.2, label=r"Univariate score ($-Log(p_{value})$)" ) plt.bar(X_indices - 0.25, svm_weights, width=0.2, label="SVM weight") plt.bar( X_indices[selector.get_support()] - 0.05, svm_weights_selected, width=0.2, label="SVM weights after selection", ) plt.title("Comparing feature selection") plt.xlabel("Feature number") plt.yticks(()) plt.axis("tight") plt.legend(loc="upper right") plt.show() """ 结果分析:使用特征选择后,SVM准确率从0.79->0.87。而且从图中可以看到SVM的权重更关注原来的4个重要的特征,而不关注噪声。 """
示例:
Comparison of F-test and mutual information
这个例子说明了单变量F检验统计量和互信息之间的区别。
我们考虑均匀分布在[0,1]上的3个特征
x
1
,
x
2
,
x
3
x_1,x_2,x_3
x1,x2,x3,目标函数依赖于它们如下:
y
=
x
1
+
s
i
n
(
6
∗
p
i
∗
x
2
)
+
0.1
∗
N
(
0
,
1
)
y = x_1 + sin(6 * pi * x_2) + 0.1 * N(0, 1)
y=x1+sin(6∗pi∗x2)+0.1∗N(0,1)
即第三个特征是完全不相关的。
下面的代码绘制了
y
y
y对单个
x
i
x_i
xi的依赖关系以及单变量F检验统计量和互信息的归一化值。
import numpy as np import matplotlib.pyplot as plt from sklearn.feature_selection import f_regression, mutual_info_regression np.random.seed(0) X = np.random.rand(1000, 3) y = X[:, 0] + np.sin(6 * np.pi * X[:, 1]) + 0.1 * np.random.randn(1000) f_test, _ = f_regression(X, y) f_test /= np.max(f_test) mi = mutual_info_regression(X, y) mi /= np.max(mi) plt.figure(figsize=(15, 5)) for i in range(3): plt.subplot(1, 3, i + 1) plt.scatter(X[:, i], y, edgecolor="black", s=20) plt.xlabel("$x_{}$".format(i + 1), fontsize=14) if i == 0: plt.ylabel("$y$", fontsize=14) plt.title("F-test={:.2f}, MI={:.2f}".format(f_test[i], mi[i]), fontsize=16) plt.show()
分析:由于 F − t e s t F-test F−test只捕获了线性相关性,它将 x 1 x_1 x1列为最具鉴别力的特征。另一方面,互信息可以捕捉变量之间的任何类型的依赖关系,它将 x 2 x_2 x2评为最有鉴别力的特征,这可能更符合我们对这个例子的直觉感知。这两种方法都正确地将 x 3 x_3 x3标记为不相关。
给定一个外部的估计,分配权重的特征(例如,一个线性模型的系数),递归特征消除(RFE)的目标是选择特征递归考虑越来越小的特征集。首先,估计器在初始特征集上训练,每个特征的重要性通过任何特定属性(如coef_,feature_importance_)或可调用获得。然后,从当前的特征集合中删除最不重要的特征。该过程在修剪集上递归地重复,直到最终达到所需选择的特征数。
RFECV在交叉验证循环中执行RFE,以找到最佳的特征数量。
示例:
from sklearn.svm import SVC from sklearn.datasets import load_digits from sklearn.feature_selection import RFE import matplotlib.pyplot as plt # Load the digits dataset digits = load_digits() X = digits.images.reshape((len(digits.images), -1)) y = digits.target # Create the RFE object and rank each pixel svc = SVC(kernel="linear", C=1) rfe = RFE(estimator=svc, n_features_to_select=1, step=1) rfe.fit(X, y) ranking = rfe.ranking_.reshape(digits.images[0].shape) # Plot pixel ranking plt.matshow(ranking, cmap=plt.cm.Blues) plt.colorbar() plt.title("Ranking of pixels with RFE") plt.show()
结合网格搜索的pipline的示例:
import matplotlib.pyplot as plt from sklearn.svm import SVC from sklearn.model_selection import StratifiedKFold from sklearn.feature_selection import RFECV from sklearn.datasets import make_classification # Build a classification task using 3 informative features X, y = make_classification( n_samples=1000, n_features=25, n_informative=3, n_redundant=2, n_repeated=0, n_classes=8, n_clusters_per_class=1, random_state=0, ) # Create the RFE object and compute a cross-validated score. svc = SVC(kernel="linear") # The "accuracy" scoring shows the proportion of correct classifications min_features_to_select = 1 # Minimum number of features to consider rfecv = RFECV( estimator=svc, step=1, cv=StratifiedKFold(2), scoring="accuracy", min_features_to_select=min_features_to_select, ) rfecv.fit(X, y) print("Optimal number of features : %d" % rfecv.n_features_) # Plot number of features VS. cross-validation scores plt.figure() plt.xlabel("Number of features selected") plt.ylabel("Cross validation score (accuracy)") plt.plot( range(min_features_to_select, len(rfecv.grid_scores_) + min_features_to_select), rfecv.grid_scores_, ) plt.show()
1、L1范数惩罚的线性模型
受L1范数惩罚的线性模型具有稀疏的解:它们的许多估计系数为零。当目标是降低数据的维数以与另一个分类器一起使用时,它们可以与SelectFromModel一起使用以选择非零系数。特别是,用于此目的的稀疏估计是Lasso用于回归,LogisticRegression和LinearSVC用于分类:
from sklearn.svm import LinearSVC
from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectFromModel
X, y = load_iris(return_X_y=True)
X.shape
lsvc = LinearSVC(C=0.01, penalty="l1", dual=False).fit(X, y)
model = SelectFromModel(lsvc, prefit=True)
X_new = model.transform(X)
X_new.shape
L1和压缩感知
对于一个很好的alpha选择,Lasso可以完全恢复非零变量的精确集合,只需使用很少的观测,前提是满足某些特定条件。特别是,样本的数量应该“足够大”,或者L1模型将随机执行,其中“足够大”取决于非零系数的数目、特征数的对数、噪声量、非零参数的最小绝对值以及设计矩阵X的结构。此外,设计矩阵必须显示某些特定属性,例如相关性不太高。
对于非零系数的恢复,没有选择alpha参数的一般规则。它可以通过交叉验证(LassoCV或LassoLarsCV)来设置,尽管这可能会导致惩罚不足的模型:包括少量的非相关变量对预测得分是无害的。BIC(LassoLarsIC)则倾向于设置较高的alpha值。
2、基于树的
基于树的估计器(请参见sklearn.tree模块和sklear.ensemble模块中的森林)可用于计算基于杂质的特征重要性,而这反过来又可用于丢弃不相关的特征(与SelectFromModel元转换器结合使用时)
import matplotlib.pyplot as plt # 生成3个带有信息的特征的合成数据集 from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split X, y = make_classification( n_samples=1000, n_features=10, n_informative=3, n_redundant=0, n_repeated=0, n_classes=2, random_state=0, shuffle=False, ) X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42) # 随机森林分类器去拟合数据,计算特征重要性; from sklearn.ensemble import RandomForestClassifier feature_names = [f"feature {i}" for i in range(X.shape[1])] forest = RandomForestClassifier(random_state=0) forest.fit(X_train, y_train) # feature_importances_属性并且计算在每一棵树的不纯度减少的均值和方差。 #基于不纯特性的重要性可能会误导high cardinality特征(许多独特的值)。可以使用'permutation_importance'。 import time import numpy as np start_time = time.time() importances = forest.feature_importances_ std = np.std([tree.feature_importances_ for tree in forest.estimators_], axis=0) elapsed_time = time.time() - start_time print(f"Elapsed time to compute the importances: {elapsed_time:.3f} seconds") # %% # Let's plot the impurity-based importance. import pandas as pd forest_importances = pd.Series(importances, index=feature_names) fig, ax = plt.subplots() forest_importances.plot.bar(yerr=std, ax=ax) ax.set_title("Feature importances using MDI") ax.set_ylabel("Mean decrease in impurity") fig.tight_layout() # 可以看到,Permutation feature克服impurity-based feature importance的限制:没有对于high-cardinality features的偏差,并且可以计算留出的测试集。 from sklearn.inspection import permutation_importance start_time = time.time() result = permutation_importance( forest, X_test, y_test, n_repeats=10, random_state=42, n_jobs=2 ) elapsed_time = time.time() - start_time print(f"Elapsed time to compute the importances: {elapsed_time:.3f} seconds") forest_importances = pd.Series(result.importances_mean, index=feature_names) # 计算全部的permutation importance很费时,特征被混洗n次,并且模型重新训练去估计其重要性。画出重要性排序:importance ranking. fig, ax = plt.subplots() forest_importances.plot.bar(yerr=result.importances_std, ax=ax) ax.set_title("Feature importances using permutation on full model") ax.set_ylabel("Mean accuracy decrease") fig.tight_layout() plt.show() #两种方法的效果是相同的特征被检测到,虽然重要的程度不同,MDI更不可能比permutation importance去完全忽略一个特征。
示例:对于人脸数据集进行并行的随机森林进行特征筛选:
#这个例子展示了如何使用树林来评估杂质,基于人脸图像数据集分类任务中像素的重要性,像素越热,越重要。 #下面的代码演示了如何可以在多个作业中并行化构造和预测。 # 首先,加载olivetti faces dataset和限制数据集前面5类,然后训练随机森林并且评估impurity-based feature importance,有一个缺点是不能评估分离的测试集。对于这个例子,我们感兴趣的是表示从完整的数据集学习的信息。此外,我们将设置用于任务的内核数。 from sklearn.datasets import fetch_olivetti_faces n_jobs = -1 # 全部使用核心数,线程数 #加载数据集 data = fetch_olivetti_faces() X, y = data.data, data.target # 限制5类 mask = y < 5 X = X[mask] y = y[mask] # 随机森林分类器可以计算feature importances. from sklearn.ensemble import RandomForestClassifier forest = RandomForestClassifier(n_estimators=750, n_jobs=n_jobs, random_state=42) forest.fit(X, y) # %% # Feature importance based on mean decrease in impurity (MDI) # feature_importances_属性并且计算在每一棵树的不纯度减少的均值和方差。 #基于不纯特性的重要性可能会误导high cardinality特征(许多独特的值)。可以使用'permutation_importance'。 import time import matplotlib.pyplot as plt start_time = time.time() img_shape = data.images[0].shape importances = forest.feature_importances_ elapsed_time = time.time() - start_time print(f"Elapsed time to compute the importances: {elapsed_time:.3f} seconds") imp_reshaped = importances.reshape(img_shape) plt.matshow(imp_reshaped, cmap=plt.cm.hot) plt.title("Pixel importances using impurity values") plt.colorbar() plt.show() # 有限制的MDI对于这个数据集来说不是一个问题,因为: # 1、所有的特征是有序的数字和并且没有基数偏差cardinality bias; # 2、只对森林在训练集获得的表示知识感兴趣; #如果不满足上面两个条件,需要用sklearn.inspection.permutation_importance`
另一种选择特征的方法是使用SequentialFeatureSelector(SFS)。SFS是一个贪婪的过程,在每次迭代时,我们选择最佳的新特征添加到我们所选择的特征基于交叉验证得分。也就是说,我们从0个特征开始,选择得分最高的最佳单特征。这个过程是重复的,直到我们达到所需数量的选定特征。我们也可以反方向(向后SFS),即从所有的特征开始,贪婪地选择要一个一个删除的特征。我们在这里说明这两种方法。
有趣的是,向前和向后选择选择了相同的特征集。一般来说,情况并非如此,两种方法会导致不同的结果。
我们还注意到,SFS选择的特征不同于那些选择的特征重要性:SFS选择BMI,而不是s1。虽然这听起来很合理,因为根据系数,BMI对应于第三个最重要的特征。考虑到SFS完全没有使用系数,这是相当了不起的。
最后,我们应该注意到SelectFromModel比SFS快得多。事实上,SelectFromModel只需要拟合一个模型一次,而SFS需要为每个迭代交叉验证许多不同的模型。然而,SFS适用于任何模型,而SelectFromModel要求底层估计器公开coef_attribute或feature_importance_attribute。向前SFS比向后SFS快,因为它只需要执行n_features_to_select=2次迭代,而向后的SFS需要执行N_特征n_features_to_select=8次迭代次数
from sklearn.datasets import load_diabetes diabetes = load_diabetes() X, y = diabetes.data, diabetes.target print(diabetes.DESCR) # %% # Feature importance from coefficients # ------------------------------------ # # To get an idea of the importance of the features, we are going to use the # :class:`~sklearn.linear_model.RidgeCV` estimator. The features with the # highest absolute `coef_` value are considered the most important. # We can observe the coefficients directly without needing to scale them (or # scale the data) because from the description above, we know that the features # were already standardized. # For a more complete example on the interpretations of the coefficients of # linear models, you may refer to # :ref:`sphx_glr_auto_examples_inspection_plot_linear_model_coefficient_interpretation.py`. import matplotlib.pyplot as plt import numpy as np from sklearn.linear_model import RidgeCV ridge = RidgeCV(alphas=np.logspace(-6, 6, num=5)).fit(X, y) importance = np.abs(ridge.coef_) feature_names = np.array(diabetes.feature_names) plt.bar(height=importance, x=feature_names) plt.title("Feature importances via coefficients") plt.show() # %% # Selecting features based on importance # -------------------------------------- # # Now we want to select the two features which are the most important according # to the coefficients. The :class:`~sklearn.feature_selection.SelectFromModel` # is meant just for that. :class:`~sklearn.feature_selection.SelectFromModel` # accepts a `threshold` parameter and will select the features whose importance # (defined by the coefficients) are above this threshold. # # Since we want to select only 2 features, we will set this threshold slightly # above the coefficient of third most important feature. from sklearn.feature_selection import SelectFromModel from time import time threshold = np.sort(importance)[-3] + 0.01 tic = time() sfm = SelectFromModel(ridge, threshold=threshold).fit(X, y) toc = time() print(f"Features selected by SelectFromModel: {feature_names[sfm.get_support()]}") print(f"Done in {toc - tic:.3f}s") # Selecting features with Sequential Feature Selection # Another way of selecting features is to use from sklearn.feature_selection import SequentialFeatureSelector tic_fwd = time() sfs_forward = SequentialFeatureSelector( ridge, n_features_to_select=2, direction="forward" ).fit(X, y) toc_fwd = time() tic_bwd = time() sfs_backward = SequentialFeatureSelector( ridge, n_features_to_select=2, direction="backward" ).fit(X, y) toc_bwd = time() print( "Features selected by forward sequential selection: " f"{feature_names[sfs_forward.get_support()]}" ) print(f"Done in {toc_fwd - tic_fwd:.3f}s") print( "Features selected by backward sequential selection: " f"{feature_names[sfs_backward.get_support()]}" ) print(f"Done in {toc_bwd - tic_bwd:.3f}s")
clf = Pipeline([
('feature_selection', SelectFromModel(LinearSVC(penalty="l1"))),
('classification', RandomForestClassifier())
])
clf.fit(X, y)
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。