【六 （5）机器学习-分类任务-kaggle泰坦尼克号宇宙飞船实战】

文章导航

【一 简明数据分析进阶路径介绍（文章导航）】

一、什么是分类任务

分类任务是机器学习中的一个基本任务，其核心目的是根据输入数据的特征，将数据分配到预先定义的类别或标签中。在分类任务中，首先会收到一个包含多个属性的数据集，每个数据点或记录都由一组属性（特征）和一个类标号（目标属性）组成，类标号是用于指示数据点所属类别的一组离散值。分类任务的目标是学习一个目标函数（也称为分类模型），该函数能够将数据集中的每个属性集映射到一个特定的类标号。这个分类模型可以用于描述性建模，解释不同类中的对象，也可以用于预测性建模，预测新数据的类标号。分类技术适用于处理二元或标称类型的数据集，但不适用于序数数据，因为分类技术不考虑类之间的序关系。分类任务在自然语言处理（NLP）、计算机视觉（CV）等多个领域有着广泛的应用，例如垃圾邮件分类、情感分析、图像识别等。

二、分类任务常见算法

三、代码实现

字段说明

PassengerId - 每位乘客的唯一 ID。每个 Id 采用 gggg_pp 的形式，其中 gggg 表示乘客旅行的组，pp 是他们在组中的编号。群体中的人通常是家庭成员，但并非总是如此。
HomePlanet - 乘客离开的星球，通常是他们的永久居住星球。
CryoSleep - 指示乘客是否选择在航行期间进入假死状态。处于低温睡眠状态的乘客被限制在他们的客舱内。
Cabin - 乘客入住的客舱编号。采用deck/num/side 形式，其中side 可以是P 代表左舷或S 代表右舷。
Destination - 乘客将要去的星球。
Age - 乘客的年龄。
VIP - 乘客在航程中是否支付了特殊的 VIP 服务费用。
RoomService、FoodCourt、ShoppingMall、Spa、VRDeck - 乘客在泰坦尼克号宇宙飞船的众多豪华设施中所支付的金额。
Name - 乘客的名字和姓氏。
Transported - 乘客是否被运送到另一个维度。这是目标，您要预测的列。
1
2
3
4
5
6
7
8
9
10

1、导入类库

# 导入类库
import numpy as np
import pandas as pd
import scipy.stats as stats

import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px  

import warnings
warnings.filterwarnings('ignore')
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import RobustScaler

from sklearn.decomposition import PCA
from sklearn.model_selection import cross_val_score, GridSearchCV, KFold

from sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin
from sklearn.base import clone
from sklearn.linear_model import Lasso
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor, ExtraTreesRegressor
from sklearn.svm import SVR, LinearSVR
from sklearn.linear_model import ElasticNet, SGDRegressor, BayesianRidge
from sklearn.kernel_ridge import KernelRidge
from xgboost import XGBRegressor
import catboost
from catboost import CatBoostClassifier
from bayes_opt import BayesianOptimization

# 显示中文
plt.rcParams['font.sans-serif'] = ['SimHei']
plt.rcParams['axes.unicode_minus'] = False

# pandas显示所有行和列 
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

2、读取数据

test = pd.read_csv('test.csv')
train = pd.read_csv('train.csv')
train.head()
1
2
3

3、查看缺失值和数据类型

train.info()
1

输出：

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 8693 entries, 0 to 8692
Data columns (total 14 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   PassengerId   8693 non-null   object 
 1   HomePlanet    8492 non-null   object 
 2   CryoSleep     8476 non-null   object 
 3   Cabin         8494 non-null   object 
 4   Destination   8511 non-null   object 
 5   Age           8514 non-null   float64
 6   VIP           8490 non-null   object 
 7   RoomService   8512 non-null   float64
 8   FoodCourt     8510 non-null   float64
 9   ShoppingMall  8485 non-null   float64
 10  Spa           8510 non-null   float64
 11  VRDeck        8505 non-null   float64
 12  Name          8493 non-null   object 
 13  Transported   8693 non-null   bool   
dtypes: bool(1), float64(6), object(7)
memory usage: 891.5+ KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

4、特征衍生

Cabin - 乘客入住的客舱编号。采用deck/num/side形式，可以拆分为deck/num/side三个特征。

RoomService/FoodCourt/hoppingMall/Spa/VRDeck是乘客支付的各项费用，可以衍生出总消费金额特征

train[['deck','num', 'side']] = train['Cabin'].str.split('/', expand=True)
test[['deck','num', 'side']] = test['Cabin'].str.split('/', expand=True)

train.drop('Cabin', axis=1, inplace=True)
test.drop('Cabin', axis=1, inplace=True)

col_to_sum = ['RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck']
train['SumSpends'] = train[col_to_sum].sum(axis=1)
test['SumSpends'] = test[col_to_sum].sum(axis=1)
train.head()
1
2
3
4
5
6
7
8
9
10

5、缺失值处理

# 看到每个变量都有少量缺失
def audit_missing_values(df):  
    audit_result = {}  
    for col in df.columns:  
        count = df[col].isnull().sum()  
        ratio = count / len(df) if len(df) > 0 else 0  
        audit_result[col] = {'缺失数量': count, '缺失百分比': ratio}  
    return audit_result

audit_missing_values(train)
1
2
3
4
5
6
7
8
9
10

输出：

{'PassengerId': {'缺失数量': 0, '缺失百分比': 0.0},
 'HomePlanet': {'缺失数量': 201, '缺失百分比': 0.023122052225928908},
 'CryoSleep': {'缺失数量': 217, '缺失百分比': 0.02496261359714713},
 'Destination': {'缺失数量': 182, '缺失百分比': 0.02093638559760727},
 'Age': {'缺失数量': 179, '缺失百分比': 0.020591280340503854},
 'VIP': {'缺失数量': 203, '缺失百分比': 0.023352122397331185},
 'RoomService': {'缺失数量': 181, '缺失百分比': 0.02082135051190613},
 'FoodCourt': {'缺失数量': 183, '缺失百分比': 0.021051420683308408},
 'ShoppingMall': {'缺失数量': 208, '缺失百分比': 0.02392729782583688},
 'Spa': {'缺失数量': 183, '缺失百分比': 0.021051420683308408},
 'VRDeck': {'缺失数量': 188, '缺失百分比': 0.021626596111814105},
 'Name': {'缺失数量': 200, '缺失百分比': 0.023007017140227768},
 'Transported': {'缺失数量': 0, '缺失百分比': 0.0},
 'deck': {'缺失数量': 199, '缺失百分比': 0.02289198205452663},
 'num': {'缺失数量': 199, '缺失百分比': 0.02289198205452663},
 'side': {'缺失数量': 199, '缺失百分比': 0.02289198205452663},
 'SumSpends': {'缺失数量': 0, '缺失百分比': 0.0}}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# 特征变量按照数据类型分成定量变量和定性变量
train_columns = list(train.columns)
train_columns.remove('PassengerId')
train_columns.remove('Transported')
train_columns.remove('Name')

quantitative = [feature for feature in train_columns if train.dtypes[feature] != 'object'] # 定量变量
print('定量变量')
print(quantitative)
qualitative = [feature for feature in train_columns if train.dtypes[feature] == 'object'] # 定性变量
print('定性变量')
print(qualitative)
1
2
3
4
5
6
7
8
9
10
11
12

输出：

定量变量
['Age', 'RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck', 'SumSpends']
定性变量
['HomePlanet', 'CryoSleep', 'Destination', 'VIP', 'deck', 'num', 'side']
1
2
3
4

# 我选择对于定量变量用中位数填充，对定性变量用众数填充
for col in quantitative:  
    mode_val = train[col].median()  
    train[col].fillna(mode_val, inplace=True) 
    
for col in qualitative:  
    mode_val = train[col].mode()[0]  
    train[col].fillna(mode_val, inplace=True)   
1
2
3
4
5
6
7
8

6、特征编码

把非数值特征编码处理为数值特征

label_encoder = LabelEncoder()  
  
# 遍历定性变量列  
for col in qualitative:  
    # 对每一列进行标签编码  
    train[col] = label_encoder.fit_transform(train[col])  
    
train.Transported = label_encoder.fit_transform(train.Transported)      
1
2
3
4
5
6
7
8

7、建立模型并评估

import numpy as np  
import pandas as pd  
from sklearn.model_selection import train_test_split, GridSearchCV  
from sklearn.preprocessing import StandardScaler  
from sklearn.linear_model import LogisticRegression  
from sklearn.svm import SVC  
from sklearn.tree import DecisionTreeClassifier  
from sklearn.neighbors import KNeighborsClassifier  
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier  
import xgboost as xgb  
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score  
from sklearn.metrics import roc_curve, auc  
from mlxtend.classifier import StackingClassifier  
import matplotlib.pyplot as plt  
from joblib import dump, load  
# 数据预处理：划分训练集和测试集，标准化特征  
X_train, X_test, y_train, y_test = train_test_split(train.drop(columns=['PassengerId','Name','Transported']), train['Transported'], test_size=0.2,
                                                    random_state=42)    
scaler = StandardScaler()  
X_train_scaled = scaler.fit_transform(X_train)  
X_test_scaled = scaler.transform(X_test)  
  
# 定义模型列表  
models = [  
    ('Logistic Regression', LogisticRegression(max_iter=1000)),  
    ('Support Vector Machine', SVC(probability=True)),  
    ('Decision Tree', DecisionTreeClassifier()),  
    ('K-nearest neighbors', KNeighborsClassifier()),  
    ('Random Forest', RandomForestClassifier()),  
    ('Gradient Boosting', GradientBoostingClassifier()),  
    ('XGBoost', xgb.XGBClassifier()),
    ('CatBoost',CatBoostClassifier())
]  
  
# 循环构建并评估模型  
for name, model in models:  
    print(f"Training and evaluating {name}...")  
    model.fit(X_train_scaled, y_train)
    # 预测概率和类别  
    y_pred_proba = model.predict_proba(X_test_scaled)[:, 1]  
    y_pred = model.predict(X_test_scaled)  
      
    # 计算性能指标  
    accuracy = accuracy_score(y_test, y_pred)  
    precision = precision_score(y_test, y_pred, average='binary')  
    recall = recall_score(y_test, y_pred, average='binary')  
    f1 = f1_score(y_test, y_pred, average='binary')  
    auc = roc_auc_score(y_test, y_pred_proba)  
      
    # 输出性能指标  
    print(f"Model: {name}")  
    print(f"Accuracy: {accuracy:.4f}")  
    print(f"Precision: {precision:.4f}")  
    print(f"Recall: {recall:.4f}")  
    print(f"F1 Score: {f1:.4f}")  
    print(f"AUC: {auc:.4f}\n")  
      
    # 绘制ROC曲线  
    fpr, tpr, thresholds = roc_curve(y_test, y_pred_proba)  
    plt.plot(fpr, tpr, label=f'{name} (AUC = {auc:.2f})')  
  
# 显示ROC曲线图  
plt.plot([0, 1], [0, 1], 'k--')  # 绘制对角线  
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend(loc='lower right')
plt.show()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

输出：

Training and evaluating Logistic Regression...
Model: Logistic Regression
Accuracy: 0.7746
Precision: 0.7569
Recall: 0.8155
F1 Score: 0.7851
AUC: 0.8605

Training and evaluating Support Vector Machine...
Model: Support Vector Machine
Accuracy: 0.7941
Precision: 0.7814
Recall: 0.8223
F1 Score: 0.8013
AUC: 0.8707

Training and evaluating Decision Tree...
Model: Decision Tree
Accuracy: 0.7430
Precision: 0.7397
Recall: 0.7574
F1 Score: 0.7485
AUC: 0.7427

Training and evaluating K-nearest neighbors...
Model: K-nearest neighbors
Accuracy: 0.7700
Precision: 0.7691
Recall: 0.7779
F1 Score: 0.7735
AUC: 0.8504

Training and evaluating Random Forest...
Model: Random Forest
Accuracy: 0.7872
Precision: 0.8053
Recall: 0.7631
F1 Score: 0.7836
AUC: 0.8770

Training and evaluating Gradient Boosting...
Model: Gradient Boosting
Accuracy: 0.7872
Precision: 0.7657
Recall: 0.8337
F1 Score: 0.7983
AUC: 0.8855

Training and evaluating XGBoost...
Model: XGBoost
Accuracy: 0.7936
Precision: 0.8021
Recall: 0.7847
F1 Score: 0.7933
AUC: 0.8888

Training and evaluating CatBoost...
Model: CatBoost
Accuracy: 0.7999
Precision: 0.7874
Recall: 0.8269
F1 Score: 0.8067
AUC: 0.9006
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

8、模型融合

使用堆叠方法融合模型,Accuracy只有0.778，而CatBoost就有0.799

# 使用堆叠方法融合模型
base_learners = [
    ('lr', LogisticRegression(C=0.01, max_iter=1000, solver='liblinear')),
    ('svm', SVC(probability=True)),
    ('dt', DecisionTreeClassifier(max_depth=5, min_samples_leaf=1, min_samples_split=2)),
    ('knn', KNeighborsClassifier(metric='euclidean', n_neighbors=10, weights='distance')),
    ('rf', RandomForestClassifier(max_depth=10, min_samples_leaf=2, min_samples_split=2, n_estimators=200)),
    ('gb', GradientBoostingClassifier(learning_rate=0.1, max_depth=5, n_estimators=100, subsample=0.5)),
    ('xgb', xgb.XGBClassifier(colsample_bytree=1.0, gamma=0, learning_rate=0.1, max_depth=3, n_estimators=200, subsample=0.5)),
    ('CatBoost', CatBoostClassifier(depth= 6,iterations=50,learning_rate=0.2,rsm=0.725,subsample=0.6))
]

final_learner = LogisticRegression(max_iter=1000)
stacking_classifier = StackingClassifier(classifiers=[model for name, model in base_learners], meta_classifier=final_learner)
# 训练堆叠模型
stacking_classifier.fit(X_train_scaled, y_train)

# 预测堆叠模型的概率和类别
y_stacking_pred_proba = stacking_classifier.predict_proba(X_test_scaled)[:, 1]
y_stacking_pred = stacking_classifier.predict(X_test_scaled)

# 计算堆叠模型的性能指标
stacking_accuracy = accuracy_score(y_test, y_stacking_pred)
stacking_precision = precision_score(y_test, y_stacking_pred, average='binary')
stacking_recall = recall_score(y_test, y_stacking_pred, average='binary')
stacking_f1 = f1_score(y_test, y_stacking_pred, average='binary')
stacking_auc = roc_auc_score(y_test, y_stacking_pred_proba)

# 输出堆叠模型的性能指标
print(f"Stacking Model Performance:")
print(f"Accuracy: {stacking_accuracy:.4f}")
print(f"Precision: {stacking_precision:.4f}")
print(f"Recall: {stacking_recall:.4f}")
print(f"F1 Score: {stacking_f1:.4f}")
print(f"AUC: {stacking_auc:.4f}\n")
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

输出：

Stacking Model Performance:
Accuracy: 0.7780
Precision: 0.7841
Recall: 0.7733
F1 Score: 0.7787
AUC: 0.8360
1
2
3
4
5
6