devbox
小丑西瓜9
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

【金融风控-贷款违约预测】数据挖掘学习：3.特征工程_this could be a false alarm, with some parameters

作者：小丑西瓜9 | 2024-02-26 01:10:34

赞

踩

this could be a false alarm, with some parameters getting used by language b

目录

学习目标

内容介绍

代码示例

导入包并读取数据

特征预处理

缺失值填充

时间格式处理

对象类型特征转换到数值

类别特征处理

异常值处理

检测异常的方法一：均方差

检测异常的方法二：箱型图

数据分桶

固定宽度分桶

分位数分桶

特征交互

特征编码

labelEncode 直接放入树模型中

逻辑回归等模型要单独增加的特征工程

特征选择

Filter

Wrapper （Recursive feature elimination，RFE）

Embedded

总结

学习目标

  • 学习特征预处理、缺失值、异常值处理、数据分桶等特征处理方法；
  • 学习特征交互、编码、选择的相应方法；

内容介绍

  • 数据预处理
    • 缺失值的填充
    • 时间格式处理
    • 对象类型特征转换到数值
  • 异常值处理
    • 基于3segama原则
    • 基于箱型图
  • 数据分箱
    • 固定宽度分箱
    • 分位数分箱
      • 离散数值型数据分箱
      • 连续数值型数据分箱
    • 卡方分箱
  • 特征交互
    • 特征和特征之间组合
    • 特征和特征之间衍生
    • 其他特征衍生的尝试
  • 特征编码
    • one-hot编码
    • label-encode编码
  • 特征选择
    • 1 Filter
    • 2 Wrapper （RFE）
    • 3 Embedded

代码示例

导入包并读取数据

  1. import pandas as pd
  2. import numpy as np
  3. import matplotlib.pyplot as plt
  4. import seaborn as sns
  5. import datetime
  6. from tqdm import tqdm
  7. from sklearn.preprocessing import LabelEncoder
  8. from sklearn.feature_selection import SelectKBest
  9. from sklearn.feature_selection import chi2
  10. from sklearn.preprocessing import MinMaxScaler
  11. import xgboost as xgb
  12. import lightgbm as lgb
  13. from catboost import CatBoostRegressor
  14. import warnings
  15. from sklearn.model_selection import StratifiedKFold, KFold
  16. from sklearn.metrics import accuracy_score, f1_score, roc_auc_score, log_loss
  17. warnings.filterwarnings('ignore')
  1. data_train = pd.read_csv('../train.csv')
  2. data_test_a = pd.read_csv('../testA.csv')

特征预处理

  • 数据EDA部分我们已经了解数据的大概和某些特征分布了，数据预处理部分一般我们要处理一些EDA阶段分析出来的问题，这里介绍了数据缺失值的填充，时间格式特征的转化处理，某些对象类别特征的处理。
  • 首先我们查找出数据中的类别特征和数值特征：
  1. numerical_fea = list(data_train.select_dtypes(exclude=['object']).columns)  # 数值特征
  2. category_fea = list(filter(lambda x: x not in numerical_fea, list(data_train.columns)))  # 类别特征
  3. label = 'isDefault'  # 目标特征
  4. numerical_fea.remove(label)
  • 在比赛中数据预处理是必不可少的一部分，对于缺失值的填充往往会影响比赛的结果，在比赛中不妨尝试多种填充然后比较结果选择结果最优的一种； 比赛数据相比真实场景的数据相对要“干净”一些，但是还是会有一定的“脏”数据存在，清洗一些异常值往往会获得意想不到的效果。

缺失值填充

  • 把所有缺失值替换为指定的值0：
data_train = data_train.fillna(0)
  • 使用缺失值上面的值替换缺失值：
data_train = data_train.fillna(axis=0, method='ffill')
  • 纵向用缺失值下面的值替换缺失值，且设置最多只填充两个连续的缺失值：
data_train = data_train.fillna(axis=0, method='bfill', limit=2)
  • 查看缺失值情况：
  1. data_train.isnull().sum()
  2.  
  3. # id                        0
  4. # loanAmnt                  0
  5. # term                      0
  6. # interestRate              0
  7. # installment               0
  8. # grade                     0
  9. # subGrade                  0
  10. # employmentTitle           1
  11. # employmentLength      46799
  12. # homeOwnership             0
  13. # annualIncome              0
  14. # verificationStatus        0
  15. # issueDate                 0
  16. # isDefault                 0
  17. # purpose                   0
  18. # postCode                  1
  19. # regionCode                0
  20. # dti                     239
  21. # delinquency_2years        0
  22. # ficoRangeLow              0
  23. # ficoRangeHigh             0
  24. # openAcc                   0
  25. # pubRec                    0
  26. # pubRecBankruptcies      405
  27. # revolBal                  0
  28. # revolUtil               531
  29. # totalAcc                  0
  30. # initialListStatus         0
  31. # applicationType           0
  32. # earliesCreditLine         0
  33. # title                     1
  34. # policyCode                0
  35. # n0                    40270
  36. # n1                    40270
  37. # n2                    40270
  38. # n2.1                  40270
  39. # n4                    33239
  40. # n5                    40270
  41. # n6                    40270
  42. # n7                    40270
  43. # n8                    40271
  44. # n9                    40270
  45. # n10                   33239
  46. # n11                   69752
  47. # n12                   40270
  48. # n13                   40270
  49. # n14                   40270
  50. # dtype: int64
  1. # 按照中值填充数值型特征
  2. data_train[numerical_fea] = data_train[numerical_fea].fillna(data_train[numerical_fea].median())
  3. data_test_a[numerical_fea] = data_test_a[numerical_fea].fillna(data_train[numerical_fea].median())
  4.  
  5. # 按照众数填充类别型特征
  6. data_train[category_fea] = data_train[category_fea].fillna(data_train[category_fea].mode())
  7. data_test_a[category_fea] = data_test_a[category_fea].fillna(data_train[category_fea].mode())
  1. data_train.isnull().sum()
  2.  
  3. # id                        0
  4. # loanAmnt                  0
  5. # term                      0
  6. # interestRate              0
  7. # installment               0
  8. # grade                     0
  9. # subGrade                  0
  10. # employmentTitle           0
  11. # employmentLength      46799
  12. # homeOwnership             0
  13. # annualIncome              0
  14. # verificationStatus        0
  15. # issueDate                 0
  16. # isDefault                 0
  17. # purpose                   0
  18. # postCode                  0
  19. # regionCode                0
  20. # dti                       0
  21. # delinquency_2years        0
  22. # ficoRangeLow              0
  23. # ficoRangeHigh             0
  24. # openAcc                   0
  25. # pubRec                    0
  26. # pubRecBankruptcies        0
  27. # revolBal                  0
  28. # revolUtil                 0
  29. # totalAcc                  0
  30. # initialListStatus         0
  31. # applicationType           0
  32. # earliesCreditLine         0
  33. # title                     0
  34. # policyCode                0
  35. # n0                        0
  36. # n1                        0
  37. # n2                        0
  38. # n2.1                      0
  39. # n4                        0
  40. # n5                        0
  41. # n6                        0
  42. # n7                        0
  43. # n8                        0
  44. # n9                        0
  45. # n10                       0
  46. # n11                       0
  47. # n12                       0
  48. # n13                       0
  49. # n14                       0
  50. # dtype: int64

时间格式处理

  1. # 转化成时间格式
  2. for data in [data_train, data_test_a]:
  3.     data['issueDate'] = pd.to_datetime(data['issueDate'], format='%Y-%m-%d')
  4.     startdate = datetime.datetime.strptime('2007-06-01', '%Y-%m-%d')
  5.     # 构造时间特征
  6.     data['issueDateDT'] = data['issueDate'].apply(lambda x: x - startdate).dt.days
  1. data_train['employmentLength'].value_counts(dropna=False).sort_index()
  2.  
  3. # 1 year        52489
  4. # 10+ years    262753
  5. # 2 years       72358
  6. # 3 years       64152
  7. # 4 years       47985
  8. # 5 years       50102
  9. # 6 years       37254
  10. # 7 years       35407
  11. # 8 years       36192
  12. # 9 years       30272
  13. # < 1 year      64237
  14. # NaN           46799
  15. # Name: employmentLength, dtype: int64

对象类型特征转换到数值

  1. def employmentLength_to_int(s):
  2.     if pd.isnull(s):
  3.         return s
  4.     else:
  5.         return np.int8(s.split()[0])
  6.     
  7. for data in [data_train, data_test_a]:
  8.     data['employmentLength'].replace(to_replace='10+ years', value='10 years', inplace=True)
  9.     data['employmentLength'].replace('< 1 year', '0 years', inplace=True)
  10.     data['employmentLength'] = data['employmentLength'].apply(employmentLength_to_int)
  11.     
  12. data['employmentLength'].value_counts(dropna=False).sort_index()
  13.  
  14. # 0.0     15989
  15. # 1.0     13182
  16. # 2.0     18207
  17. # 3.0     16011
  18. # 4.0     11833
  19. # 5.0     12543
  20. # 6.0      9328
  21. # 7.0      8823
  22. # 8.0      8976
  23. # 9.0      7594
  24. # 10.0    65772
  25. # NaN     11742
  26. # Name: employmentLength, dtype: int64
  • 对earliesCreditLine进行预处理：
  1. data_train['earliesCreditLine'].sample(5)
  2.  
  3. # 642880    Jun-1992
  4. # 77423     Aug-1983
  5. # 356008    Mar-1999
  6. # 84346     Aug-2007
  7. # 574182    Sep-2005
  8. # Name: earliesCreditLine, dtype: object
  1. for data in [data_train, data_test_a]:
  2.     data['earliesCreditLine'] = data['earliesCreditLine'].apply(lambda s: int(s[-4:]))

类别特征处理

  1. # 部分类别特征
  2. cate_features = ['grade', 'subGrade', 'employmentTitle', 'homeOwnership', 'verificationStatus', 'purpose', 'postCode', 'regionCode', \
  3.                  'applicationType', 'initialListStatus', 'title', 'policyCode']
  4. for f in cate_features:
  5.     print(f, '类型数：', data_train[f].nunique())
  6.  
  7. # grade 类型数： 7
  8. # subGrade 类型数： 35
  9. # employmentTitle 类型数： 248683
  10. # homeOwnership 类型数： 6
  11. # verificationStatus 类型数： 3
  12. # purpose 类型数： 14
  13. # postCode 类型数： 932
  14. # regionCode 类型数： 51
  15. # applicationType 类型数： 2
  16. # initialListStatus 类型数： 2
  17. # title 类型数： 39644
  18. # policyCode 类型数： 1
  • 像等级这种类别特征，是有优先级的可以labelencode或者自映射：
  1. for data in [data_train, data_test_a]:
  2.     data['grade'] = data['grade'].map({'A': 1,'B': 2,'C': 3,'D': 4,'E': 5,'F': 6,'G': 7})
  1. # 类型数在2之上，又不是高维稀疏的,且纯分类特征
  2. for data in [data_train, data_test_a]:
  3.     data = pd.get_dummies(data, columns=['subGrade', 'homeOwnership', 'verificationStatus', 'purpose', 'regionCode'], drop_first=True)

异常值处理

  • 当你发现异常值后，一定要先分清是什么原因导致的异常值，然后再考虑如何处理。首先，如果这一异常值并不代表一种规律性的，而是极其偶然的现象，或者说你并不想研究这种偶然的现象，这时可以将其删除。其次，如果异常值存在且代表了一种真实存在的现象，那就不能随便删除。在现有的欺诈场景中很多时候欺诈数据本身相对于正常数据来说就是异常的，我们要把这些异常点纳入，重新拟合模型，研究其规律。能用监督的用监督模型，不能用的还可以考虑用异常检测的算法来做。
  • 注意test的数据不能删。

检测异常的方法一：均方差

  • 在统计学中，如果一个数据分布近似正态，那么大约 68% 的数据值会在均值的一个标准差范围内，大约 95% 会在两个标准差范围内，大约 99.7% 会在三个标准差范围内。
  • 得到特征的异常值后可以进一步分析变量异常值和目标变量的关系：
  1. def find_outliers_by_3segama(data, fea):
  2.     data_std = np.std(data[fea])
  3.     data_mean = np.mean(data[fea])
  4.     outliers_cut_off = data_std * 3
  5.     lower_rule = data_mean - outliers_cut_off
  6.     upper_rule = data_mean + outliers_cut_off
  7.     data[fea + '_outliers'] = data[fea].apply(lambda x: str('异常值') if x > upper_rule or x < lower_rule else '正常值')
  8.     return data
  9.  
  10. data_train = data_train.copy()
  11. for fea in numerical_fea:
  12.     data_train = find_outliers_by_3segama(data_train, fea)
  13.     print(data_train[fea + '_outliers'].value_counts())
  14.     print(data_train.groupby(fea + '_outliers')['isDefault'].sum())
  15.     print('*' * 10)
  16.  
  17. # 正常值    800000
  18. # Name: id_outliers, dtype: int64
  19. # id_outliers
  20. # 正常值    159610
  21. # Name: isDefault, dtype: int64
  22. # **********
  23. # 正常值    800000
  24. # Name: loanAmnt_outliers, dtype: int64
  25. # loanAmnt_outliers
  26. # 正常值    159610
  27. # Name: isDefault, dtype: int64
  28. # **********
  29. # 正常值    800000
  30. # Name: term_outliers, dtype: int64
  31. # term_outliers
  32. # 正常值    159610
  33. # Name: isDefault, dtype: int64
  34. # **********
  35. # 正常值    794259
  36. # 异常值      5741
  37. # Name: interestRate_outliers, dtype: int64
  38. # interestRate_outliers
  39. # 异常值      2916
  40. # 正常值    156694
  41. # Name: isDefault, dtype: int64
  42. # **********
  43. # 正常值    792046
  44. # 异常值      7954
  45. # Name: installment_outliers, dtype: int64
  46. # installment_outliers
  47. # 异常值      2152
  48. # 正常值    157458
  49. # Name: isDefault, dtype: int64
  50. # **********
  51. # 正常值    800000
  52. # Name: employmentTitle_outliers, dtype: int64
  53. # employmentTitle_outliers
  54. # 正常值    159610
  55. # Name: isDefault, dtype: int64
  56. # **********
  57. # 正常值    799701
  58. # 异常值       299
  59. # Name: homeOwnership_outliers, dtype: int64
  60. # homeOwnership_outliers
  61. # 异常值        62
  62. # 正常值    159548
  63. # Name: isDefault, dtype: int64
  64. # **********
  65. # 正常值    793973
  66. # 异常值      6027
  67. # Name: annualIncome_outliers, dtype: int64
  68. # annualIncome_outliers
  69. # 异常值       756
  70. # 正常值    158854
  71. # Name: isDefault, dtype: int64
  72. # **********
  73. # 正常值    800000
  74. # Name: verificationStatus_outliers, dtype: int64
  75. # verificationStatus_outliers
  76. # 正常值    159610
  77. # Name: isDefault, dtype: int64
  78. # **********
  79. # 正常值    783003
  80. # 异常值     16997
  81. # Name: purpose_outliers, dtype: int64
  82. # purpose_outliers
  83. # 异常值      3635
  84. # 正常值    155975
  85. # Name: isDefault, dtype: int64
  86. # **********
  87. # 正常值    798931
  88. # 异常值      1069
  89. # Name: postCode_outliers, dtype: int64
  90. # postCode_outliers
  91. # 异常值       221
  92. # 正常值    159389
  93. # Name: isDefault, dtype: int64
  94. # **********
  95. # 正常值    799994
  96. # 异常值         6
  97. # Name: regionCode_outliers, dtype: int64
  98. # regionCode_outliers
  99. # 异常值         1
  100. # 正常值    159609
  101. # Name: isDefault, dtype: int64
  102. # **********
  103. # 正常值    798440
  104. # 异常值      1560
  105. # Name: dti_outliers, dtype: int64
  106. # dti_outliers
  107. # 异常值       466
  108. # 正常值    159144
  109. # Name: isDefault, dtype: int64
  110. # **********
  111. # 正常值    778245
  112. # 异常值     21755
  113. # Name: delinquency_2years_outliers, dtype: int64
  114. # delinquency_2years_outliers
  115. # 异常值      5089
  116. # 正常值    154521
  117. # Name: isDefault, dtype: int64
  118. # **********
  119. # 正常值    788261
  120. # 异常值     11739
  121. # Name: ficoRangeLow_outliers, dtype: int64
  122. # ficoRangeLow_outliers
  123. # 异常值       778
  124. # 正常值    158832
  125. # Name: isDefault, dtype: int64
  126. # **********
  127. # 正常值    788261
  128. # 异常值     11739
  129. # Name: ficoRangeHigh_outliers, dtype: int64
  130. # ficoRangeHigh_outliers
  131. # 异常值       778
  132. # 正常值    158832
  133. # Name: isDefault, dtype: int64
  134. # **********
  135. # 正常值    790889
  136. # 异常值      9111
  137. # Name: openAcc_outliers, dtype: int64
  138. # openAcc_outliers
  139. # 异常值      2195
  140. # 正常值    157415
  141. # Name: isDefault, dtype: int64
  142. # **********
  143. # 正常值    792471
  144. # 异常值      7529
  145. # Name: pubRec_outliers, dtype: int64
  146. # pubRec_outliers
  147. # 异常值      1701
  148. # 正常值    157909
  149. # Name: isDefault, dtype: int64
  150. # **********
  151. # 正常值    794120
  152. # 异常值      5880
  153. # Name: pubRecBankruptcies_outliers, dtype: int64
  154. # pubRecBankruptcies_outliers
  155. # 异常值      1423
  156. # 正常值    158187
  157. # Name: isDefault, dtype: int64
  158. # **********
  159. # 正常值    790001
  160. # 异常值      9999
  161. # Name: revolBal_outliers, dtype: int64
  162. # revolBal_outliers
  163. # 异常值      1359
  164. # 正常值    158251
  165. # Name: isDefault, dtype: int64
  166. # **********
  167. # 正常值    799948
  168. # 异常值        52
  169. # Name: revolUtil_outliers, dtype: int64
  170. # revolUtil_outliers
  171. # 异常值        23
  172. # 正常值    159587
  173. # Name: isDefault, dtype: int64
  174. # **********
  175. # 正常值    791663
  176. # 异常值      8337
  177. # Name: totalAcc_outliers, dtype: int64
  178. # totalAcc_outliers
  179. # 异常值      1668
  180. # 正常值    157942
  181. # Name: isDefault, dtype: int64
  182. # **********
  183. # 正常值    800000
  184. # Name: initialListStatus_outliers, dtype: int64
  185. # initialListStatus_outliers
  186. # 正常值    159610
  187. # Name: isDefault, dtype: int64
  188. # **********
  189. # 正常值    784586
  190. # 异常值     15414
  191. # Name: applicationType_outliers, dtype: int64
  192. # applicationType_outliers
  193. # 异常值      3875
  194. # 正常值    155735
  195. # Name: isDefault, dtype: int64
  196. # **********
  197. # 正常值    775134
  198. # 异常值     24866
  199. # Name: title_outliers, dtype: int64
  200. # title_outliers
  201. # 异常值      3900
  202. # 正常值    155710
  203. # Name: isDefault, dtype: int64
  204. # **********
  205. # 正常值    800000
  206. # Name: policyCode_outliers, dtype: int64
  207. # policyCode_outliers
  208. # 正常值    159610
  209. # Name: isDefault, dtype: int64
  210. # **********
  211. # 正常值    782773
  212. # 异常值     17227
  213. # Name: n0_outliers, dtype: int64
  214. # n0_outliers
  215. # 异常值      3485
  216. # 正常值    156125
  217. # Name: isDefault, dtype: int64
  218. # **********
  219. # 正常值    790500
  220. # 异常值      9500
  221. # Name: n1_outliers, dtype: int64
  222. # n1_outliers
  223. # 异常值      2491
  224. # 正常值    157119
  225. # Name: isDefault, dtype: int64
  226. # **********
  227. # 正常值    789067
  228. # 异常值     10933
  229. # Name: n2_outliers, dtype: int64
  230. # n2_outliers
  231. # 异常值      3205
  232. # 正常值    156405
  233. # Name: isDefault, dtype: int64
  234. # **********
  235. # 正常值    789067
  236. # 异常值     10933
  237. # Name: n2.1_outliers, dtype: int64
  238. # n2.1_outliers
  239. # 异常值      3205
  240. # 正常值    156405
  241. # Name: isDefault, dtype: int64
  242. # **********
  243. # 正常值    788660
  244. # 异常值     11340
  245. # Name: n4_outliers, dtype: int64
  246. # n4_outliers
  247. # 异常值      2476
  248. # 正常值    157134
  249. # Name: isDefault, dtype: int64
  250. # **********
  251. # 正常值    790355
  252. # 异常值      9645
  253. # Name: n5_outliers, dtype: int64
  254. # n5_outliers
  255. # 异常值      1858
  256. # 正常值    157752
  257. # Name: isDefault, dtype: int64
  258. # **********
  259. # 正常值    786006
  260. # 异常值     13994
  261. # Name: n6_outliers, dtype: int64
  262. # n6_outliers
  263. # 异常值      3182
  264. # 正常值    156428
  265. # Name: isDefault, dtype: int64
  266. # **********
  267. # 正常值    788430
  268. # 异常值     11570
  269. # Name: n7_outliers, dtype: int64
  270. # n7_outliers
  271. # 异常值      2746
  272. # 正常值    156864
  273. # Name: isDefault, dtype: int64
  274. # **********
  275. # 正常值    789625
  276. # 异常值     10375
  277. # Name: n8_outliers, dtype: int64
  278. # n8_outliers
  279. # 异常值      2131
  280. # 正常值    157479
  281. # Name: isDefault, dtype: int64
  282. # **********
  283. # 正常值    786384
  284. # 异常值     13616
  285. # Name: n9_outliers, dtype: int64
  286. # n9_outliers
  287. # 异常值      3953
  288. # 正常值    155657
  289. # Name: isDefault, dtype: int64
  290. # **********
  291. # 正常值    788979
  292. # 异常值     11021
  293. # Name: n10_outliers, dtype: int64
  294. # n10_outliers
  295. # 异常值      2639
  296. # 正常值    156971
  297. # Name: isDefault, dtype: int64
  298. # **********
  299. # 正常值    799434
  300. # 异常值       566
  301. # Name: n11_outliers, dtype: int64
  302. # n11_outliers
  303. # 异常值       112
  304. # 正常值    159498
  305. # Name: isDefault, dtype: int64
  306. # **********
  307. # 正常值    797585
  308. # 异常值      2415
  309. # Name: n12_outliers, dtype: int64
  310. # n12_outliers
  311. # 异常值       545
  312. # 正常值    159065
  313. # Name: isDefault, dtype: int64
  314. # **********
  315. # 正常值    788907
  316. # 异常值     11093
  317. # Name: n13_outliers, dtype: int64
  318. # n13_outliers
  319. # 异常值      2482
  320. # 正常值    157128
  321. # Name: isDefault, dtype: int64
  322. # **********
  323. # 正常值    788884
  324. # 异常值     11116
  325. # Name: n14_outliers, dtype: int64
  326. # n14_outliers
  327. # 异常值      3364
  328. # 正常值    156246
  329. # Name: isDefault, dtype: int64
  330. # **********
  1. # 删除异常值
  2. for fea in numerical_fea:
  3.     data_train = data_train[data_train[fea + '_outliers'] == '正常值']
  4.     data_train = data_train.reset_index(drop=True)

检测异常的方法二：箱型图

  • 总结一句话：四分位数会将数据分为三个点和四个区间，IQR = Q3 -Q1，下触须=Q1 − 1.5x IQR，上触须=Q3 + 1.5x IQR；

数据分桶

  • 特征分桶的目的：
    • 从模型效果上来看，特征分桶主要是为了降低变量的复杂性，减少变量噪音对模型的影响，提高自变量和因变量的相关度。从而使模型更加稳定。
  • 数据分桶的对象：
    • 将连续变量离散化
    • 将多状态的离散变量合并成少状态
  • 分桶的原因：
    • 数据特征内的值跨度可能比较大，对有监督和无监督中如k-均值聚类它使用欧氏距离作为相似度函数来测量数据点之间的相似度。都会造成大吃小的影响，其中一种解决方法是对计数值进行区间量化即数据分桶也叫做数据分箱，然后使用量化后的结果。
  • 分桶的优点：
    • 处理缺失值：当数据源可能存在缺失值，此时可以把null单独作为一个分桶。
    • 处理异常值：当数据中存在离群点时，可以把其通过分桶离散化处理，从而提高变量的鲁棒性（抗干扰能力）。例如，age若出现200这种异常值，可分入“age > 60”这个分桶里，排除影响。
    • 业务解释性：我们习惯于线性判断变量的作用，当x越来越大，y就越来越大。但实际x与y之间经常存在着非线性关系，此时可经过WOE变换。
  • 特别要注意一下分桶的基本原则：
    • 最小分桶占比不低于5%
    • 桶内不能全部是好数据
    • 连续桶单调

固定宽度分桶

  • 当数值横跨多个数量级时，最好按照 10 的幂（或任何常数的幂）来进行分组。固定宽度分桶非常容易计算，但如果计数值中有比较大的缺口，就会产生很多没有任何数据的空桶。
  1. # 通过除法映射到间隔均匀的分桶中，每个分桶的取值范围都是loanAmnt/1000
  2. data['loanAmnt_bin1'] = np.floor_divide(data['loanAmnt'], 1000)
  3. data['loanAmnt_bin1']
  4.  
  5. # 0         14.0
  6. # 1         20.0
  7. # 2         12.0
  8. # 3         17.0
  9. # 4         35.0
  10. #           ... 
  11. # 199995     7.0
  12. # 199996     6.0
  13. # 199997    14.0
  14. # 199998     8.0
  15. # 199999     8.0
  16. # Name: loanAmnt_bin1, Length: 200000, dtype: float64
  1. # 通过对数函数映射到指数宽度分桶
  2. data['loanAmnt_bin2'] = np.floor(np.log10(data['loanAmnt']))
  3. data['loanAmnt_bin2']
  4.  
  5. # 0         4.0
  6. # 1         4.0
  7. # 2         4.0
  8. # 3         4.0
  9. # 4         4.0
  10. #          ... 
  11. # 199995    3.0
  12. # 199996    3.0
  13. # 199997    4.0
  14. # 199998    3.0
  15. # 199999    3.0
  16. # Name: loanAmnt_bin2, Length: 200000, dtype: float64

分位数分桶

  1. data['loanAmnt_bin3'] = pd.qcut(data['loanAmnt'], 10, labels=False)
  2. data['loanAmnt_bin3']
  3.  
  4. # 0         5
  5. # 1         7
  6. # 2         4
  7. # 3         6
  8. # 4         9
  9. #          ..
  10. # 199995    2
  11. # 199996    1
  12. # 199997    5
  13. # 199998    2
  14. # 199999    2
  15. # Name: loanAmnt_bin3, Length: 200000, dtype: int64

特征交互

  • 交互特征的构造非常简单，使用起来却代价不菲。如果线性模型中包含有交互特征对，那它的训练时间和评分时间就会从 O(n) 增加到 O(n2)，其中 n 是单一特征的数量。
  1. for col in ['grade', 'subGrade']: 
  2.     temp_dict = data_train.groupby([col])['isDefault'].agg(['mean']).reset_index().rename(columns={'mean': col + '_target_mean'})
  3.     temp_dict.index = temp_dict[col].values
  4.     temp_dict = temp_dict[col + '_target_mean'].to_dict()
  5.  
  6.     data_train[col + '_target_mean'] = data_train[col].map(temp_dict)
  7.     data_test_a[col + '_target_mean'] = data_test_a[col].map(temp_dict)
  1. # 其他衍生变量 mean 和 std
  2. for df in [data_train, data_test_a]:
  3.     for item in ['n0','n1','n2','n2.1','n4','n5','n6','n7','n8','n9','n10','n11','n12','n13','n14']:
  4.         df['grade_to_mean_' + item] = df['grade'] / df.groupby([item])['grade'].transform('mean')
  5.         df['grade_to_std_' + item] = df['grade'] / df.groupby([item])['grade'].transform('std')

特征编码

labelEncode 直接放入树模型中

  1. # label-encode: subGrade,postCode,title
  2. # 高维类别特征需要进行转换
  3. for col in tqdm(['employmentTitle', 'postCode', 'title', 'subGrade']):
  4.     le = LabelEncoder()
  5.     le.fit(list(data_train[col].astype(str).values) + list(data_test_a[col].astype(str).values))
  6.     data_train[col] = le.transform(list(data_train[col].astype(str).values))
  7.     data_test_a[col] = le.transform(list(data_test_a[col].astype(str).values))
  8. print('Label Encoding 完成')
  9.  
  10. # 100%|██████████| 4/4 [00:07<00:00,  1.76s/it]
  11. # Label Encoding 完成

逻辑回归等模型要单独增加的特征工程

  • 对特征做归一化，去除相关性高的特征
  • 归一化目的是让训练过程更好更快的收敛，避免特征大吃小的问题
  • 去除相关性是增加模型的可解释性，加快预测过程。
  1. # 举例归一化过程
  2. # 伪代码
  3. for fea in [要归一化的特征列表]：
  4.     data[fea] = ((data[fea] - np.min(data[fea])) / (np.max(data[fea]) - np.min(data[fea])))

特征选择

  • 特征选择技术可以精简掉无用的特征，以降低最终模型的复杂性，它的最终目的是得到一个简约模型，在不降低预测准确率或对预测准确率影响不大的情况下提高计算速度。特征选择不是为了减少训练时间（实际上，一些技术会增加总体训练时间），而是为了减少模型评分时间。

Filter

方差选择法

  • 方差选择法中，先要计算各个特征的方差，然后根据设定的阈值，选择方差大于阈值的特征
  1. from sklearn.feature_selection import VarianceThreshold
  2. # 其中参数threshold为方差的阈值
  3. VarianceThreshold(threshold=3).fit_transform(train, target_train)

相关系数法

  • Pearson 相关系数是一种最简单的，可以帮助理解特征和响应变量之间关系的方法，该方法衡量的是变量之间的线性相关性。 结果的取值区间为 [-1，1] ， -1 表示完全的负相关， +1表示完全的正相关，0 表示没有线性相关。
  1. from sklearn.feature_selection import SelectKBest
  2. from scipy.stats import pearsonr
  3. # 选择K个最好的特征，返回选择特征后的数据
  4. # 第一个参数为计算评估特征是否好的函数，该函数输入特征矩阵和目标向量，
  5. # 输出二元组（评分，P值）的数组，数组第i项为第i个特征的评分和P值。在此定义为计算相关系数
  6. # 参数k为选择的特征个数
  7.  
  8. SelectKBest(k=5).fit_transform(train, target_train)

卡方检验

  • 经典的卡方检验是用于检验自变量对因变量的相关性。 假设自变量有N种取值，因变量有M种取值，考虑自变量等于i且因变量等于j的样本频数的观察值与期望的差距。 其统计量如下： χ2=∑(A−T)2T，其中A为实际值，T为理论值
  • 注：卡方只能运用在正定矩阵上，否则会报错Input X must be non-negative
  1. from sklearn.feature_selection import SelectKBest
  2. from sklearn.feature_selection import chi2
  3. # 参数k为选择的特征个数
  4.  
  5. SelectKBest(chi2, k=5).fit_transform(train, target_train)

互信息法

  • 经典的互信息也是评价自变量对因变量的相关性的。 在feature_selection库的SelectKBest类结合最大信息系数法可以用于选择特征，相关代码如下：
  1. from sklearn.feature_selection import SelectKBest
  2. from minepy import MINE
  3. # 由于MINE的设计不是函数式的，定义mic方法将其作为函数式的，
  4. # 返回一个二元组，二元组的第2项设置成固定的P值0.5
  5. def mic(x, y):
  6.     m = MINE()
  7.     m.compute_score(x, y)
  8.     return (m.mic(), 0.5)
  9.  
  10. # 参数k为选择的特征个数
  11. SelectKBest(lambda X, Y: array(map(lambda x: mic(x, Y), X.T)).T, k=2).fit_transform(train, target_train)

Wrapper （Recursive feature elimination，RFE）

  • 递归消除特征法使用一个基模型来进行多轮训练，每轮训练后，消除若干权值系数的特征，再基于新的特征集进行下一轮训练。 在feature_selection库的RFE类可以用于选择特征，相关代码如下（以逻辑回归为例）：
  1. from sklearn.feature_selection import RFE
  2. from sklearn.linear_model import LogisticRegression
  3. # 递归特征消除法，返回特征选择后的数据
  4. # 参数estimator为基模型
  5. # 参数n_features_to_select为选择的特征个数
  6.  
  7. RFE(estimator=LogisticRegression(), n_features_to_select=2).fit_transform(train, target_train)

Embedded

  • 基于惩罚项的特征选择法使用带惩罚项的基模型，除了筛选出特征外，同时也进行了降维。 在feature_selection库的SelectFromModel类结合逻辑回归模型可以用于选择特征，相关代码如下：
  1. from sklearn.feature_selection import SelectFromModel
  2. from sklearn.linear_model import LogisticRegression
  3. # 带L1惩罚项的逻辑回归作为基模型的特征选择
  4.  
  5. SelectFromModel(LogisticRegression(penalty="l1", C=0.1)).fit_transform(train, target_train)
  • 基于树模型的特征选择 树模型中GBDT也可用来作为基模型进行特征选择。 在feature_selection库的SelectFromModel类结合GBDT模型可以用于选择特征，相关代码如下：
  1. from sklearn.feature_selection import SelectFromModel
  2. from sklearn.ensemble import GradientBoostingClassifier
  3. # GBDT作为基模型的特征选择
  4. SelectFromModel(GradientBoostingClassifier()).fit_transform(train, target_train)
  • 本数据集中我们删除非入模特征后，并对缺失值填充，然后用计算协方差的方式看一下特征间相关性，然后进行模型训练
  1. # 删除不需要的数据
  2. for data in [data_train, data_test_a]:
  3.     data.drop(['issueDate','id'], axis=1, inplace=True)
  1. # 纵向用缺失值上面的值替换缺失值
  2. data_train = data_train.fillna(axis=0, method='ffill')
  1. x_train = data_train.drop(['isDefault'], axis=1)
  2. # 计算协方差
  3. data_corr = x_train.corrwith(data_train.isDefault)  # 计算相关性
  4. result = pd.DataFrame(columns=['features', 'corr'])
  5. result['features'] = data_corr.index
  6. result['corr'] = data_corr.values
  1. # 当然也可以直接看图
  2. data_numeric = data_train[numerical_fea]
  3. correlation = data_numeric.corr()
  4.  
  5. f , ax = plt.subplots(figsize = (7, 7))
  6. plt.title('Correlation of Numeric Features with Price',y=1,size=16)
  7. sns.heatmap(correlation,square = True,  vmax=0.8)

  1. features = [f for f in data_train.columns if f not in ['id','issueDate','isDefault'] and '_outliers' not in f]
  2. x_train = data_train[features]
  3. x_test = data_test_a[features]
  4. y_train = data_train['isDefault']
  1. def cv_model(clf, train_x, train_y, test_x, clf_name):
  2.     folds = 5
  3.     seed = 2020
  4.     kf = KFold(n_splits=folds, shuffle=True, random_state=seed)
  5.  
  6.     train = np.zeros(train_x.shape[0])
  7.     test = np.zeros(test_x.shape[0])
  8.  
  9.     cv_scores = []
  10.  
  11.     for i, (train_index, valid_index) in enumerate(kf.split(train_x, train_y)):
  12.         print('************************************ {} ************************************'.format(str(i+1)))
  13.         trn_x, trn_y, val_x, val_y = train_x.iloc[train_index], train_y[train_index], train_x.iloc[valid_index], train_y[valid_index]
  14.  
  15.         if clf_name == "lgb":
  16.             train_matrix = clf.Dataset(trn_x, label=trn_y)
  17.             valid_matrix = clf.Dataset(val_x, label=val_y)
  18.  
  19.             params = {
  20.                 'boosting_type': 'gbdt',
  21.                 'objective': 'binary',
  22.                 'metric': 'auc',
  23.                 'min_child_weight': 5,
  24.                 'num_leaves': 2 ** 5,
  25.                 'lambda_l2': 10,
  26.                 'feature_fraction': 0.8,
  27.                 'bagging_fraction': 0.8,
  28.                 'bagging_freq': 4,
  29.                 'learning_rate': 0.1,
  30.                 'seed': 2020,
  31.                 'nthread': 28,
  32.                 'n_jobs':24,
  33.                 'silent': True,
  34.                 'verbose': -1,
  35.             }
  36.  
  37.             model = clf.train(params, train_matrix, 50000, valid_sets=[train_matrix, valid_matrix], verbose_eval=200,early_stopping_rounds=200)
  38.             val_pred = model.predict(val_x, num_iteration=model.best_iteration)
  39.             test_pred = model.predict(test_x, num_iteration=model.best_iteration)
  40.             
  41.             # print(list(sorted(zip(features, model.feature_importance("gain")), key=lambda x: x[1], reverse=True))[:20])
  42.                 
  43.         if clf_name == "xgb":
  44.             train_matrix = clf.DMatrix(trn_x , label=trn_y)
  45.             valid_matrix = clf.DMatrix(val_x , label=val_y)
  46.             
  47.             params = {'booster': 'gbtree',
  48.                       'objective': 'binary:logistic',
  49.                       'eval_metric': 'auc',
  50.                       'gamma': 1,
  51.                       'min_child_weight': 1.5,
  52.                       'max_depth': 5,
  53.                       'lambda': 10,
  54.                       'subsample': 0.7,
  55.                       'colsample_bytree': 0.7,
  56.                       'colsample_bylevel': 0.7,
  57.                       'eta': 0.04,
  58.                       'tree_method': 'exact',
  59.                       'seed': 2020,
  60.                       'nthread': 36,
  61.                       "silent": True,
  62.                       }
  63.             
  64.             watchlist = [(train_matrix, 'train'),(valid_matrix, 'eval')]
  65.             
  66.             model = clf.train(params, train_matrix, num_boost_round=50000, evals=watchlist, verbose_eval=200, early_stopping_rounds=200)
  67.             val_pred  = model.predict(valid_matrix, ntree_limit=model.best_ntree_limit)
  68.             test_pred = model.predict(test_x , ntree_limit=model.best_ntree_limit)
  69.                  
  70.         if clf_name == "cat":
  71.             params = {'learning_rate': 0.05, 'depth': 5, 'l2_leaf_reg': 10, 'bootstrap_type': 'Bernoulli',
  72.                       'od_type': 'Iter', 'od_wait': 50, 'random_seed': 11, 'allow_writing_files': False}
  73.             
  74.             model = clf(iterations=20000, **params)
  75.             model.fit(trn_x, trn_y, eval_set=(val_x, val_y),
  76.                       cat_features=[], use_best_model=True, verbose=500)
  77.             
  78.             val_pred  = model.predict(val_x)
  79.             test_pred = model.predict(test_x)
  80.             
  81.         train[valid_index] = val_pred
  82.         test = test_pred / kf.n_splits
  83.         cv_scores.append(roc_auc_score(val_y, val_pred))
  84.         
  85.         print(cv_scores)
  86.         
  87.     print("%s_scotrainre_list:" % clf_name, cv_scores)
  88.     print("%s_score_mean:" % clf_name, np.mean(cv_scores))
  89.     print("%s_score_std:" % clf_name, np.std(cv_scores))
  90.     return train, test
  1. def lgb_model(x_train, y_train, x_test):
  2.     lgb_train, lgb_test = cv_model(lgb, x_train, y_train, x_test, "lgb")
  3.     return lgb_train, lgb_test
  4.  
  5. def xgb_model(x_train, y_train, x_test):
  6.     xgb_train, xgb_test = cv_model(xgb, x_train, y_train, x_test, "xgb")
  7.     return xgb_train, xgb_test
  8.  
  9. def cat_model(x_train, y_train, x_test):
  10.     cat_train, cat_test = cv_model(CatBoostRegressor, x_train, y_train, x_test, "cat")
  11.     return cat_train, cat_test
  1. lgb_train, lgb_test = lgb_model(x_train, y_train, x_test)
  2.  
  3. # ************************************ 1 ************************************
  4. # [LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24
  5. # [LightGBM] [Warning] Unknown parameter: silent
  6. # Training until validation scores don't improve for 200 rounds
  7. # [200]	training's auc: 0.749114	valid_1's auc: 0.729275
  8. # [400]	training's auc: 0.764716	valid_1's auc: 0.730125
  9. # [600]	training's auc: 0.778489	valid_1's auc: 0.729928
  10. # Early stopping, best iteration is:
  11. # [446]	training's auc: 0.768137	valid_1's auc: 0.730186
  12. # [0.7301862239949224]
  13. # ************************************ 2 ************************************
  14. # [LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24
  15. # [LightGBM] [Warning] Unknown parameter: silent
  16. # Training until validation scores don't improve for 200 rounds
  17. # [200]	training's auc: 0.748999	valid_1's auc: 0.731035
  18. # [400]	training's auc: 0.764879	valid_1's auc: 0.731436
  19. # [600]	training's auc: 0.778506	valid_1's auc: 0.730823
  20. # Early stopping, best iteration is:
  21. # [414]	training's auc: 0.765823	valid_1's auc: 0.731478
  22. # [0.7301862239949224, 0.7314779648434573]
  23. # ************************************ 3 ************************************
  24. # [LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24
  25. # [LightGBM] [Warning] Unknown parameter: silent
  26. # Training until validation scores don't improve for 200 rounds
  27. # [200]	training's auc: 0.748145	valid_1's auc: 0.73253
  28. # [400]	training's auc: 0.763814	valid_1's auc: 0.733272
  29. # [600]	training's auc: 0.777895	valid_1's auc: 0.733354
  30. # Early stopping, best iteration is:
  31. # [475]	training's auc: 0.769215	valid_1's auc: 0.73355
  32. # [0.7301862239949224, 0.7314779648434573, 0.7335502065719879]
  33. # ************************************ 4 ************************************
  34. # [LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24
  35. # [LightGBM] [Warning] Unknown parameter: silent
  36. # Training until validation scores don't improve for 200 rounds
  37. # [200]	training's auc: 0.749417	valid_1's auc: 0.727507
  38. # [400]	training's auc: 0.765066	valid_1's auc: 0.728261
  39. # Early stopping, best iteration is:
  40. # [353]	training's auc: 0.761647	valid_1's auc: 0.728349
  41. # [0.7301862239949224, 0.7314779648434573, 0.7335502065719879, 0.7283491938614568]
  42. # ************************************ 5 ************************************
  43. # [LightGBM] [Warning] num_threads is set with n_jobs=24, nthread=28 will be ignored. Current value: num_threads=24
  44. # [LightGBM] [Warning] Unknown parameter: silent
  45. # Training until validation scores don't improve for 200 rounds
  46. # [200]	training's auc: 0.748562	valid_1's auc: 0.73262
  47. # [400]	training's auc: 0.764493	valid_1's auc: 0.733365
  48. # Early stopping, best iteration is:
  49. # [394]	training's auc: 0.764109	valid_1's auc: 0.733381
  50. # [0.7301862239949224, 0.7314779648434573, 0.7335502065719879, 0.7283491938614568, 0.7333810157041901]
  51. # lgb_scotrainre_list: [0.7301862239949224, 0.7314779648434573, 0.7335502065719879, 0.7283491938614568, 0.7333810157041901]
  52. # lgb_score_mean: 0.7313889209952029
  53. # lgb_score_std: 0.001966415347937543
  1. lgb_train, lgb_test = xgb_model(x_train, y_train, x_test)
  2.  
  3. # ************************************ 1 ************************************
  4. # [15:02:32] WARNING: ../src/learner.cc:516: 
  5. # Parameters: { silent } might not be used.
  6.  
  7. #   This may not be accurate due to some parameters are only used in language bindings but
  8. #   passed down to XGBoost core.  Or some parameters are not used but slip through this
  9. #   verification. Please open an issue if you find above cases.
  10.  
  11.  
  12. # [0]	train-auc:0.69713	eval-auc:0.69580
  13. # Multiple eval metrics have been passed: 'eval-auc' will be used for early stopping.
  14.  
  15. # Will train until eval-auc hasn't improved in 200 rounds.
  16. # [200]	train-auc:0.73103	eval-auc:0.72371
  17. # [400]	train-auc:0.74040	eval-auc:0.72807
  18. # [600]	train-auc:0.74624	eval-auc:0.72966
  19. # [800]	train-auc:0.75132	eval-auc:0.73055
  20. # [1000]	train-auc:0.75580	eval-auc:0.73101
  21. # [1200]	train-auc:0.76004	eval-auc:0.73127
  22. # [1400]	train-auc:0.76409	eval-auc:0.73156
  23. # [1600]	train-auc:0.76791	eval-auc:0.73169
  24. # [1800]	train-auc:0.77156	eval-auc:0.73173
  25. # [2000]	train-auc:0.77506	eval-auc:0.73167
  26. # Stopping. Best iteration:
  27. # [1852]	train-auc:0.77251	eval-auc:0.73177
  28.  
  29. # [0.731769339538683]
  30. # ************************************ 2 ************************************
  31. # [15:07:16] WARNING: ../src/learner.cc:516: 
  32. # Parameters: { silent } might not be used.
  33.  
  34. #   This may not be accurate due to some parameters are only used in language bindings but
  35. #   passed down to XGBoost core.  Or some parameters are not used but slip through this
  36. #   verification. Please open an issue if you find above cases.
  37.  
  38.  
  39. # [0]	train-auc:0.69687	eval-auc:0.69574
  40. # Multiple eval metrics have been passed: 'eval-auc' will be used for early stopping.
  41.  
  42. # Will train until eval-auc hasn't improved in 200 rounds.
  43. # [200]	train-auc:0.73078	eval-auc:0.72640
  44. # [400]	train-auc:0.74020	eval-auc:0.73023
  45. # [600]	train-auc:0.74605	eval-auc:0.73156
  46. # [800]	train-auc:0.75114	eval-auc:0.73231
  47. # [1000]	train-auc:0.75562	eval-auc:0.73275
  48. # [1200]	train-auc:0.75987	eval-auc:0.73310
  49. # [1400]	train-auc:0.76372	eval-auc:0.73317
  50. # [1600]	train-auc:0.76757	eval-auc:0.73330
  51. # [1800]	train-auc:0.77123	eval-auc:0.73335
  52. # [2000]	train-auc:0.77484	eval-auc:0.73339
  53. # Stopping. Best iteration:
  54. # [1829]	train-auc:0.77173	eval-auc:0.73340
  55.  
  56. # [0.731769339538683, 0.733395913606802]
  57. # ************************************ 3 ************************************
  58. # [15:11:52] WARNING: ../src/learner.cc:516: 
  59. # Parameters: { silent } might not be used.
  60.  
  61. #   This may not be accurate due to some parameters are only used in language bindings but
  62. #   passed down to XGBoost core.  Or some parameters are not used but slip through this
  63. #   verification. Please open an issue if you find above cases.
  64.  
  65.  
  66. # [0]	train-auc:0.69730	eval-auc:0.69647
  67. # Multiple eval metrics have been passed: 'eval-auc' will be used for early stopping.
  68.  
  69. # Will train until eval-auc hasn't improved in 200 rounds.
  70. # [200]	train-auc:0.73072	eval-auc:0.72604
  71. # [400]	train-auc:0.73965	eval-auc:0.73076
  72. # [600]	train-auc:0.74548	eval-auc:0.73241
  73. # [800]	train-auc:0.75050	eval-auc:0.73356
  74. # [1000]	train-auc:0.75501	eval-auc:0.73416
  75. # [1200]	train-auc:0.75898	eval-auc:0.73460
  76. # [1400]	train-auc:0.76303	eval-auc:0.73487
  77. # [1600]	train-auc:0.76689	eval-auc:0.73507
  78. # [1800]	train-auc:0.77059	eval-auc:0.73507
  79. # Stopping. Best iteration:
  80. # [1703]	train-auc:0.76871	eval-auc:0.73515
  81.  
  82. # [0.731769339538683, 0.733395913606802, 0.7351456720593506]
  83. # ************************************ 4 ************************************
  84. # [15:16:15] WARNING: ../src/learner.cc:516: 
  85. # Parameters: { silent } might not be used.
  86.  
  87. #   This may not be accurate due to some parameters are only used in language bindings but
  88. #   passed down to XGBoost core.  Or some parameters are not used but slip through this
  89. #   verification. Please open an issue if you find above cases.
  90.  
  91.  
  92. # [0]	train-auc:0.69737	eval-auc:0.69375
  93. # Multiple eval metrics have been passed: 'eval-auc' will be used for early stopping.
  94.  
  95. # Will train until eval-auc hasn't improved in 200 rounds.
  96. # [200]	train-auc:0.73148	eval-auc:0.72250
  97. # [400]	train-auc:0.74044	eval-auc:0.72639
  98. # [600]	train-auc:0.74649	eval-auc:0.72804
  99. # [800]	train-auc:0.75154	eval-auc:0.72887
  100. # [1000]	train-auc:0.75598	eval-auc:0.72934
  101. # [1200]	train-auc:0.75997	eval-auc:0.72954
  102. # [1400]	train-auc:0.76401	eval-auc:0.72977
  103. # [1600]	train-auc:0.76793	eval-auc:0.72989
  104. # [1800]	train-auc:0.77159	eval-auc:0.72993
  105. # [2000]	train-auc:0.77511	eval-auc:0.73002
  106. # [2200]	train-auc:0.77850	eval-auc:0.72996
  107. # Stopping. Best iteration:
  108. # [2011]	train-auc:0.77531	eval-auc:0.73004
  109.  
  110. # [0.731769339538683, 0.733395913606802, 0.7351456720593506, 0.7300361842852358]
  111. # ************************************ 5 ************************************
  112. # [15:21:18] WARNING: ../src/learner.cc:516: 
  113. # Parameters: { silent } might not be used.
  114.  
  115. #   This may not be accurate due to some parameters are only used in language bindings but
  116. #   passed down to XGBoost core.  Or some parameters are not used but slip through this
  117. #   verification. Please open an issue if you find above cases.
  118.  
  119.  
  120. # [0]	train-auc:0.69647	eval-auc:0.69701
  121. # Multiple eval metrics have been passed: 'eval-auc' will be used for early stopping.
  122.  
  123. # Will train until eval-auc hasn't improved in 200 rounds.
  124. # [200]	train-auc:0.73059	eval-auc:0.72675
  125. # [400]	train-auc:0.73972	eval-auc:0.73089
  126. # [600]	train-auc:0.74589	eval-auc:0.73256
  127. # [800]	train-auc:0.75073	eval-auc:0.73347
  128. # [1000]	train-auc:0.75523	eval-auc:0.73401
  129. # [1200]	train-auc:0.75941	eval-auc:0.73419
  130. # [1400]	train-auc:0.76342	eval-auc:0.73438
  131. # [1600]	train-auc:0.76730	eval-auc:0.73458
  132. # [1800]	train-auc:0.77105	eval-auc:0.73454
  133. # Stopping. Best iteration:
  134. # [1694]	train-auc:0.76910	eval-auc:0.73464
  135.  
  136. # [0.731769339538683, 0.733395913606802, 0.7351456720593506, 0.7300361842852358, 0.734639280693211]
  137. # xgb_scotrainre_list: [0.731769339538683, 0.733395913606802, 0.7351456720593506, 0.7300361842852358, 0.734639280693211]
  138. # xgb_score_mean: 0.7329972780366564
  139. # xgb_score_std: 0.0018839633265100187
  1. lgb_train, lgb_test = cat_model(x_train, y_train, x_test)
  2.  
  3. # ************************************ 1 ************************************
  4. # 0:	learn: 0.3944330	test: 0.3964727	best: 0.3964727 (0)	total: 138ms	remaining: 45m 59s
  5. # 500:	learn: 0.3728126	test: 0.3756408	best: 0.3756408 (500)	total: 28.1s	remaining: 18m 13s
  6. # 1000:	learn: 0.3711980	test: 0.3750523	best: 0.3750523 (1000)	total: 56.2s	remaining: 17m 47s
  7. # 1500:	learn: 0.3699538	test: 0.3748118	best: 0.3748107 (1476)	total: 1m 23s	remaining: 17m 11s
  8. # 2000:	learn: 0.3688546	test: 0.3746815	best: 0.3746815 (2000)	total: 1m 51s	remaining: 16m 44s
  9. # Stopped by overfitting detector  (50 iterations wait)
  10.  
  11. # bestTest = 0.3746253358
  12. # bestIteration = 2266
  13.  
  14. # Shrink model to first 2267 iterations.
  15. # [0.7306375926022922]
  16. # ************************************ 2 ************************************
  17. # 0:	learn: 0.3947513	test: 0.3951211	best: 0.3951211 (0)	total: 71.1ms	remaining: 23m 41s
  18. # 500:	learn: 0.3731076	test: 0.3743412	best: 0.3743412 (500)	total: 28.6s	remaining: 18m 32s
  19. # 1000:	learn: 0.3714544	test: 0.3737577	best: 0.3737570 (999)	total: 56.7s	remaining: 17m 56s
  20. # 1500:	learn: 0.3702186	test: 0.3735397	best: 0.3735396 (1498)	total: 1m 24s	remaining: 17m 23s
  21. # 2000:	learn: 0.3691118	test: 0.3734092	best: 0.3734074 (1977)	total: 1m 52s	remaining: 16m 54s
  22. # 2500:	learn: 0.3680796	test: 0.3733234	best: 0.3733218 (2484)	total: 2m 21s	remaining: 16m 28s
  23. # Stopped by overfitting detector  (50 iterations wait)
  24.  
  25. # bestTest = 0.373251629
  26. # bestIteration = 2919
  27.  
  28. # Shrink model to first 2920 iterations.
  29. # [0.7306375926022922, 0.7325015175914498]
  30. # ************************************ 3 ************************************
  31. # 0:	learn: 0.3951060	test: 0.3937487	best: 0.3937487 (0)	total: 70.2ms	remaining: 23m 24s
  32. # 500:	learn: 0.3734715	test: 0.3730983	best: 0.3730983 (500)	total: 28.4s	remaining: 18m 26s
  33. # 1000:	learn: 0.3718399	test: 0.3724184	best: 0.3724184 (1000)	total: 56.5s	remaining: 17m 53s
  34. # 1500:	learn: 0.3706048	test: 0.3721639	best: 0.3721639 (1500)	total: 1m 24s	remaining: 17m 24s
  35. # 2000:	learn: 0.3695127	test: 0.3720199	best: 0.3720199 (2000)	total: 1m 52s	remaining: 16m 52s
  36. # 2500:	learn: 0.3685041	test: 0.3719052	best: 0.3719025 (2479)	total: 2m 20s	remaining: 16m 20s
  37. # Stopped by overfitting detector  (50 iterations wait)
  38.  
  39. # bestTest = 0.3719024831
  40. # bestIteration = 2479
  41.  
  42. # Shrink model to first 2480 iterations.
  43. # [0.7306375926022922, 0.7325015175914498, 0.7340103693991001]
  44. # ************************************ 4 ************************************
  45. # 0:	learn: 0.3949491	test: 0.3943298	best: 0.3943298 (0)	total: 66.8ms	remaining: 22m 16s
  46. # 500:	learn: 0.3732214	test: 0.3741316	best: 0.3741316 (500)	total: 28.2s	remaining: 18m 18s
  47. # 1000:	learn: 0.3715666	test: 0.3735451	best: 0.3735414 (995)	total: 56s	remaining: 17m 42s
  48. # 1500:	learn: 0.3703238	test: 0.3733058	best: 0.3733045 (1498)	total: 1m 23s	remaining: 17m 9s
  49. # 2000:	learn: 0.3692105	test: 0.3731636	best: 0.3731634 (1999)	total: 1m 51s	remaining: 16m 41s
  50. # 2500:	learn: 0.3681907	test: 0.3730490	best: 0.3730490 (2500)	total: 2m 19s	remaining: 16m 13s
  51. # Stopped by overfitting detector  (50 iterations wait)
  52.  
  53. # bestTest = 0.3730185197
  54. # bestIteration = 2723
  55.  
  56. # Shrink model to first 2724 iterations.
  57. # [0.7306375926022922, 0.7325015175914498, 0.7340103693991001, 0.7291287412227256]
  58. # ************************************ 5 ************************************
  59. # 0:	learn: 0.3948860	test: 0.3944692	best: 0.3944692 (0)	total: 68.4ms	remaining: 22m 47s
  60. # 500:	learn: 0.3733508	test: 0.3734623	best: 0.3734623 (500)	total: 28.7s	remaining: 18m 37s
  61. # 1000:	learn: 0.3717222	test: 0.3729094	best: 0.3729094 (1000)	total: 57s	remaining: 18m 2s
  62. # 1500:	learn: 0.3704933	test: 0.3726407	best: 0.3726407 (1500)	total: 1m 25s	remaining: 17m 32s
  63. # 2000:	learn: 0.3693930	test: 0.3725202	best: 0.3725200 (1998)	total: 1m 53s	remaining: 17m 4s
  64. # 2500:	learn: 0.3683883	test: 0.3724494	best: 0.3724494 (2500)	total: 2m 22s	remaining: 16m 36s
  65. # Stopped by overfitting detector  (50 iterations wait)
  66.  
  67. # bestTest = 0.3724045318
  68. # bestIteration = 2904
  69.  
  70. # Shrink model to first 2905 iterations.
  71. # [0.7306375926022922, 0.7325015175914498, 0.7340103693991001, 0.7291287412227256, 0.7342835786894728]
  72. # cat_scotrainre_list: [0.7306375926022922, 0.7325015175914498, 0.7340103693991001, 0.7291287412227256, 0.7342835786894728]
  73. # cat_score_mean: 0.7321123599010082
  74. # cat_score_std: 0.0019771188023493848

总结

特征工程是机器学习，甚至是深度学习中最为重要的一部分，在实际应用中往往也是所花费时间最多的一步。各种算法书中对特征工程部分的讲解往往少得可怜，因为特征工程和具体的数据结合的太紧密，很难系统地覆盖所有场景。本章主要是通过一些常用的方法来做介绍，例如缺失值异常值的处理方法详细对任何数据集来说都是适用的。但对于分箱等操作本章给出了具体的几种思路，需要读者自己探索。在特征工程中比赛和具体的应用还是有所不同的，在实际的金融风控评分卡制作过程中，由于强调特征的可解释性，特征分箱尤其重要。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/小丑西瓜9/article/detail/142863
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号