devbox
小蓝xlanll
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

炼丹记之solo四川诈骗电话识别复赛rank13分享

作者：小蓝xlanll | 2024-02-27 11:08:34

赞

踩

炼丹记之solo四川诈骗电话识别复赛rank13分享

赛题地址：http://www.scdata.net.cn/kfds/urgent2/pages/index.html  ，诈骗电话识别是算法对抗赛的赛题之一。

attention：由于签了保密协议，本文不会提供代码涉及到的数据。

参赛历程：初赛时，由于大佬开的baseline分数已经很高，于是本人只调了调baseline然后就弃赛了。然后初赛结束的时候竟然收到了进入复赛的短信，于是交保密协议，下载数据，算是复赛才开始正式参赛吧。本来给一个大佬的结果融合可以进入top4的，可惜没选中。

本文参考的bl：https://github.com/biaobiao2/DC_phone

本方案简要说明：

特征工程：主要使用了统计特征，在bl的特征基础上新增了sem、skew等统计特征，实践证明sem、skew是该赛题数据非常优秀的特征。

模型搭建：lgb+xgb+cat，三模输出概率按0.25:0.25:0.5累加求0-1的概率，将1概率大于0.6的输出为1，否则为0。

输出结果：复赛B榜0.9005

其它trick：训练特征未做填充，测试数据做分位数填充。单模建模时通过求所有预测值0-1概率差的绝对值的和，找出和为最大的分位数作为填充，从B榜分数来看，能带来几个千分位的提升。

代码部分：

特征工程：

  1. # coding=utf-8
  2. '''
  3. @author: csdn xuxml
  4. '''
  5. import os
  6. import gc
  7. import time
  8. import psutil
  9. import datetime
  10. import numpy as np
  11. import pandas as pd
  12. import catboost as cat
  13. import lightgbm as lgb
  14. from sklearn.metrics import f1_score
  15. from catboost import CatBoostClassifier
  16. from scipy.stats import entropy, pearsonr, stats
  17. from sklearn.model_selection import GridSearchCV
  18. from sklearn.model_selection import StratifiedKFold
  19. from sklearn.preprocessing import MinMaxScaler, LabelEncoder
  20.  
  21. pd.set_option('display.unicode.ambiguous_as_wide', True)
  22. pd.set_option('display.unicode.east_asian_width', True)
  23. pd.set_option('display.max_columns', None)
  24. pd.set_option("display.max_colwidth",100)
  25. pd.set_option('display.max_rows', None)
  26. pd.set_option('display.width',100)
  27.  
  28. path = "./0527/"
  29. feat_path = path + "data/"
  30.  
  31. def get_app_feats(df):
  32.     phones_app = df[["phone_no_m"]].copy()
  33.     phones_app = phones_app.drop_duplicates(subset=['phone_no_m'], keep='last')
  34.     tmp = df.groupby("phone_no_m")["busi_name"].agg(busi_count="nunique")
  35.     phones_app = phones_app.merge(tmp, on="phone_no_m", how="left")
  36.     """使用的流量统计
  37.     """
  38.     tmp = df.groupby("phone_no_m")["flow"].agg(flow_mean="mean", 
  39.                                                flow_median = "median", 
  40.                                                flow_min  = "min", 
  41.                                                flow_max = "max", 
  42.                                                flow_var = "var",
  43.                                                flow_skew = "skew",
  44.                                                flow_std = "std",
  45.                                                flow_quantile = "quantile",
  46.                                                flow_sem = "sem",
  47.                                                flow_sum = "sum")
  48.     phones_app = phones_app.merge(tmp, on="phone_no_m", how="left")
  49.     tmp = df.groupby("phone_no_m")["month_id"].agg(month_ids ="nunique")
  50.     phones_app = phones_app.merge(tmp, on="phone_no_m", how="left")
  51.     #月流量使用统计
  52.     phones_app["flow_month"] = phones_app["flow_sum"] / phones_app["month_ids"]
  53.     return phones_app
  54.  
  55. def get_voc_feat(df):
  56.     df["start_datetime"] = pd.to_datetime(df['start_datetime'] )
  57.     df["hour"] = df['start_datetime'].dt.hour
  58.     df["day"] = df['start_datetime'].dt.day
  59.     phone_no_m = df[["phone_no_m"]].copy()
  60.     phone_no_m = phone_no_m.drop_duplicates(subset=['phone_no_m'], keep='last')
  61.     #对话人数和对话次数
  62.     tmp = df.groupby("phone_no_m")["opposite_no_m"].agg(opposite_count="count", opposite_unique="nunique")
  63.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  64.     """主叫通话
  65.     """
  66.     df_call = df[df["calltype_id"]==1].copy()
  67.     tmp = df_call.groupby("phone_no_m")["imei_m"].agg(voccalltype1="count", imeis="nunique")
  68.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  69.     phone_no_m["voc_calltype1"] = phone_no_m["voccalltype1"] / phone_no_m["opposite_count"] 
  70.     tmp = df_call.groupby("phone_no_m")["city_name"].agg(city_name_call="nunique")
  71.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  72.     tmp = df_call.groupby("phone_no_m")["county_name"].agg(county_name_call="nunique")
  73.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  74.     
  75.     """和固定通话者的对话统计
  76.     """
  77.     tmp = df.groupby(["phone_no_m","opposite_no_m"])["call_dur"].agg(count="count", sum="sum")
  78.     phone2opposite = tmp.groupby("phone_no_m")["count"].agg(phone2opposite_mean="mean"
  79.                                                             , phone2opposite_median="median"
  80.                                                             , phone2opposite_max="max"
  81.                                                             , phone2opposite_min="min"
  82.                                                             , phone2opposite_var="var"
  83.                                                             , phone2opposite_skew="skew"
  84.                                                             , phone2opposite_sem="sem"
  85.                                                             , phone2opposite_std="std"
  86.                                                             , phone2opposite_quantile="quantile"
  87.                                                            )
  88.     phone_no_m = phone_no_m.merge(phone2opposite, on="phone_no_m", how="left")
  89.     phone2opposite = tmp.groupby("phone_no_m")["sum"].agg(phone2oppo_sum_mean="mean"
  90.                                                           , phone2oppo_sum_median="median"
  91.                                                           , phone2oppo_sum_max="max"
  92.                                                           , phone2oppo_sum_min="min"
  93.                                                           , phone2oppo_sum_var="var"
  94.                                                           , phone2oppo_sum_skew="skew"
  95.                                                           , phone2oppo_sum_sem="sem"
  96.                                                           , phone2oppo_sum_std="std"
  97.                                                           , phone2oppo_sum_quantile="quantile"
  98.                                                          )
  99.     phone_no_m = phone_no_m.merge(phone2opposite, on="phone_no_m", how="left")
  100.     
  101.     """通话时间长短统计
  102.     """
  103.     tmp = df.groupby("phone_no_m")["call_dur"].agg(call_dur_mean="mean"
  104.                                                    , call_dur_median="median"
  105.                                                    , call_dur_max="max"
  106.                                                    , call_dur_min="min"
  107.                                                    , call_dur_var="var"
  108.                                                    , call_dur_skew="skew"
  109.                                                    , call_dur_sem="sem"
  110.                                                    , call_dur_std="std"
  111.                                                    , call_dur_quantile="quantile"
  112.                                                   )
  113.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  114.     
  115.     tmp = df.groupby("phone_no_m")["city_name"].agg(city_name_nunique="nunique")
  116.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  117.     tmp = df.groupby("phone_no_m")["county_name"].agg(county_name_nunique="nunique")
  118.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  119.     tmp = df.groupby("phone_no_m")["calltype_id"].agg(calltype_id_unique="nunique")
  120.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  121.     
  122.     """通话时间点偏好
  123.     """
  124.     tmp = df.groupby("phone_no_m")["hour"].agg(voc_hour_mode = lambda x:stats.mode(x)[0][0], 
  125.                                                voc_hour_mode_count = lambda x:stats.mode(x)[1][0], 
  126.                                                voc_hour_nunique="nunique")
  127.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  128.     
  129.     tmp = df.groupby("phone_no_m")["day"].agg(voc_day_mode = lambda x:stats.mode(x)[0][0], 
  130.                                                voc_day_mode_count = lambda x:stats.mode(x)[1][0], 
  131.                                                voc_day_nunique="nunique")
  132.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  133.     return phone_no_m
  134.  
  135. def get_sms_feats(df):
  136.     df['request_datetime'] = pd.to_datetime(df['request_datetime'] )
  137.     df["hour"] = df['request_datetime'].dt.hour
  138.     df["day"] = df['request_datetime'].dt.day
  139.  
  140.     phone_no_m = df[["phone_no_m"]].copy()
  141.     phone_no_m = phone_no_m.drop_duplicates(subset=['phone_no_m'], keep='last')
  142.     #对话人数和对话次数
  143.     tmp = df.groupby("phone_no_m")["opposite_no_m"].agg(sms_count="count", sms_nunique="nunique")
  144.     tmp["sms_rate"] = tmp["sms_count"]/tmp["sms_nunique"]
  145.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  146.     """短信下行比例
  147.     """
  148.     calltype2 = df[df["calltype_id"]==2].copy()
  149.     calltype2 = calltype2.groupby("phone_no_m")["calltype_id"].agg(calltype_2="count")
  150.     phone_no_m = phone_no_m.merge(calltype2, on="phone_no_m", how="left")
  151.     phone_no_m["calltype_rate"] = phone_no_m["calltype_2"] / phone_no_m["sms_count"]
  152.     """短信时间
  153.     """
  154.     tmp = df.groupby("phone_no_m")["hour"].agg(hour_mode = lambda x:stats.mode(x)[0][0], 
  155.                                                hour_mode_count = lambda x:stats.mode(x)[1][0], 
  156.                                                hour_nunique="nunique")
  157.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  158.     
  159.     tmp = df.groupby("phone_no_m")["day"].agg(day_mode = lambda x:stats.mode(x)[0][0], 
  160.                                                day_mode_count = lambda x:stats.mode(x)[1][0], 
  161.                                                day_nunique="nunique")
  162.     phone_no_m = phone_no_m.merge(tmp, on="phone_no_m", how="left")
  163.     
  164.     return phone_no_m
  165.  
  166. def feats():
  167.     test_voc=pd.read_csv(path+'test/test_voc.csv',)
  168.     test_voc_feat = get_voc_feat(test_voc)
  169.     test_voc_feat.to_csv(feat_path + "test_voc_feat.csv", index=False)
  170.  
  171.     test_app=pd.read_csv(path+'test/test_app.csv',)
  172.     test_app_feat = get_app_feats(test_app)
  173.     test_app_feat.to_csv(feat_path + "test_app_feat.csv", index=False)
  174.     
  175.     test_sms=pd.read_csv(path+'test/test_sms.csv',)
  176.     test_sms_feat = get_sms_feats(test_sms)
  177.     test_sms_feat.to_csv(feat_path + "test_sms_feat.csv", index=False)
  178.      
  179.     train_voc=pd.read_csv(path+'train/train_voc.csv',)
  180.     train_voc_feat = get_voc_feat(train_voc)
  181.     train_voc_feat.to_csv(feat_path + "train_voc_feat.csv", index=False)
  182.  
  183.     train_app=pd.read_csv(path+'train/train_app.csv',)
  184.     train_app_feat = get_app_feats(train_app)
  185.     train_app_feat.to_csv(feat_path + "train_app_feat.csv", index=False)
  186.  
  187.     train_sms=pd.read_csv(path+'train/train_sms.csv',)
  188.     train_sms_feat = get_sms_feats(train_sms)
  189.     train_sms_feat.to_csv(feat_path + "train_sms_feat.csv", index=False)
  190.     
  191.     
  192.     test_vocfs=pd.read_csv(path + 'zpfsdata/test_voc.csv',)
  193.     test_voc_featfs = get_voc_feat(test_vocfs)
  194.     test_voc_featfs.to_csv(path + "zpfsdata/test_voc_feat.csv", index=False)
  195.  
  196.     test_appfs=pd.read_csv(path + 'zpfsdata/test_app.csv',)
  197.     test_app_featfs = get_app_feats(test_appfs)
  198.     test_app_featfs.to_csv(path + "zpfsdata/test_app_feat.csv", index=False)
  199.     
  200.     test_smsfs=pd.read_csv(path + 'zpfsdata/test_sms.csv',)
  201.     test_sms_featfs = get_sms_feats(test_smsfs)
  202.     test_sms_featfs.to_csv(path + "zpfsdata/test_sms_feat.csv", index=False)

生成特征：

  1. #create and save voc、app、sms features
  2. feats()

数据加载：

  1. #load april features
  2. test_app_feat=pd.read_csv(feat_path+'test_app_feat.csv')
  3. test_voc_feat=pd.read_csv(feat_path+'test_voc_feat.csv')
  4. test_sms_feat=pd.read_csv(feat_path + "test_sms_feat.csv")
  5. test_user=pd.read_csv(path+'test/test_user.csv')
  6. test_user = test_user.merge(test_app_feat, on="phone_no_m", how="left")
  7. test_user = test_user.merge(test_voc_feat, on="phone_no_m", how="left")
  8. test_user = test_user.merge(test_sms_feat, on="phone_no_m", how="left")
  9. test_user["city_name"] = LabelEncoder().fit_transform(test_user["city_name"].astype(np.str))
  10. test_user["county_name"] = LabelEncoder().fit_transform(test_user["county_name"].astype(np.str))
  11. #load april label
  12. test_user_lb1 = pd.read_csv(path + 'zpfsdata/4yuelabel1.csv')
  13. test_user_lb2 = pd.read_csv(path + 'zpfsdata/4yuelabel2.csv')
  14. #concat april label and merge with features 
  15. test_user_label = pd.concat([test_user_lb1, test_user_lb2])
  16. test_user = test_user.merge(test_user_label, on="phone_no_m", how="left")
  17. test_user.rename(columns={"arpu_202004":"arpu_202005"},inplace=True)
  18.  
  19. #load train features and label
  20. train_app_feat = pd.read_csv(feat_path + "train_app_feat.csv")
  21. train_voc_feat = pd.read_csv(feat_path + "train_voc_feat.csv")
  22. train_sms_feat = pd.read_csv(feat_path + "train_sms_feat.csv")
  23. train_user=pd.read_csv(path+'train/train_user.csv')
  24. drop_r = ["arpu_201908","arpu_201909","arpu_201910","arpu_201911","arpu_201912","arpu_202001","arpu_202002"]
  25. train_user.drop(drop_r, axis=1,inplace=True)
  26. train_user.rename(columns={"arpu_202003":"arpu_202005"},inplace=True)
  27. train_user = train_user.merge(train_app_feat, on="phone_no_m", how="left")
  28. train_user = train_user.merge(train_voc_feat, on="phone_no_m", how="left")
  29. train_user = train_user.merge(train_sms_feat, on="phone_no_m", how="left")
  30. train_user["city_name"] = LabelEncoder().fit_transform(train_user["city_name"].astype(np.str))
  31. train_user["county_name"] = LabelEncoder().fit_transform(train_user["county_name"].astype(np.str))
  32.  
  33. #concat preli data(train and test)
  34. train_user = pd.concat([train_user, test_user])
  35. #final label
  36. train_label = train_user[["label"]].copy()
  37.  
  38. #drop phone_no_m
  39. test_user.drop(["phone_no_m"], axis=1,inplace=True)
  40. train_user.drop(["phone_no_m", "label"], axis=1,inplace=True)
  41.  
  42. #load final test features as testfs, fs means fusai
  43. test_app_featfs=pd.read_csv(path + 'zpfsdata/test_app_feat.csv')
  44. test_voc_featfs=pd.read_csv(path + 'zpfsdata/test_voc_feat.csv')
  45. test_sms_featfs=pd.read_csv(path + 'zpfsdata/test_sms_feat.csv')
  46. test_userfs=pd.read_csv(path + 'zpfsdata/test_user.csv')
  47. test_userfs = test_userfs.merge(test_app_featfs, on="phone_no_m", how="left")
  48. test_userfs = test_userfs.merge(test_voc_featfs, on="phone_no_m", how="left")
  49. test_userfs = test_userfs.merge(test_sms_featfs, on="phone_no_m", how="left")
  50. test_userfs["city_name"] = LabelEncoder().fit_transform(test_userfs["city_name"].astype(np.str))
  51. test_userfs["county_name"] = LabelEncoder().fit_transform(test_userfs["county_name"].astype(np.str))
  52.  
  53. #create submission dataframe
  54. submission = test_userfs[["phone_no_m"]].copy()
  55. #drop phone_no_m
  56. test_userfs.drop(["phone_no_m"], axis=1,inplace=True)
  57. #test_userfs.replace([r'\\N'], np.nan, inplace=True)
  58. test_userfs.replace([u'\\N'], np.nan, inplace=True)
  59. test_userfs['arpu_202005'] = test_userfs['arpu_202005'].astype(np.float32)
  60. #col = list(test_userfs.columns)
  61. #test_userfs[col] = test_userfs[col].apply(pd.to_numeric, errors='coerce')
  62. #test_userfs['arpu_202005'] = test_userfs['arpu_202005'].apply(lambda x:x.replace('\n', '').replace('\r', '').replace('\\N', '')).astype(np.float32)
  63. test_userfs_ori = test_userfs

模型搭建：

由于数据量少，服务器性能还可以，于是建模部分非常简单粗暴，直接grid search

  1. depth = 8
  2. cv = 5
  3.  
  4. #create catboost model
  5. catclf = cat.CatBoostClassifier(
  6.     allow_writing_files = False
  7.     , od_type= 'Iter'
  8.     , silent=True
  9. )
  10. #final parameters
  11. cat_grid = {'depth':[depth]
  12.             , 'bootstrap_type':['Bernoulli']
  13.             , 'od_type':['Iter']
  14.             , 'l2_leaf_reg':[15]
  15.             , 'learning_rate': [0.1]
  16.             , 'allow_writing_files':[False]
  17.             , 'silent':[True]
  18.  
  19.            }
  20. #search and fit
  21. catgrid = GridSearchCV(cat.CatBoostClassifier(), param_grid=cat_grid, cv=cv, scoring='f1_macro', n_jobs=-1, verbose = 10)
  22. catgrid.fit( train_user, train_label['label'] )
  23.  
  24. #predict output prob
  25. test_userfs = test_userfs_ori.fillna( test_userfs_ori.quantile(0.39) )
  26. test_userfs['arpu_202005'] = test_userfs['arpu_202005'].astype(np.float32)
  27. cat_proba = catgrid.predict_proba( test_userfs )
  28. rslt_prob_cat = pd.DataFrame( cat_proba )
  29. rslt_prob_cat.columns = ['lb0','lb1']
  30.  
  31. #create lgb model
  32. #final parameters
  33. lgb_grid = {'booster':['gbdt']
  34.             , 'num_leaves':[256]
  35.             , 'min_child_weight':[4]
  36.             , 'feature_fraction':[0.7]
  37.             , 'bagging_fraction':[0.8]
  38.             , 'bagging_freq': [1]
  39.            }
  40.  
  41. #search and fit
  42. lgbgrid = GridSearchCV(lgb.LGBMClassifier(), param_grid=lgb_grid, cv=cv, scoring='f1_macro', n_jobs=-1, verbose = 10)
  43. lgbgrid.fit( train_user, train_label['label'] )
  44. #predict output prob
  45. test_userfs = test_userfs_ori.fillna( test_userfs_ori.quantile(0.34) )
  46. test_userfs['arpu_202005'] = test_userfs['arpu_202005'].astype(np.float32)
  47. lgb_proba = lgbgrid.predict_proba( test_userfs )
  48. rslt_prob_lgb = pd.DataFrame(lgb_proba)
  49. rslt_prob_lgb.columns = ['lb0','lb1']
  50.  
  51. #create xgb model
  52. #final parameters
  53. from xgboost import XGBClassifier
  54. xgbclf=XGBClassifier(base_score=0.5
  55.                      , booster='gbtree'
  56.                      , colsample_bytree=0.9
  57.                      , learning_rate=0.1
  58.                      , max_depth=8
  59.                      , min_child_weight=7
  60.                      , n_estimators=100
  61.                      , n_jobs=-1
  62.                      , objective='binary:logistic'
  63.                      , subsample=0.75
  64.                      , verbosity=1)
  65. #fit
  66. xgbclf.fit( train_user, train_label['label'] )
  67. #predict output prob
  68. test_userfs = test_userfs_ori.fillna( test_userfs_ori.quantile(0.319) )
  69. test_userfs['arpu_202005'] = test_userfs['arpu_202005'].astype(np.float32)
  70. xgb_proba = xgbclf.predict_proba( test_userfs )
  71. rslt_prob_xgb = pd.DataFrame(lgb_proba)
  72. rslt_prob_xgb.columns = ['lb0','lb1']

调整概率输出：

  1. bestnew112 =  0.25*rslt_prob_lgb + 0.25*rslt_prob_xgb + 0.5*rslt_prob_cat
  2.  
  3. bestnew112["label"]=bestnew112["lb1"]
  4. bestnew112["label"][bestnew112.label>60/100]=1
  5. bestnew112["label"][bestnew112.label<60/100]=0
  6.     
  7. sub['label'] = bestnew112['label']
  8.  
  9. print(sub['label'].value_counts())
  10. print(sub['label'].value_counts()/sub.shape[0])
  11.  
  12. sub.to_csv('lgb25xgb25cat50threshold60.csv',index=None)

以上结果B榜0.9005

本文内容由网友自发贡献，转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号