devbox
秋刀鱼在做梦
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

论文代码阅读及部分复现：Revisiting Deep Learning Models for Tabular Data

作者：秋刀鱼在做梦 | 2024-07-03 03:51:05

赞

踩

revisiting deep learning models for tabular data

论文地址：https://arxiv.org/pdf/2106.11959.pdf

项目地址：GitHub - yandex-research/rtdl-revisiting-models: (NeurIPS 2021) Revisiting Deep Learning Models for Tabular Data

相关数据：https://www.dropbox.com/s/o53umyg6mn3zhxy/ 

2024年2月11日补充：

此处的LassoNet模型实际上只是带skip层MLP；实际的LassoNet作特征筛选时还需要更新lambda等值，耗时较长，此处仅取了中间的循环。

一、论文概述

现有的关于表格数据做深度学习的模型层出不穷，但是作者认为，由于在真实使用模型时有着不同的基准以及实验场合，这些提出的模型没有被很好地比较。因此，论文作者在论文中对各类模型进行了综述，并且自身提出了一个对Transformer作简单改进的模型：FT-Transformer，最终将ResNet-like类模型、Transformer-like类模型以及其他MLP模型在不同的数据集上训练、对比效果，最终确定了一个较好的衡量针对表格数据的深度学习模型的标准（bennchmark）。但是和梯度提升的决策树模型相比，还没有很好的基于深度学习的模型。

二、使用到的模型

1、MLP:也就是最常见的多层感知器，使用Relu激活函数与Dropout层

2、ResNet:残差神经网络，由残差块（ResNetBlock）组成，残差块可以理解为以下函数：H(x)=x+F(x) 其中F(x)=Dropout(Linear(Dropout(Linear(BatchNorm(x)))))

 3、FT-Transformer:这是论文作者提出的一个Transformer的简单变种，简而言之就是在传入Transformer之前加了一个Feature Tokenizer：将连续变量作线性变换，离散变量作Embedding，最后再加入一个CLS向量作为结果向量；将处理完的数据传入一个Encoder-only的Transformer中。

4、SNN：自归一化神经网络（注意不是“脉冲神经网络”），在原本的MLP模型基础上使用了SELU激活函数，能够训练更深的模型。

5、NODE:Neural oblivious decision ensembles，在神经网络中加入了决策树原理

6、TabNet:和NODE一样，也是在神经网络中加入了决策树原理

7、GrowNet:在神经网络中加入Boosting的思想

8、DCN:在Wide and Deep上改进，将线性模型部分换成Cross Network。在Cross Network中，每一层的输出都会乘以原始的输入特征。这个模型也是统一输入使用Embedding处理离散特征。

9、AutoInt:认为浅层模型收到交叉阶数限制，且DNN在高阶隐性交叉结果不好；这个模型加入了注意力机制。模型在输入时，会同时将离散特征与连续特征进行Embedding，将其分为三个矩阵:Query,Key,Values 将Query与Key作内积衡量相似度，使用Softmax得到attention，最终将Attention乘以values，得到一个head的结果。

10、CatBoost:使用了Ordered Target Statistics来处理多分类变量，避免了多分类变量作One-hot处理会产生维数爆炸的问题。

11、XGBoost:对损失函数进行了二阶泰勒展开，从而可以在训练时使用二阶导。随着版本迭代，现在的XGBoost也可以处理多分类变量了。

三、实验策略

首先，实验对每个模型在每个数据集上都使用了相同的预处理，大多数数据都是用了分位数转换处理，而数据集Helena和ALOI则使用了标准化（standard），Epsilon没有使用任何预处理。对于回归任务，所有的应变量都被做了标准化。

而对于每个模型，首先会使用Optuna作贝叶斯回归在验证集上调优获得一个“最优”的超参数，然后使用15个不同的随机数种子在，将这15个模型分成3组，每组平均单个预测模型。

调优流程

论文作者将所有调优的过程放到了tune.py这一个文件中，想要调优时只要运行这一个文件然后加上要调优的配置文件(toml)就可以了，github上的例子就是：

python bin/tune.py output/california_housing/mlp/tuning/reproduced.toml

以tf-transformer的toml配置文件为例：

  1. program = 'bin/ft_transformer.py'
  2.  
  3. [base_config]
  4. seed = 0
  5.  
  6.     [base_config.data]
  7.     normalization = 'quantile'
  8.     path = 'data/california_housing'
  9.     y_policy = 'mean_std'
  10.  
  11.     [base_config.model]
  12.     activation = 'reglu'
  13.     initialization = 'kaiming'
  14.     n_heads = 8
  15.     prenormalization = true
  16.  
  17.     [base_config.training]
  18.     batch_size = 256
  19.     eval_batch_size = 8192
  20.     n_epochs = 1000000000
  21.     optimizer = 'adamw'
  22.     patience = 16
  23.  
  24. [optimization.options]
  25. n_trials = 100
  26.  
  27. [optimization.sampler]
  28. seed = 0
  29.  
  30. [optimization.space.model]
  31. attention_dropout = [ 'uniform', 0.0, 0.5 ]
  32. d_ffn_factor = [ '$d_ffn_factor', 1.0, 4.0 ]
  33. d_token = [ '$d_token', 64, 512 ]
  34. ffn_dropout = [ 'uniform', 0.0, 0.5 ]
  35. n_layers = [ 'int', 1, 4 ]
  36. residual_dropout = [ '?uniform', 0.0, 0.0, 0.2 ]
  37.  
  38. [optimization.space.training]
  39. lr = [ 'loguniform', 1e-05, 0.001 ]
  40. weight_decay = [ 'loguniform', 1e-06, 0.001 ]

其中，program表明模型定义以及训练函数定义的位置，base_config是在训练对应数据集的固定参数，里面包含有以下信息：

seed:模型训练的随机数种子

data:记录了一切对数据进行的预处理，如正则化，数据路径以及y值预处理操作

model:记录了模型固定的参数，通常是决定模型本身结构与深度的参数，这部分参数不会作调优

training:记录了使用模型作训练以及验证时的参数，如batch_size，epoch数量、优化器以及patience等。这里的patience指的是又连续多少个epoch在验证集上没有改进后才会停止训练。

optimization:使用Optuna调优时的所用到的参数，其中options与sampler是作贝叶斯回归时的参数以及抽样参数，space.model是模型的参数空间，Optuna会在这些参数中找到最优的组合，space.training是训练时的参数，如学习率/权重衰减等，也是Optuna所要进行抽样选取最优解的对象。

读取toml格式的文件会转化为字典。

下面是实验调优的代码。

  1. program = lib.get_path(args['program'])
  2. program_copy = program.with_name(
  3.     program.stem + '___' + str(uuid.uuid4()).replace('-', '') + program.suffix
  4. )
  5. shutil.copyfile(program, program_copy)
  6. atexit.register(lambda: program_copy.unlink())
  7.  
  8. checkpoint_path = output / 'checkpoint.pt'
  9. if checkpoint_path.exists():
  10.     checkpoint = torch.load(checkpoint_path)
  11.     trial_configs, trial_stats, study, stats, timer = (
  12.         checkpoint['trial_configs'],
  13.         checkpoint['trial_stats'],
  14.         checkpoint['study'],
  15.         checkpoint['stats'],
  16.         checkpoint['timer'],
  17.     )
  18.     zero.set_random_state(checkpoint['random_state'])
  19.     if 'n_trials' in args['optimization']['options']:
  20.         args['optimization']['options']['n_trials'] -= len(study.trials)
  21.     if 'timeout' in args['optimization']['options']:
  22.         args['optimization']['options']['timeout'] -= timer()
  23.     stats.setdefault('continuations', []).append(len(study.trials))
  24.     print(f'Loading checkpoint ({len(study.trials)})')
  25. else:
  26.     stats = lib.load_json(output / 'stats.json')
  27.     trial_configs = []
  28.     trial_stats = []
  29.     timer = zero.Timer()
  30.     study = optuna.create_study(
  31.         direction='maximize',
  32.         sampler=optuna.samplers.TPESampler(**args['optimization']['sampler']),
  33.     )
  34.  
  35. timer.run()
  36. # ignore the progress bar warning
  37. warnings.filterwarnings('ignore', category=optuna.exceptions.ExperimentalWarning)
  38. study.optimize(
  39.     objective,
  40.     **args['optimization']['options'],
  41.     callbacks=[save_checkpoint],
  42.     show_progress_bar=True,
  43. )
  44.  
  45. best_trial_id = study.best_trial.number
  46. lib.dump_toml(trial_configs[best_trial_id], output / 'best.toml')
  47. stats['best_stats'] = trial_stats[best_trial_id]
  48. stats['time'] = lib.format_seconds(timer())
  49. lib.dump_stats(stats, output, True)
  50. lib.backup_output(output)

首先程序会临时拷贝一份模型的定义文件，等到进程结束之后删除。此处的目的我估计是为了避免有多个进程访问同一个py文件。

之后开始检查有没有checkpoint，如果有checkpoint的话，会在checkpoint的基础上继续做调优，减去原本已经做完的的trial个数，否则的话就从头开始进行调优。

之后开始运行study.optimize函数，开始进行调优，其中objective对象就是我们定义的要调优的对象。

  1. def sample_parameters(
  2.     trial: optuna.trial.Trial,
  3.     space: ty.Dict[str, ty.Any],
  4.     base_config: ty.Dict[str, ty.Any],
  5. ) -> ty.Dict[str, ty.Any]:
  6.     def get_distribution(distribution_name):
  7.         return getattr(trial, f'suggest_{distribution_name}')
  8.  
  9.     result = {}
  10.     for label, subspace in space.items():
  11.         if isinstance(subspace, dict):
  12.             result[label] = sample_parameters(trial, subspace, base_config)
  13.         else:
  14.             assert isinstance(subspace, list)
  15.             distribution, *args = subspace
  16.  
  17.             if distribution.startswith('?'):
  18.                 default_value = args[0]
  19.                 result[label] = (
  20.                     get_distribution(distribution.lstrip('?'))(label, *args[1:])
  21.                     if trial.suggest_categorical(f'optional_{label}', [False, True])
  22.                     else default_value
  23.                 )
  24.  
  25.             elif distribution == '$mlp_d_layers':
  26.                 min_n_layers, max_n_layers, d_min, d_max = args
  27.                 n_layers = trial.suggest_int('n_layers', min_n_layers, max_n_layers)
  28.                 suggest_dim = lambda name: trial.suggest_int(name, d_min, d_max)  # noqa
  29.                 d_first = [suggest_dim('d_first')] if n_layers else []
  30.                 d_middle = (
  31.                     [suggest_dim('d_middle')] * (n_layers - 2) if n_layers > 2 else []
  32.                 )
  33.                 d_last = [suggest_dim('d_last')] if n_layers > 1 else []
  34.                 result[label] = d_first + d_middle + d_last
  35.  
  36.             elif distribution == '$d_token':
  37.                 assert len(args) == 2
  38.                 try:
  39.                     n_heads = base_config['model']['n_heads']
  40.                 except KeyError:
  41.                     n_heads = base_config['model']['n_latent_heads']
  42.  
  43.                 for x in args:
  44.                     assert x % n_heads == 0
  45.                 result[label] = trial.suggest_int('d_token', *args, n_heads)  # type: ignore[code]
  46.  
  47.             elif distribution in ['$d_ffn_factor', '$d_hidden_factor']:
  48.                 if base_config['model']['activation'].endswith('glu'):
  49.                     args = (args[0] * 2 / 3, args[1] * 2 / 3)
  50.                 result[label] = trial.suggest_uniform('d_ffn_factor', *args)
  51.  
  52.             else:
  53.                 result[label] = get_distribution(distribution)(label, *args)
  54.     return result
  55.  
  56. def merge_sampled_parameters(config, sampled_parameters):
  57.     for k, v in sampled_parameters.items():
  58.         if isinstance(v, dict):
  59.             merge_sampled_parameters(config.setdefault(k, {}), v)
  60.         else:
  61.             assert k not in config
  62.             config[k] = v
  63.  
  64. def objective(trial: optuna.trial.Trial) -> float:
  65.     config = deepcopy(args['base_config'])
  66.     merge_sampled_parameters(
  67.         config, sample_parameters(trial, args['optimization']['space'], config)
  68.     )
  69.     if args.get('config_type') in ['trv2', 'trv4']:
  70.         config['model']['d_token'] -= (
  71.             config['model']['d_token'] % config['model']['n_heads']
  72.         )
  73.     if args.get('config_type') == 'trv4':
  74.         if config['model']['activation'].endswith('glu'):
  75.             # This adjustment is needed to keep the number of parameters roughly in the
  76.             # same range as for non-glu activations
  77.             config['model']['d_ffn_factor'] *= 2 / 3
  78.     trial_configs.append(config)
  79.  
  80.     with tempfile.TemporaryDirectory() as dir_:
  81.         dir_ = Path(dir_)
  82.         out = dir_ / f'trial_{trial.number}'
  83.         config_path = out.with_suffix('.toml')
  84.         lib.dump_toml(config, config_path)
  85.         python = Path('/miniconda3/envs/main/bin/python')
  86.         subprocess.run(
  87.             [
  88.                 str(python) if python.exists() else "python",
  89.                 str(program_copy),
  90.                 str(config_path),
  91.             ],
  92.             check=True,
  93.         ) #训练 subprocess.run可以取得返回结果等信息
  94.         stats = lib.load_json(out / 'stats.json')
  95.         stats['algorithm'] = stats['algorithm'].rsplit('___', 1)[0]
  96.         trial_stats.append(
  97.             {
  98.                 **stats,
  99.                 'trial_id': trial.number,
  100.                 'tuning_time': lib.format_seconds(timer()),
  101.             }
  102.         )
  103.         lib.dump_json(trial_stats, output / 'trial_stats.json', indent=4)
  104.         lib.backup_output(output)
  105.         print(f'Time: {lib.format_seconds(timer())}')
  106.         return stats['metrics'][lib.VAL]['score']

trv2和trv4在这个项目中没有用到，应该是论文作者的团队在别的项目中用到的参数。

首先使用sample_parameters函数将toml中的那些参数空间转化为Optuna中trial的属性。注意suggest_uniform在Optuna3.0版本开始传入参数不再支持单个*args了，所以最后2个分支需要改为：

  1.                 result[label] = trial.suggest_float('d_ffn_factor', args[0], args[1])#trial.suggest_uniform('d_ffn_factor', *args)
  2.  
  3.             else:
  4.                 if distribution == "uniform":
  5.                     result[label] = trial.suggest_float(label,args[0], args[1])
  6.                 else:
  7.                     result[label] = get_distribution(distribution)(label, *args)
  8.     return result

最终转换为trial中的需要调优的参数：

再使用merge_sampled_parameters将所有的参数放在一个config字典变量中，最后使用subprocess.run函数，将config对象传入模型定义的py文件进行训练，将结果保存并返回。这样，就能构建出用于Optuna优化的objective函数了。

注意此处某些参数的空间会有特殊处理：

如果参数在toml文件中标了"?"，如dropout = [ '?uniform', 0.0, 0.0, 0.5 ]，就意味着这个参数是“可选调优项”。此处的dropout的配置的含义是：首先在“要不要对这个参数调优”这个样本空间里进行抽样，如果不要调优的话，就分配其默认值0。如果要调优的话，就在后面0~0.5的均匀分布中抽样。

而对于$mlp_d_layers的特殊参数空间而言，会对以下4个部分进行参数空间的创建：需要多少层中间层，第一层，最后一层以及中间层分别要多大。

对于$d_token参数，通常是在Transformer-like的模型中，故而会检测其能否被注意力头数(n_head)整除。

剩下$d_ffn_factor和$d_hidden_factor这2个参数，当使用glu系的激活函数时，会进行特殊处理（除以3）。

下面纤细讲解一下数据预处理流程以及模型训练流程。

数据预处理流程

首先会创建一个数据集对象，读取数据文件中的npy文件。

  1. @dc.dataclass
  2. class Dataset:
  3.     N: ty.Optional[ArrayDict]
  4.     C: ty.Optional[ArrayDict]
  5.     y: ArrayDict
  6.     info: ty.Dict[str, ty.Any]
  7.     folder: ty.Optional[Path]
  8.  
  9.     @classmethod
  10.     def from_dir(cls, dir_: ty.Union[Path, str]) -> 'Dataset':
  11.         dir_ = Path(dir_)
  12.  
  13.         def load(item) -> ArrayDict:
  14.             return {
  15.                 x: ty.cast(np.ndarray, np.load(dir_ / f'{item}_{x}.npy'))  # type: ignore[code]
  16.                 for x in ['train', 'val', 'test']
  17.             }
  18.  
  19.         return Dataset(
  20.             load('N') if dir_.joinpath('N_train.npy').exists() else None,
  21.             load('C') if dir_.joinpath('C_train.npy').exists() else None,
  22.             load('y'),
  23.             util.load_json(dir_ / 'info.json'),
  24.             dir_,
  25.         )

其中，C是离散类变量，N是连续型变量，Y是因变量。 之后，分别对X和Y进行数据预处理。

  1. def normalize(
  2.     X: ArrayDict, normalization: str, seed: int, noise: float = 1e-3
  3. ) -> ArrayDict:
  4.     X_train = X['train'].copy()
  5.     if normalization == 'standard':
  6.         normalizer = sklearn.preprocessing.StandardScaler()
  7.     elif normalization == 'quantile':
  8.         normalizer = sklearn.preprocessing.QuantileTransformer(
  9.             output_distribution='normal',
  10.             n_quantiles=max(min(X['train'].shape[0] // 30, 1000), 10),
  11.             subsample=1e9,
  12.             random_state=seed,
  13.         )
  14.         if noise:
  15.             stds = np.std(X_train, axis=0, keepdims=True)
  16.             noise_std = noise / np.maximum(stds, noise)  # type: ignore[code]
  17.             X_train += noise_std * np.random.default_rng(seed).standard_normal(  # type: ignore[code]
  18.                 X_train.shape
  19.             )
  20.     else:
  21.         util.raise_unknown('normalization', normalization)
  22.     normalizer.fit(X_train)
  23.     return {k: normalizer.transform(v) for k, v in X.items()}  # type: ignore[code]
  24.  
  25. def build_X(
  26. 	self,
  27. 	*,
  28. 	normalization: ty.Optional[str],
  29. 	num_nan_policy: str,
  30. 	cat_nan_policy: str,
  31. 	cat_policy: str,
  32. 	cat_min_frequency: float = 0.0,
  33. 	seed: int,
  34. ) -> ty.Union[ArrayDict, ty.Tuple[ArrayDict, ArrayDict]]:
  35. 	cache_path = (
  36. 		self.folder
  37. 		/ f'build_X__{normalization}__{num_nan_policy}__{cat_nan_policy}__{cat_policy}__{seed}.pickle'  # noqa
  38. 		if self.folder
  39. 		else None
  40. 	)
  41. 	if cache_path and cat_min_frequency:
  42. 		cache_path = cache_path.with_name(
  43. 			cache_path.name.replace('.pickle', f'__{cat_min_frequency}.pickle')
  44. 		)
  45. 	if cache_path and cache_path.exists():
  46. 		print(f'Using cached X: {cache_path}')
  47. 		with open(cache_path, 'rb') as f:
  48. 			return pickle.load(f)
  49.  
  50. 	def save_result(x):
  51. 		if cache_path:
  52. 			with open(cache_path, 'wb') as f:
  53. 				pickle.dump(x, f)
  54.  
  55. 	if self.N:
  56. 		N = deepcopy(self.N)
  57.  
  58. 		num_nan_masks = {k: np.isnan(v) for k, v in N.items()}
  59. 		if any(x.any() for x in num_nan_masks.values()):  # type: ignore[code]
  60. 			if num_nan_policy == 'mean':
  61. 				num_new_values = np.nanmean(self.N['train'], axis=0)
  62. 			else:
  63. 				util.raise_unknown('numerical NaN policy', num_nan_policy)
  64. 			for k, v in N.items():
  65. 				num_nan_indices = np.where(num_nan_masks[k])
  66. 				v[num_nan_indices] = np.take(num_new_values, num_nan_indices[1])
  67. 		if normalization:
  68. 			N = normalize(N, normalization, seed)
  69.  
  70. 	else:
  71. 		N = None
  72.  
  73. 	if cat_policy == 'drop' or not self.C:
  74. 		assert N is not None
  75. 		save_result(N)
  76. 		return N
  77.  
  78. 	C = deepcopy(self.C)
  79.  
  80. 	cat_nan_masks = {k: v == 'nan' for k, v in C.items()}
  81. 	if any(x.any() for x in cat_nan_masks.values()):  # type: ignore[code]
  82. 		if cat_nan_policy == 'new':
  83. 			cat_new_value = '___null___'
  84. 			imputer = None
  85. 		elif cat_nan_policy == 'most_frequent':
  86. 			cat_new_value = None
  87. 			imputer = SimpleImputer(strategy=cat_nan_policy)  # type: ignore[code]
  88. 			imputer.fit(C['train'])
  89. 		else:
  90. 			util.raise_unknown('categorical NaN policy', cat_nan_policy)
  91. 		if imputer:
  92. 			C = {k: imputer.transform(v) for k, v in C.items()}
  93. 		else:
  94. 			for k, v in C.items():
  95. 				cat_nan_indices = np.where(cat_nan_masks[k])
  96. 				v[cat_nan_indices] = cat_new_value
  97.  
  98. 	if cat_min_frequency:
  99. 		C = ty.cast(ArrayDict, C)
  100. 		min_count = round(len(C['train']) * cat_min_frequency)
  101. 		rare_value = '___rare___'
  102. 		C_new = {x: [] for x in C}
  103. 		for column_idx in range(C['train'].shape[1]):
  104. 			counter = Counter(C['train'][:, column_idx].tolist())
  105. 			popular_categories = {k for k, v in counter.items() if v >= min_count}
  106. 			for part in C_new:
  107. 				C_new[part].append(
  108. 					[
  109. 						(x if x in popular_categories else rare_value)
  110. 						for x in C[part][:, column_idx].tolist()
  111. 					]
  112. 				)
  113. 		C = {k: np.array(v).T for k, v in C_new.items()}
  114.  
  115. 	unknown_value = np.iinfo('int64').max - 3
  116. 	encoder = sklearn.preprocessing.OrdinalEncoder(
  117. 		handle_unknown='use_encoded_value',  # type: ignore[code]
  118. 		unknown_value=unknown_value,  # type: ignore[code]
  119. 		dtype='int64',  # type: ignore[code]
  120. 	).fit(C['train'])
  121. 	C = {k: encoder.transform(v) for k, v in C.items()}
  122. 	max_values = C['train'].max(axis=0)
  123. 	for part in ['val', 'test']:
  124. 		for column_idx in range(C[part].shape[1]):
  125. 			C[part][C[part][:, column_idx] == unknown_value, column_idx] = (
  126. 				max_values[column_idx] + 1
  127. 			)
  128.  
  129. 	if cat_policy == 'indices':
  130. 		result = (N, C)
  131. 	elif cat_policy == 'ohe':
  132. 		ohe = sklearn.preprocessing.OneHotEncoder(
  133. 			handle_unknown='ignore', sparse=False, dtype='float32'  # type: ignore[code]
  134. 		)
  135. 		ohe.fit(C['train'])
  136. 		C = {k: ohe.transform(v) for k, v in C.items()}
  137. 		result = C if N is None else {x: np.hstack((N[x], C[x])) for x in N}
  138. 	elif cat_policy == 'counter':
  139. 		assert seed is not None
  140. 		loo = LeaveOneOutEncoder(sigma=0.1, random_state=seed, return_df=False)
  141. 		loo.fit(C['train'], self.y['train'])
  142. 		C = {k: loo.transform(v).astype('float32') for k, v in C.items()}  # type: ignore[code]
  143. 		if not isinstance(C['train'], np.ndarray):
  144. 			C = {k: v.values for k, v in C.items()}  # type: ignore[code]
  145. 		if normalization:
  146. 			C = normalize(C, normalization, seed, inplace=True)  # type: ignore[code]
  147. 		result = C if N is None else {x: np.hstack((N[x], C[x])) for x in N}
  148. 	else:
  149. 		util.raise_unknown('categorical policy', cat_policy)
  150. 	save_result(result)
  151. 	return result  # type: ignore[code]

在最前面的那个cache_path是用来储存相同处理参数下预处理过后的自变量的，在使用相同参数时可以直接拉取出来不用再做一遍预处理过程。之后分别对连续型变量和离散型变量作预处理：先做缺失值填充，再作数据标准化。连续变量的缺失值填充只有平均值填充一个策略，而对于离散型变量，缺失值有2个处理策略：一种是当做一个新的类别作为处理，另外一种则是使用对应特征中种类最多的类别进行填充。还有就是有一个cat_min_frequency参数，可以指定将出现低于cat_min_frequency频率的那些类别进行合并作为'___rare___'类。之后使用OrdinalEncoder将类别变量转换为数字。最后还有个cat_policy决定最终的输出结果：indices表示直接使用类别变量对应的index，one代表使用One-hotEncoder来处理类别变量，而counter则代表使用LeaveOneOutEncoder：将对应行的因变量排除后，其他对应这一类别的特征变量的均值作encoder。

  1. def build_y(
  2. 	self, policy: ty.Optional[str]
  3. ) -> ty.Tuple[ArrayDict, ty.Optional[ty.Dict[str, ty.Any]]]:
  4. 	if self.is_regression:
  5. 		assert policy == 'mean_std'
  6. 	y = deepcopy(self.y)
  7. 	if policy:
  8. 		if not self.is_regression:
  9. 			warnings.warn('y_policy is not None, but the task is NOT regression')
  10. 			info = None
  11. 		elif policy == 'mean_std':
  12. 			mean, std = self.y['train'].mean(), self.y['train'].std()
  13. 			y = {k: (v - mean) / std for k, v in y.items()}
  14. 			info = {'policy': policy, 'mean': mean, 'std': std}
  15. 		else:
  16. 			util.raise_unknown('y policy', policy)
  17. 	else:
  18. 		info = None
  19. 	return y, info

而对于y值的处理，连续变量用单纯地标准化（减去平均值除以方差），离散变量则不使用任何的处理方式（使用了会给出一个warning，而且离散变量作标准化实际上也没有意义）。

训练流程

 此处以论文作者所作的FT-Transformer举例。

  1. if __name__ == "__main__":
  2.     args, output = lib.load_config()
  3.     args['model'].setdefault('token_bias', True)
  4.     args['model'].setdefault('kv_compression', None)
  5.     args['model'].setdefault('kv_compression_sharing', None)
  6.  
  7.     # %%
  8.     zero.set_randomness(args['seed'])
  9.     dataset_dir = lib.get_path(args['data']['path'])
  10.     stats: ty.Dict[str, ty.Any] = {
  11.         'dataset': dataset_dir.name,
  12.         'algorithm': Path(__file__).stem,
  13.         **lib.load_json(output / 'stats.json'), #**用以扩展字典，将另一个字典中的键值对传入
  14.     }
  15.     timer = zero.Timer()
  16.     timer.run()
  17.  
  18.     D = lib.Dataset.from_dir(dataset_dir)
  19.     X = D.build_X(
  20.         normalization=args['data'].get('normalization'),
  21.         num_nan_policy='mean',
  22.         cat_nan_policy='new',
  23.         cat_policy=args['data'].get('cat_policy', 'indices'),
  24.         cat_min_frequency=args['data'].get('cat_min_frequency', 0.0),
  25.         seed=args['seed'],
  26.     )
  27.     if not isinstance(X, tuple):
  28.         X = (X, None)
  29.     zero.set_randomness(args['seed'])
  30.     Y, y_info = D.build_y(args['data'].get('y_policy'))
  31.     lib.dump_pickle(y_info, output / 'y_info.pickle')
  32.     X = tuple(None if x is None else lib.to_tensors(x) for x in X)
  33.     Y = lib.to_tensors(Y)
  34.     device = lib.get_device()
  35.     if device.type != 'cpu':
  36.         X = tuple(
  37.             None if x is None else {k: v.to(device) for k, v in x.items()} for x in X
  38.         )
  39.         Y_device = {k: v.to(device) for k, v in Y.items()}
  40.     else:
  41.         Y_device = Y
  42.     X_num, X_cat = X
  43.     del X
  44.     if not D.is_multiclass:
  45.         Y_device = {k: v.float() for k, v in Y_device.items()}
  46.  
  47.     train_size = D.size(lib.TRAIN)
  48.     batch_size = args['training']['batch_size']
  49.     epoch_size = stats['epoch_size'] = math.ceil(train_size / batch_size)
  50.     eval_batch_size = args['training']['eval_batch_size']
  51.     chunk_size = None
  52.  
  53.     loss_fn = (
  54.         F.binary_cross_entropy_with_logits
  55.         if D.is_binclass
  56.         else F.cross_entropy
  57.         if D.is_multiclass
  58.         else F.mse_loss
  59.     )
  60.     model = Transformer(
  61.         d_numerical=0 if X_num is None else X_num['train'].shape[1],
  62.         categories=lib.get_categories(X_cat),
  63.         d_out=D.info['n_classes'] if D.is_multiclass else 1,
  64.         **args['model'],
  65.     ).to(device)
  66.     if torch.cuda.device_count() > 1:  # type: ignore[code]
  67.         print('Using nn.DataParallel')
  68.         model = nn.DataParallel(model)
  69.     stats['n_parameters'] = lib.get_n_parameters(model)
  70.  
  71.     def needs_wd(name):
  72.         return all(x not in name for x in ['tokenizer', '.norm', '.bias'])
  73.  
  74.     for x in ['tokenizer', '.norm', '.bias']:
  75.         assert any(x in a for a in (b[0] for b in model.named_parameters()))
  76.     parameters_with_wd = [v for k, v in model.named_parameters() if needs_wd(k)]
  77.     parameters_without_wd = [v for k, v in model.named_parameters() if not needs_wd(k)]
  78.     optimizer = lib.make_optimizer(
  79.         args['training']['optimizer'],
  80.         (
  81.             [
  82.                 {'params': parameters_with_wd},
  83.                 {'params': parameters_without_wd, 'weight_decay': 0.0},
  84.             ]
  85.         ),
  86.         args['training']['lr'],
  87.         args['training']['weight_decay'],
  88.     )
  89.  
  90.     stream = zero.Stream(lib.IndexLoader(train_size, batch_size, True, device))
  91.     progress = zero.ProgressTracker(args['training']['patience'])
  92.     training_log = {lib.TRAIN: [], lib.VAL: [], lib.TEST: []}
  93.     timer = zero.Timer()
  94.     checkpoint_path = output / 'checkpoint.pt'
  95.  
  96.     def print_epoch_info():
  97.         print(f'\n>>> Epoch {stream.epoch} | {lib.format_seconds(timer())} | {output}')
  98.         print(
  99.             ' | '.join(
  100.                 f'{k} = {v}'
  101.                 for k, v in {
  102.                     'lr': lib.get_lr(optimizer),
  103.                     'batch_size': batch_size,
  104.                     'chunk_size': chunk_size,
  105.                     'epoch_size': stats['epoch_size'],
  106.                     'n_parameters': stats['n_parameters'],
  107.                 }.items()
  108.             )
  109.         )
  110.  
  111.     def apply_model(part, idx):
  112.         return model(
  113.             None if X_num is None else X_num[part][idx],
  114.             None if X_cat is None else X_cat[part][idx],
  115.         )
  116.  
  117.     @torch.no_grad()
  118.     def evaluate(parts):
  119.         global eval_batch_size
  120.         model.eval()
  121.         metrics = {}
  122.         predictions = {}
  123.         for part in parts:
  124.             while eval_batch_size:
  125.                 try:
  126.                     predictions[part] = (
  127.                         torch.cat(
  128.                             [
  129.                                 apply_model(part, idx)
  130.                                 for idx in lib.IndexLoader(
  131.                                     D.size(part), eval_batch_size, False, device
  132.                                 )
  133.                             ]
  134.                         )
  135.                         .cpu()
  136.                         .numpy()
  137.                     )
  138.                 except RuntimeError as err:
  139.                     if not lib.is_oom_exception(err):
  140.                         raise
  141.                     eval_batch_size //= 2
  142.                     print('New eval batch size:', eval_batch_size)
  143.                     stats['eval_batch_size'] = eval_batch_size
  144.                 else:
  145.                     break
  146.             if not eval_batch_size:
  147.                 RuntimeError('Not enough memory even for eval_batch_size=1')
  148.             metrics[part] = lib.calculate_metrics(
  149.                 D.info['task_type'],
  150.                 Y[part].numpy(),  # type: ignore[code]
  151.                 predictions[part],  # type: ignore[code]
  152.                 'logits',
  153.                 y_info,
  154.             )
  155.         for part, part_metrics in metrics.items():
  156.             print(f'[{part:<5}]', lib.make_summary(part_metrics))
  157.         return metrics, predictions
  158.  
  159.     def save_checkpoint(final):
  160.         torch.save(
  161.             {
  162.                 'model': model.state_dict(),
  163.                 'optimizer': optimizer.state_dict(),
  164.                 'stream': stream.state_dict(),
  165.                 'random_state': zero.get_random_state(),
  166.                 **{
  167.                     x: globals()[x]
  168.                     for x in [
  169.                         'progress',
  170.                         'stats',
  171.                         'timer',
  172.                         'training_log',
  173.                     ]
  174.                 },
  175.             },
  176.             checkpoint_path,
  177.         )
  178.         lib.dump_stats(stats, output, final)
  179.         lib.backup_output(output)
  180.  
  181.     # %%
  182.     timer.run()
  183.     for epoch in stream.epochs(args['training']['n_epochs']):
  184.         print_epoch_info()
  185.  
  186.         model.train()
  187.         epoch_losses = []
  188.         for batch_idx in epoch:
  189.             loss, new_chunk_size = lib.train_with_auto_virtual_batch( #一次训练的代码
  190.                 optimizer,
  191.                 loss_fn,
  192.                 lambda x: (apply_model(lib.TRAIN, x), Y_device[lib.TRAIN][x]),
  193.                 batch_idx,
  194.                 chunk_size or batch_size,
  195.             )
  196.             epoch_losses.append(loss.detach())
  197.             if new_chunk_size and new_chunk_size < (chunk_size or batch_size):
  198.                 stats['chunk_size'] = chunk_size = new_chunk_size
  199.                 print('New chunk size:', chunk_size)
  200.         epoch_losses = torch.stack(epoch_losses).tolist()
  201.         training_log[lib.TRAIN].extend(epoch_losses)
  202.         print(f'[{lib.TRAIN}] loss = {round(sum(epoch_losses) / len(epoch_losses), 3)}')
  203.  
  204.         metrics, predictions = evaluate([lib.VAL, lib.TEST])
  205.         for k, v in metrics.items():
  206.             training_log[k].append(v)
  207.         progress.update(metrics[lib.VAL]['score'])
  208.  
  209.         if progress.success:
  210.             print('New best epoch!')
  211.             stats['best_epoch'] = stream.epoch
  212.             stats['metrics'] = metrics
  213.             save_checkpoint(False)
  214.             for k, v in predictions.items():
  215.                 np.save(output / f'p_{k}.npy', v)
  216.  
  217.         elif progress.fail:
  218.             break
  219.  
  220.     # %%
  221.     print('\nRunning the final evaluation...')
  222.     model.load_state_dict(torch.load(checkpoint_path)['model'])
  223.     stats['metrics'], predictions = evaluate(lib.PARTS)
  224.     for k, v in predictions.items():
  225.         np.save(output / f'p_{k}.npy', v)
  226.     stats['time'] = lib.format_seconds(timer())
  227.     save_checkpoint(True)
  228.     print('Done!')

首先使用定义在util.py中的load_config函数将命令行中传入的toml文件读入进来，然后生成一个stats.json文件，之后使用data.py中的方法构造并预处理X和Y。之后除了定义在每个py文件中的模型架构外，还会有用一些共有方法对模型训练进行汇总：

对于优化器，会使用一个make_optimizer方法来构建，其中对于特定的参数会加入权重衰减从策略weight_decay。

作者使用了libzero包中的Stream方法来维护每个epoch与batch_size的循环。这个Stream的作用就是能够随时存储与回复循环中的状态并自定义epoch。同时使用这个包下的ProgressTracker来设定模型的early stop。当16个epoch后验证集上依然没有改进时，进入progress.fail分支提前结束训练。

对于训练部分，作者构造了train_with_auto_virtual_batch函数，此处的chunk_size当一个batch的size对内存而言过大时，会更新成原本的1/2

  1. def train_with_auto_virtual_batch(
  2.     optimizer,
  3.     loss_fn,
  4.     step,
  5.     batch,
  6.     chunk_size: int,
  7. ) -> ty.Tuple[Tensor, int]:
  8.     batch_size = len(batch)
  9.     random_state = zero.get_random_state()
  10.     while chunk_size != 0:
  11.         try:
  12.             zero.set_random_state(random_state)
  13.             optimizer.zero_grad()
  14.             if batch_size <= chunk_size:
  15.                 loss = loss_fn(*step(batch))
  16.                 loss.backward()
  17.             else:
  18.                 loss = None
  19.                 for chunk in zero.iter_batches(batch, chunk_size):
  20.                     chunk_loss = loss_fn(*step(chunk))
  21.                     chunk_loss = chunk_loss * (len(chunk) / batch_size)
  22.                     chunk_loss.backward()
  23.                     if loss is None:
  24.                         loss = chunk_loss.detach()
  25.                     else:
  26.                         loss += chunk_loss.detach()
  27.         except RuntimeError as err:
  28.             if not is_oom_exception(err):
  29.                 raise
  30.             chunk_size //= 2
  31.         else:
  32.             break
  33.     if not chunk_size:
  34.         raise RuntimeError('Not enough memory even for batch_size=1')
  35.     optimizer.step()
  36.     return loss, chunk_size  # type: ignore[code]

zero.get/set_random_state()可以全局地给numpy/torch/random赋予相同的随机数种子。注意这个方法在0.0.8版本中已经没有了，如果需要和代码一样调试的话记得按照requirements.txt中的版本：

pip install libzero==0.0.3.dev7

zero.iter_batches的作用上和把数据放到DataLoader里面一样，但是zero这个包里说zero.iter_batches函数更好，因为它是基于batch的索引而非DataLoader那样的基于项的索引。

而对于CatBoost和XGBoost而言，由于fit函数已经有了，所以直接将toml的各个参数传入就可以了，唯一需要注意的就是XGBoost不会自动保存验证集上效果最好的模型，需要我们传入early_stop参数去控制。

四、部分实验内容复现

由于笔者本身的硬件资源有限，只针对Adult数据集选取了FT-Transformer，ResNet，LightGBM，XGBoost并使用作者给出的超参数进行实验。其中FT-Transformer同时在自己的电脑上进行了调优以作对比组。除此以外，还加入了LassoNet进行了调优以及作为对照组。实验的详细代码见第五部分。

各个模型结果：

modelaccuracyrecall
FT_Transformer0.8590340480.646160513
FT_Transformer自己调优的0.8601764840.609707055
lassoNet0.8566263330.593898423
lightGBM0.868099830.645103137
ResNet0.8534119530.639781591
xgboost0.872313330.640006934

 从结果中可以看出，就准确率而言集成树模型（XGBoost和LightGBM）依旧要优于各个深度学习模型；而在各个深度学习模型中，作者给出的FT-Transformer成绩较优。此外，由于Adult数据集本身0-1的比例约为77:23；所以我还记录了各个模型对于正例（1）的召回率。从结果中可以看出，召回率依旧是集成树模型更高。尽管其中的FT-Transformer也已经和树模型的结果相近，但是由于准确率的差距更大，训练的时间更长耗能更多，故而不能说它比树模型的效果更好。

五、复现代码

尽管论文作者已经给出了实验代码，然而笔者的Linux系统似乎有些问题，Windows下的虚拟环境搭建也由于网络问题需要我花一段时间解决，故而在原来的代码上进行了稍微的修改。

FT-Transformer的超参数调优

  1. #模型定义部分与上文一样，此处省略
  2.  
  3. #读取设定
  4. import pytomlpp as toml
  5.  
  6. ArrayDict = ty.Dict[str, np.ndarray]
  7. def normalize(
  8.     X, normalization,seed,noise=1e-3
  9. ):
  10.     X_train = X['train'].copy()
  11.     if normalization == 'standard':
  12.         normalizer = sklearn.preprocessing.StandardScaler()
  13.     elif normalization == 'quantile':
  14.         normalizer = sklearn.preprocessing.QuantileTransformer(
  15.             output_distribution='normal',
  16.             n_quantiles=max(min(X['train'].shape[0] // 30, 1000), 10),
  17.             subsample=int(1e9),
  18.             random_state=seed,
  19.         )
  20.         if noise:
  21.             stds = np.std(X_train, axis=0, keepdims=True)
  22.             noise_std = noise / np.maximum(stds, noise)  # type: ignore[code]
  23.             X_train += noise_std * np.random.default_rng(seed).standard_normal(  # type: ignore[code]
  24.                 X_train.shape
  25.             )
  26.     else:
  27.         raise ValueError('Unknow normalization')
  28.     normalizer.fit(X_train)
  29.     return {k: normalizer.transform(v) for k, v in X.items()}  # type: ignore[code]
  30.  
  31. class CustomDataset(Dataset):
  32.     
  33.     def __init__(self,dir_,data_part,normalization,num_nan_policy,cat_nan_policy,
  34.                 cat_policy,seed,
  35.                 y_poicy=None,cat_min_frequency=0
  36.                 ):
  37.         super(CustomDataset,self).__init__()
  38.         dir_ = Path(dir_)
  39.         def load(item) -> ArrayDict:
  40.             return {
  41.                 x: ty.cast(np.ndarray, np.load(dir_ / f'{item}_{x}.npy'))  # type: ignore[code]
  42.                 for x in ['train', 'val', 'test']
  43.             }
  44.         self.N = load('N') if dir_.joinpath('N_train.npy').exists() else None
  45.         self.C = load('C') if dir_.joinpath('C_train.npy').exists() else None
  46.         self.y = load('y')
  47.         self.info = json.loads((dir_ / 'info.json').read_text())
  48.         
  49.         #pre-process
  50.         cache_path = f"build_dataset_{normalization}__{num_nan_policy}__{cat_nan_policy}__{cat_policy}__{seed}.pickle"
  51.         
  52.         if cat_min_frequency>0:
  53.             cache_path = cache_path.replace('.pickle', f'__{cat_min_frequency}.pickle')
  54.         
  55.         cache_path = Path(cache_path)
  56.         
  57.         if cache_path.exists():
  58.             print("Using cache")
  59.             with open(cache_path, 'rb') as f:
  60.                 data = pickle.load(f)
  61.             self.x = data
  62.             
  63.         else:
  64.         
  65.             def save_result(x):
  66.                 if cache_path:
  67.                     with open(cache_path, 'wb') as f:
  68.                         pickle.dump(x, f)
  69.             if self.N:
  70.                 N = deepcopy(self.N)
  71.                 num_nan_masks = {k: np.isnan(v) for k, v in N.items()}
  72.                 if any(x.any() for x in num_nan_masks.values()):  # type: ignore[code]
  73.                     if num_nan_policy == 'mean':
  74.                         num_new_values = np.nanmean(self.N['train'], axis=0)
  75.                     else:
  76.                         raise ValueError('Unknown numerical NaN policy')
  77.                     for k, v in N.items():
  78.                         num_nan_indices = np.where(num_nan_masks[k])
  79.                         v[num_nan_indices] = np.take(num_new_values, num_nan_indices[1])
  80.                 if normalization:
  81.                     N = normalize(N, normalization, seed)
  82.             else:
  83.                 N = None
  84.  
  85.             C = deepcopy(self.C)
  86.  
  87.             cat_nan_masks = {k: v == 'nan' for k, v in C.items()}
  88.             if any(x.any() for x in cat_nan_masks.values()):  # type: ignore[code]
  89.                 if cat_nan_policy == 'new':
  90.                     cat_new_value = '___null___'
  91.                     imputer = None
  92.                 elif cat_nan_policy == 'most_frequent':
  93.                     cat_new_value = None
  94.                     imputer = SimpleImputer(strategy=cat_nan_policy)  # type: ignore[code]
  95.                     imputer.fit(C['train'])
  96.                 else:
  97.                     raise ValueError('Unknown categorical NaN policy')
  98.                 if imputer:
  99.                     C = {k: imputer.transform(v) for k, v in C.items()}
  100.                 else:
  101.                     for k, v in C.items():
  102.                         cat_nan_indices = np.where(cat_nan_masks[k])
  103.                         v[cat_nan_indices] = cat_new_value
  104.  
  105.             if cat_min_frequency:
  106.                 C = ty.cast(ArrayDict, C)
  107.                 min_count = round(len(C['train']) * cat_min_frequency)
  108.                 rare_value = '___rare___'
  109.                 C_new = {x: [] for x in C}
  110.                 for column_idx in range(C['train'].shape[1]):
  111.                     counter = Counter(C['train'][:, column_idx].tolist())
  112.                     popular_categories = {k for k, v in counter.items() if v >= min_count}
  113.                     for part in C_new:
  114.                         C_new[part].append(
  115.                             [
  116.                                 (x if x in popular_categories else rare_value)
  117.                                 for x in C[part][:, column_idx].tolist()
  118.                             ]
  119.                         )
  120.                 C = {k: np.array(v).T for k, v in C_new.items()}
  121.  
  122.             unknown_value = np.iinfo('int64').max - 3
  123.             encoder = sklearn.preprocessing.OrdinalEncoder(
  124.                 handle_unknown='use_encoded_value',  # type: ignore[code]
  125.                 unknown_value=unknown_value,  # type: ignore[code]
  126.                 dtype='int64',  # type: ignore[code]
  127.             ).fit(C['train'])
  128.             C = {k: encoder.transform(v) for k, v in C.items()}
  129.             max_values = C['train'].max(axis=0)
  130.             for part in ['val', 'test']:
  131.                 for column_idx in range(C[part].shape[1]):
  132.                     C[part][C[part][:, column_idx] == unknown_value, column_idx] = (
  133.                         max_values[column_idx] + 1
  134.                     )
  135.  
  136.             if cat_policy == 'indices':
  137.                 result = (N, C)
  138.             elif cat_policy == 'ohe':
  139.                 ohe = sklearn.preprocessing.OneHotEncoder(
  140.                     handle_unknown='ignore', sparse=False, dtype='float32'  # type: ignore[code]
  141.                 )
  142.                 ohe.fit(C['train'])
  143.                 C = {k: ohe.transform(v) for k, v in C.items()}
  144.                 result = C if N is None else {x: np.hstack((N[x], C[x])) for x in N}
  145.             elif cat_policy == 'counter':
  146.                 assert seed is not None
  147.                 loo = LeaveOneOutEncoder(sigma=0.1, random_state=seed, return_df=False)
  148.                 loo.fit(C['train'], self.y['train'])
  149.                 C = {k: loo.transform(v).astype('float32') for k, v in C.items()}  # type: ignore[code]
  150.                 if not isinstance(C['train'], np.ndarray):
  151.                     C = {k: v.values for k, v in C.items()}  # type: ignore[code]
  152.                 if normalization:
  153.                     C = normalize(C, normalization, seed, inplace=True)  # type: ignore[code]
  154.                 result = C if N is None else {x: np.hstack((N[x], C[x])) for x in N}
  155.             else:
  156.                 raise ValueError('Unknow categorical policy')
  157.  
  158.             save_result(result)
  159.             self.x = result
  160.         self.X_num,self.X_cat = self.x
  161.         self.X_num = None if self.X_num is None else self.X_num[data_part]
  162.         self.X_cat = None if self.X_cat is None else self.X_cat[data_part]
  163.         
  164.         # build Y
  165.         
  166.         if self.info['task_type'] == 'regression':
  167.             assert policy == 'mean_std'
  168.         y = deepcopy(self.y)
  169.         if y_poicy:
  170.             if not self.info['task_type'] == 'regression':
  171.                 warnings.warn('y_policy is not None, but the task is NOT regression')
  172.                 info = None
  173.             elif y_poicy == 'mean_std':
  174.                 mean, std = self.y['train'].mean(), self.y['train'].std()
  175.                 y = {k: (v - mean) / std for k, v in y.items()}
  176.                 info = {'policy': policy, 'mean': mean, 'std': std}
  177.             else:
  178.                 raise ValueError('Unknow y policy')
  179.         else:
  180.             info = None
  181.         
  182.         self.y = y[data_part]
  183.         if len(self.y.shape)==1:
  184.             self.y = self.y.reshape((self.y.shape[0],1))
  185.         self.y_info = info
  186.         
  187.     def __len__(self):
  188.         X = self.X_num if self.X_num is not None else self.X_cat
  189.         return len(X)
  190.     
  191.     def __getitem__(self,idx):
  192.         return torch.FloatTensor(self.X_num[idx]).to(device),torch.IntTensor(self.X_cat[idx]).to(device),torch.FloatTensor(self.y[idx]).to(device)
  193.  
  194. data_path_father = "D:/rtdl_data.tar/rtdl_data/data/"
  195. configs = toml.load("D:/rtdl-revisiting-models-main/output/adult/ft_transformer/tuning/0.toml")
  196. data_configs = configs["base_config"]["data"]
  197. configs["base_config"]["model"].setdefault('token_bias', True)
  198. configs["base_config"]["model"].setdefault('kv_compression', None)
  199. configs["base_config"]["model"].setdefault('kv_compression_sharing', None)
  200.  
  201. D_train = CustomDataset(
  202.     data_path_father+"adult"
  203.     ,data_part="train"
  204.     ,normalization=data_configs["normalization"]
  205.     ,num_nan_policy="mean"
  206.     ,cat_nan_policy="new"
  207.     ,cat_policy=data_configs.get("cat_policy", 'indices')
  208.     ,seed=configs["base_config"]["seed"]
  209.     ,y_poicy=data_configs.get("y_policy"),cat_min_frequency=0
  210. )
  211.  
  212. D_valid = CustomDataset(
  213.     data_path_father+"adult"
  214.     ,data_part="val"
  215.     ,normalization=data_configs["normalization"]
  216.     ,num_nan_policy="mean"
  217.     ,cat_nan_policy="new"
  218.     ,cat_policy=data_configs.get("cat_policy", 'indices')
  219.     ,seed=configs["base_config"]["seed"]
  220.     ,y_poicy=data_configs.get("y_policy"),cat_min_frequency=0
  221. )
  222.  
  223. D_test = CustomDataset(
  224.     data_path_father+"adult"
  225.     ,data_part="test"
  226.     ,normalization=data_configs["normalization"]
  227.     ,num_nan_policy="mean"
  228.     ,cat_nan_policy="new"
  229.     ,cat_policy=data_configs.get("cat_policy", 'indices')
  230.     ,seed=configs["base_config"]["seed"]
  231.     ,y_poicy=data_configs.get("y_policy"),cat_min_frequency=0
  232. )
  233.  
  234. dl_train = DataLoader(D_train,batch_size=configs["base_config"]["training"]["batch_size"])
  235. dl_val = DataLoader(D_valid,batch_size=len(D_valid))
  236. dl_test = DataLoader(D_test,batch_size=len(D_test))
  237.  
  238. def make_optimizer(
  239.     optimizer: str,
  240.     parameter_groups,
  241.     lr: float,
  242.     weight_decay: float,
  243. ) -> optim.Optimizer:
  244.     Optimizer = {
  245.         'adam': optim.Adam,
  246.         'adamw': optim.AdamW,
  247.         'sgd': optim.SGD,
  248.     }[optimizer]
  249.     momentum = (0.9,) if Optimizer is optim.SGD else ()
  250.     return Optimizer(parameter_groups, lr, *momentum, weight_decay=weight_decay)
  251.  
  252. def needs_wd(name):
  253.     return all(x not in name for x in ['tokenizer', '.norm', '.bias'])
  254.  
  255. import optuna
  256.  
  257. def sample_parameters(trial,space,base_config):
  258.     def get_distribution(distribution_name):
  259.         return getattr(trial, f'suggest_{distribution_name}')
  260.     result = {}
  261.     for label, subspace in space.items():
  262.         if isinstance(subspace, dict):
  263.             result[label] = sample_parameters(trial, subspace, base_config)
  264.         else:
  265.             assert isinstance(subspace, list)
  266.             distribution, *args = subspace
  267.  
  268.             if distribution.startswith('?'): #此处我个人的理解是:这个参数在原本的调试范围基础上还要增加一个"optional_"作取舍，可以理解为：先取舍是否使用默认值，然后在后面给定范围内作调优看看哪个更好。
  269.                 default_value = args[0]
  270.                 result[label] = (
  271.                     get_distribution(distribution.lstrip('?'))(label, *args[1:])
  272.                     if trial.suggest_categorical(f'optional_{label}', [False, True])
  273.                     else default_value
  274.                 )
  275.  
  276.             elif distribution == '$mlp_d_layers': #格式特殊
  277.                 min_n_layers, max_n_layers, d_min, d_max = args
  278.                 n_layers = trial.suggest_int('n_layers', min_n_layers, max_n_layers)
  279.                 suggest_dim = lambda name: trial.suggest_int(name, d_min, d_max)  # noqa
  280.                 d_first = [suggest_dim('d_first')] if n_layers else []
  281.                 d_middle = (
  282.                     [suggest_dim('d_middle')] * (n_layers - 2) if n_layers > 2 else []
  283.                 )
  284.                 d_last = [suggest_dim('d_last')] if n_layers > 1 else []
  285.                 result[label] = d_first + d_middle + d_last
  286.  
  287.             elif distribution == '$d_token':#多了一个检测的步骤
  288.                 assert len(args) == 2
  289.                 try:
  290.                     n_heads = base_config['model']['n_heads']
  291.                 except KeyError:
  292.                     n_heads = base_config['model']['n_latent_heads']
  293.  
  294.                 for x in args:
  295.                     assert x % n_heads == 0
  296.                 result[label] = trial.suggest_int('d_token', *args, n_heads)# n_heads是步长，确保d_token能够被n_heads整除  # type: ignore[code]
  297.  
  298.             elif distribution in ['$d_ffn_factor', '$d_hidden_factor']: #对于glu系激活函数这2个参数特殊处理特殊处理
  299.                 if base_config['model']['activation'].endswith('glu'):
  300.                     args = (args[0] * 2 / 3, args[1] * 2 / 3)
  301.                 result[label] = trial.suggest_uniform('d_ffn_factor', *args)
  302.  
  303.             else:
  304.                 result[label] = get_distribution(distribution)(label, *args)
  305.     return result
  306.             
  307. def merge_sampled_parameters(config, sampled_parameters):
  308.     for k, v in sampled_parameters.items():
  309.         if isinstance(v, dict):
  310.             merge_sampled_parameters(config.setdefault(k, {}), v)
  311.         else:
  312.             assert k not in config
  313.             config[k] = v
  314.  
  315. def objective(trial):
  316.     config = deepcopy(configs['base_config'])
  317.     merge_sampled_parameters(
  318.         config, sample_parameters(trial, configs['optimization']['space'], config)
  319.     )
  320.     model = Transformer(
  321.     d_num=0 if D_train.X_num is None else D_train.X_num.shape[1],
  322.     categories = None if D_train.X_cat is None else [len(set(D_train.X_cat[:, i].tolist())) for i in range(D_train.X_cat.shape[1])],
  323.     d_out=D_train.info['n_classes'] if D_train.info["task_type"]=="multiclass" else 1
  324.     ,**config['model']
  325.     ).to(device)
  326.     
  327.     parameters_with_wd = [v for k, v in model.named_parameters() if needs_wd(k)]
  328.     parameters_without_wd = [v for k, v in model.named_parameters() if not needs_wd(k)]
  329.     
  330.     loss_fn = (
  331.         F.binary_cross_entropy_with_logits
  332.         if D_train.info["task_type"]=="binclass"
  333.         else F.cross_entropy
  334.         if D_train.info["task_type"]=="multiclass"
  335.         else F.mse_loss
  336.     )
  337.  
  338.     optimizer = make_optimizer(
  339.         config["training"]["optimizer"],
  340.         (
  341.             [
  342.                 {'params': parameters_with_wd},
  343.                 {'params': parameters_without_wd, 'weight_decay': 0.0},
  344.             ]
  345.         ),
  346.         config["training"]["lr"],#to be trained in optuna
  347.         config["training"]["weight_decay"]#to be trained in optuna
  348.     )
  349.     
  350.     loss_best = np.nan
  351.     best_epoch = -1
  352.     patience = 0
  353.     
  354.     def save_state():
  355.         torch.save(
  356.             model.state_dict(),os.path.join(os.getcwd(),"ft_transformer_state.pickle")
  357.         )
  358.         with open(os.path.join(os.getcwd(),"best_state_ft_transformer.json"),"w") as f:
  359.             json.dump(config,f)
  360.     
  361.     #dl_train.batch_size=config['training']['batch_size']
  362.     for epoch in range(config['training']['n_epochs']):
  363.         model.train()
  364.         for i,(x_num,x_cat,y) in enumerate(dl_train):
  365.             optimizer.zero_grad()
  366.             y_batch = model(x_num,x_cat)
  367.             loss = loss_fn(y_batch.reshape((y_batch.shape[0],1)),y)
  368.             loss.backward()
  369.             optimizer.step()
  370.             
  371.         model.eval()
  372.         with torch.no_grad():
  373.             for i,(x_num,x_cat,y) in enumerate(dl_val): #只有1个迭代
  374.                 y_batch = model(x_num,x_cat)
  375.                 loss = loss_fn(y_batch.reshape((y_batch.shape[0],1)),y)
  376.                 new_loss = loss.detach()
  377.                 if np.isnan(loss_best) or new_loss.cpu().numpy() < loss_best:
  378.                     patience = 0
  379.                     best_epoch = epoch
  380.                     loss_best = new_loss.cpu().numpy()
  381.                     save_state()
  382.                 else:
  383.                     patience+=1
  384.         if patience>= config['training']['patience']:
  385.             break
  386.     return loss_best
  387.  
  388. study = optuna.create_study(
  389.         direction="minimize",
  390.         sampler=optuna.samplers.TPESampler(**configs['optimization']['sampler']),
  391.     )
  392.  
  393. study.optimize(
  394.     objective,
  395.     **configs['optimization']['options'],
  396.     #callbacks=[save_checkpoint],
  397.     show_progress_bar=True,
  398. )

LassoNet的模型定义与超参数调优

首先，定义toml文件：

  1. [base_config]
  2. seed = 0
  3.  
  4.     [base_config.data]
  5.     normalization = 'quantile'
  6.     path = 'data/adult'
  7.     cat_policy = 'indices'
  8.  
  9.     [base_config.model]
  10.     [base_config.training]
  11.     batch_size = 256
  12.     eval_batch_size = 8192
  13.     n_epochs = 1000000000
  14.     optimizer = 'adamw'
  15.     patience = 16
  16.  
  17. [optimization.options]
  18. n_trials = 100
  19.  
  20. [optimization.sampler]
  21. seed = 0
  22.  
  23. [optimization.space.model]
  24. dims = [ '$mlp_d_layers', 1, 8, 1, 512 ]
  25. d_embedding = ['int', 64, 512]
  26. dropout = [ '?uniform', 0.0, 0.0, 0.5 ]
  27. gamma = [ '?loguniform', 0, 1e-08, 100.0 ]
  28. lambda_ = [ '?loguniform', 0, 1e-08, 100.0 ]
  29. M = [ 'int', 10, 50 ]
  30. gamma_skip = [ '?loguniform', 0, 1e-08, 100.0 ]
  31.  
  32. [optimization.space.training]
  33. lr = [ 'loguniform', 1e-05, 0.01 ]
  34. weight_decay = [ '?loguniform', 0.0, 1e-06, 0.001 ]
  1. from itertools import islice
  2.  
  3. def soft_threshold(l, x):
  4.     return torch.sign(x) * torch.relu(torch.abs(x) - l)
  5.  
  6.  
  7. def sign_binary(x):
  8.     ones = torch.ones_like(x)
  9.     return torch.where(x >= 0, ones, -ones)
  10.  
  11.  
  12. def prox(v, u, *, lambda_, lambda_bar, M):
  13.     """
  14.     v has shape (m,) or (m, batches)
  15.     u has shape (k,) or (k, batches)
  17.     supports GPU tensors
  18.     """
  19.     onedim = len(v.shape) == 1
  20.     if onedim:
  21.         v = v.unsqueeze(-1)
  22.         u = u.unsqueeze(-1)
  23.  
  24.     u_abs_sorted = torch.sort(u.abs(), dim=0, descending=True).values
  25.  
  26.     k, batch = u.shape
  27.  
  28.     s = torch.arange(k + 1.0).view(-1, 1).to(v)
  29.     zeros = torch.zeros(1, batch).to(u)
  30.  
  31.     a_s = lambda_ - M * torch.cat(
  32.         [zeros, torch.cumsum(u_abs_sorted - lambda_bar, dim=0)]
  33.     )
  34.  
  35.     norm_v = torch.norm(v, p=2, dim=0)
  36.  
  37.     x = F.relu(1 - a_s / norm_v) / (1 + s * M ** 2)
  38.  
  39.     w = M * x * norm_v
  40.     intervals = soft_threshold(lambda_bar, u_abs_sorted)
  41.     lower = torch.cat([intervals, zeros])
  42.  
  43.     idx = torch.sum(lower > w, dim=0).unsqueeze(0)
  44.  
  45.     x_star = torch.gather(x, 0, idx).view(1, batch)
  46.     w_star = torch.gather(w, 0, idx).view(1, batch)
  47.  
  48.     beta_star = x_star * v
  49.     theta_star = sign_binary(u) * torch.min(soft_threshold(lambda_bar, u.abs()), w_star)
  50.  
  51.     if onedim:
  52.         beta_star.squeeze_(-1)
  53.         theta_star.squeeze_(-1)
  54.  
  55.     return beta_star, theta_star
  56.  
  57.  
  58. def inplace_prox(beta, theta, lambda_, lambda_bar, M):
  59.     beta.weight.data, theta.weight.data = prox(
  60.         beta.weight.data, theta.weight.data, lambda_=lambda_, lambda_bar=lambda_bar, M=M
  61.     )
  62.  
  63.  
  64. def inplace_group_prox(groups, beta, theta, lambda_, lambda_bar, M):
  65.     """
  66.     groups is an iterable such that group[i] contains the indices of features in group i
  67.     """
  68.     beta_ = beta.weight.data
  69.     theta_ = theta.weight.data
  70.     beta_ans = torch.empty_like(beta_)
  71.     theta_ans = torch.empty_like(theta_)
  72.     for g in groups:
  73.         group_beta = beta_[:, g]
  74.         group_beta_shape = group_beta.shape
  75.         group_theta = theta_[:, g]
  76.         group_theta_shape = group_theta.shape
  77.         group_beta, group_theta = prox(
  78.             group_beta.reshape(-1),
  79.             group_theta.reshape(-1),
  80.             lambda_=lambda_,
  81.             lambda_bar=lambda_bar,
  82.             M=M,
  83.         )
  84.         beta_ans[:, g] = group_beta.reshape(*group_beta_shape)
  85.         theta_ans[:, g] = group_theta.reshape(*group_theta_shape)
  86.     beta.weight.data, theta.weight.data = beta_ans, theta_ans
  87.  
  88. class LassoNet(nn.Module):
  89.     def __init__(self,d_numerical,categories,d_out,d_embedding, dims,gamma,gamma_skip,lambda_,M, groups=None, dropout=None):
  90.         """
  91.         first dimension is input
  92.         last dimension is output
  93.         `groups` is a list of list such that `groups[i]`
  94.         contains the indices of the features in the i-th group
  96.         """
  97.         #assert len(dims) > 2
  98.         if groups is not None:
  99.             n_inputs = dims[0]
  100.             all_indices = []
  101.             for g in groups:
  102.                 for i in g:
  103.                     all_indices.append(i)
  104.             assert len(all_indices) == n_inputs and set(all_indices) == set(
  105.                 range(n_inputs)
  106.             ), f"Groups must be a partition of range(n_inputs={n_inputs})"
  107.  
  108.         self.groups = groups
  109.  
  110.         super().__init__()
  111.         #加入numerical和categories的输入处理
  112.         d_in = d_numerical
  113.  
  114.         if categories is not None:
  115.             d_in += len(categories) * d_embedding
  116.             category_offsets = torch.tensor([0] + categories[:-1]).cumsum(0)
  117.             self.register_buffer('category_offsets', category_offsets)
  118.             self.category_embeddings = nn.Embedding(sum(categories), d_embedding)
  119.             nn.init.kaiming_uniform_(self.category_embeddings.weight, a=math.sqrt(5))
  120.             print(f'{self.category_embeddings.weight.shape=}')
  121.             
  122.         dims = [d_in]+dims+[d_out]
  123.         self.gamma = gamma
  124.         self.gamma_skip = gamma_skip
  125.         self.lambda_ = lambda_
  126.         self.M = M
  127.         # 新增部分结束
  128.         self.dropout = nn.Dropout(p=dropout) if dropout is not None else None
  129.         self.layers = nn.ModuleList(
  130.             [nn.Linear(dims[i], dims[i + 1]) for i in range(len(dims) - 1)]
  131.         )
  132.         self.skip = nn.Linear(dims[0], dims[-1], bias=False)
  133.  
  134.     def forward(self, x_num,x_cat):
  135.         inp = []
  136.         if x_num is not None:
  137.             inp.append(x_num)
  138.         if x_cat is not None:
  139.             inp.append(
  140.                 self.category_embeddings(x_cat + self.category_offsets[None]).view(
  141.                     x_cat.size(0), -1
  142.                 )
  143.             )
  144.         inp = torch.cat(inp, dim=-1)
  145.         current_layer = inp
  146.         result = self.skip(inp)
  147.         for theta in self.layers:
  148.             current_layer = theta(current_layer)
  149.             if theta is not self.layers[-1]:
  150.                 if self.dropout is not None:
  151.                     current_layer = self.dropout(current_layer)
  152.                 current_layer = F.relu(current_layer)
  153.         return result + current_layer
  154.  
  155.     def prox(self, *, lambda_, lambda_bar=0, M=1):
  156.         if self.groups is None:
  157.             with torch.no_grad():
  158.                 inplace_prox(
  159.                     beta=self.skip,
  160.                     theta=self.layers[0],
  161.                     lambda_=lambda_,
  162.                     lambda_bar=lambda_bar,
  163.                     M=M,
  164.                 )
  165.         else:
  166.             with torch.no_grad():
  167.                 inplace_group_prox(
  168.                     groups=self.groups,
  169.                     beta=self.skip,
  170.                     theta=self.layers[0],
  171.                     lambda_=lambda_,
  172.                     lambda_bar=lambda_bar,
  173.                     M=M,
  174.                 )
  175.  
  176.     def lambda_start(
  177.         self,
  178.         M=1,
  179.         lambda_bar=0,
  180.         factor=2,
  181.     ):
  182.         """Estimate when the model will start to sparsify."""
  183.  
  184.         def is_sparse(lambda_):
  185.             with torch.no_grad():
  186.                 beta = self.skip.weight.data
  187.                 theta = self.layers[0].weight.data
  188.  
  189.                 for _ in range(10000):
  190.                     new_beta, theta = prox(
  191.                         beta,
  192.                         theta,
  193.                         lambda_=lambda_,
  194.                         lambda_bar=lambda_bar,
  195.                         M=M,
  196.                     )
  197.                     if torch.abs(beta - new_beta).max() < 1e-5:
  198.                         break
  199.                     beta = new_beta
  200.                 return (torch.norm(beta, p=2, dim=0) == 0).sum()
  201.  
  202.         start = 1e-6
  203.         while not is_sparse(factor * start):
  204.             start *= factor
  205.         return start
  206.  
  207.     def l2_regularization(self):
  208.         """
  209.         L2 regulatization of the MLP without the first layer
  210.         which is bounded by the skip connection
  211.         """
  212.         ans = 0
  213.         for layer in islice(self.layers, 1, None):
  214.             ans += (
  215.                 torch.norm(
  216.                     layer.weight.data,
  217.                     p=2,
  218.                 )
  219.                 ** 2
  220.             )
  221.         return ans
  222.  
  223.     def l1_regularization_skip(self):
  224.         return torch.norm(self.skip.weight.data, p=2, dim=0).sum()
  225.  
  226.     def l2_regularization_skip(self):
  227.         return torch.norm(self.skip.weight.data, p=2)
  228.  
  229.     def input_mask(self):
  230.         with torch.no_grad():
  231.             return torch.norm(self.skip.weight.data, p=2, dim=0) != 0
  232.  
  233.     def selected_count(self):
  234.         return self.input_mask().sum().item()
  235.  
  236.     def cpu_state_dict(self):
  237.         return {k: v.detach().clone().cpu() for k, v in self.state_dict().items()}
  238.  
  239. configs = toml.load("D:/rtdl-revisiting-models-main/output/adult/lassoNet/tunning/0.toml")
  240. def objective(trial):
  241.     config = deepcopy(configs['base_config'])
  242.     merge_sampled_parameters(
  243.         config, sample_parameters(trial, configs['optimization']['space'], config)
  244.     )
  245.     model = LassoNet(
  246.     d_numerical=0 if D_train.X_num is None else D_train.X_num.shape[1],
  247.     categories = None if D_train.X_cat is None else [len(set(D_train.X_cat[:, i].tolist())) for i in range(D_train.X_cat.shape[1])],
  248.     d_out=D_train.info['n_classes'] if D_train.info["task_type"]=="multiclass" else 1
  249.     ,**config['model']
  250.     ).to(device)
  251.     
  252.     parameters_with_wd = [v for k, v in model.named_parameters() if needs_wd(k)]
  253.     parameters_without_wd = [v for k, v in model.named_parameters() if not needs_wd(k)]
  254.     
  255.     loss_fn = (
  256.         F.binary_cross_entropy_with_logits
  257.         if D_train.info["task_type"]=="binclass"
  258.         else F.cross_entropy
  259.         if D_train.info["task_type"]=="multiclass"
  260.         else F.mse_loss
  261.     )
  262.  
  263.     optimizer = make_optimizer(
  264.         config["training"]["optimizer"],
  265.         (
  266.             [
  267.                 {'params': parameters_with_wd},
  268.                 {'params': parameters_without_wd, 'weight_decay': 0.0},
  269.             ]
  270.         ),
  271.         config["training"]["lr"],#to be trained in optuna
  272.         config["training"]["weight_decay"]#to be trained in optuna
  273.     )
  274.     
  275.     loss_best = np.nan
  276.     best_epoch = -1
  277.     patience = 0
  278.     
  279.     def save_state():
  280.         torch.save(
  281.             model.state_dict(),os.path.join(os.getcwd(),"lassonet_state.pickle")
  282.         )
  283.         with open(os.path.join(os.getcwd(),"best_state_lassonet.json"),"w") as f:
  284.             json.dump(config,f)
  285.     
  286.     #dl_train.batch_size=config['training']['batch_size']
  287.     for epoch in range(config['training']['n_epochs']):
  288.         model.train()
  289.         for i,(x_num,x_cat,y) in enumerate(dl_train):
  290.             optimizer.zero_grad()
  291. #             y_batch = model(x_num,x_cat)
  292.             loss = 0
  293.             def closure():
  294.                 nonlocal loss
  295.                 optimizer.zero_grad()
  296.                 ans = (
  297.                     loss_fn(model(x_num,x_cat), y)
  298.                     + model.gamma * model.l2_regularization()
  299.                     + model.gamma_skip * model.l2_regularization_skip()
  300.                 )
  301.                 ans.backward() #相当于第7行Compute gradient of the loss
  302.                 loss += ans.item()# * len(batch) / n_train
  303.                 return ans
  304.             optimizer.step(closure)
  305.             model.prox(lambda_=model.lambda_ * optimizer.param_groups[0]["lr"], M=model.M) #Hier-Prox算法
  306.             
  307.         model.eval()
  308.         with torch.no_grad():
  309.             for i,(x_num,x_cat,y) in enumerate(dl_val): #只有1个迭代
  310.                 y_batch = model(x_num,x_cat)#.reshape((y_batch.shape[0],1))
  311.                 loss = (
  312.                     loss_fn(y_batch.reshape((y_batch.shape[0],1)), y).item()
  313.                     + model.gamma * model.l2_regularization().item()
  314.                     + model.gamma_skip * model.l2_regularization_skip().item()
  315.                     + model.lambda_ * model.l1_regularization_skip().item()
  316.                 )
  317.                 new_loss = loss#.detach()
  318.                 if np.isnan(loss_best) or new_loss < loss_best:
  319.                     patience = 0
  320.                     best_epoch = epoch
  321.                     loss_best = new_loss#.cpu().numpy()
  322.                     save_state()
  323.                 else:
  324.                     patience+=1
  325.         if patience>= config['training']['patience']:
  326.             break
  327.     return loss_best
  328.  
  329. study = optuna.create_study(
  330.         direction="minimize",
  331.         sampler=optuna.samplers.TPESampler(**configs['optimization']['sampler']),
  332.     )
  333.  
  334. study.optimize(
  335.     objective,
  336.     **configs['optimization']['options'],
  337.     #callbacks=[save_checkpoint],
  338.     show_progress_bar=True,
  339. )

对15个随机数种子进行建模并记录结果

  1. ## 模型定义部分同上，不再赘述
  2.  
  3. ArrayDict = ty.Dict[str, np.ndarray]
  4. def normalize(
  5.     X, normalization,seed,noise=1e-3
  6. ):
  7.     X_train = X['train'].copy()
  8.     if normalization == 'standard':
  9.         normalizer = sklearn.preprocessing.StandardScaler()
  10.     elif normalization == 'quantile':
  11.         normalizer = sklearn.preprocessing.QuantileTransformer(
  12.             output_distribution='normal',
  13.             n_quantiles=max(min(X['train'].shape[0] // 30, 1000), 10),
  14.             subsample=int(1e9),
  15.             random_state=seed,
  16.         )
  17.         if noise:
  18.             stds = np.std(X_train, axis=0, keepdims=True)
  19.             noise_std = noise / np.maximum(stds, noise)  # type: ignore[code]
  20.             X_train += noise_std * np.random.default_rng(seed).standard_normal(  # type: ignore[code]
  21.                 X_train.shape
  22.             )
  23.     else:
  24.         raise ValueError('Unknow normalization')
  25.     normalizer.fit(X_train)
  26.     return {k: normalizer.transform(v) for k, v in X.items()}  # type: ignore[code]
  27.  
  28. class CustomDataset(Dataset):
  29.     
  30.     def __init__(self,dir_,data_part,normalization,num_nan_policy,cat_nan_policy,
  31.                 cat_policy,seed,
  32.                 y_poicy=None,cat_min_frequency=0
  33.                 ):
  34.         super(CustomDataset,self).__init__()
  35.         dir_ = Path(dir_)
  36.         def load(item) -> ArrayDict:
  37.             return {
  38.                 x: ty.cast(np.ndarray, np.load(dir_ / f'{item}_{x}.npy'))  # type: ignore[code]
  39.                 for x in ['train', 'val', 'test']
  40.             }
  41.         self.N = load('N') if dir_.joinpath('N_train.npy').exists() else None
  42.         self.C = load('C') if dir_.joinpath('C_train.npy').exists() else None
  43.         self.y = load('y')
  44.         self.info = json.loads((dir_ / 'info.json').read_text())
  45.         
  46.         #pre-process
  47.         cache_path = f"build_dataset_{normalization}__{num_nan_policy}__{cat_nan_policy}__{cat_policy}__{seed}.pickle"
  48.         
  49.         if cat_min_frequency>0:
  50.             cache_path = cache_path.replace('.pickle', f'__{cat_min_frequency}.pickle')
  51.         
  52.         cache_path = Path(cache_path)
  53.         
  54.         if cache_path.exists():
  55.             print("Using cache")
  56.             with open(cache_path, 'rb') as f:
  57.                 data = pickle.load(f)
  58.             self.x = data
  59.             
  60.         else:
  61.         
  62.             def save_result(x):
  63.                 if cache_path:
  64.                     with open(cache_path, 'wb') as f:
  65.                         pickle.dump(x, f)
  66.             if self.N:
  67.                 N = deepcopy(self.N)
  68.                 num_nan_masks = {k: np.isnan(v) for k, v in N.items()}
  69.                 if any(x.any() for x in num_nan_masks.values()):  # type: ignore[code]
  70.                     if num_nan_policy == 'mean':
  71.                         num_new_values = np.nanmean(self.N['train'], axis=0)
  72.                     else:
  73.                         raise ValueError('Unknown numerical NaN policy')
  74.                     for k, v in N.items():
  75.                         num_nan_indices = np.where(num_nan_masks[k])
  76.                         v[num_nan_indices] = np.take(num_new_values, num_nan_indices[1])
  77.                 if normalization:
  78.                     N = normalize(N, normalization, seed)
  79.             else:
  80.                 N = None
  81.  
  82.             C = deepcopy(self.C)
  83.  
  84.             cat_nan_masks = {k: v == 'nan' for k, v in C.items()}
  85.             if any(x.any() for x in cat_nan_masks.values()):  # type: ignore[code]
  86.                 if cat_nan_policy == 'new':
  87.                     cat_new_value = '___null___'
  88.                     imputer = None
  89.                 elif cat_nan_policy == 'most_frequent':
  90.                     cat_new_value = None
  91.                     imputer = SimpleImputer(strategy=cat_nan_policy)  # type: ignore[code]
  92.                     imputer.fit(C['train'])
  93.                 else:
  94.                     raise ValueError('Unknown categorical NaN policy')
  95.                 if imputer:
  96.                     C = {k: imputer.transform(v) for k, v in C.items()}
  97.                 else:
  98.                     for k, v in C.items():
  99.                         cat_nan_indices = np.where(cat_nan_masks[k])
  100.                         v[cat_nan_indices] = cat_new_value
  101.  
  102.             if cat_min_frequency:
  103.                 C = ty.cast(ArrayDict, C)
  104.                 min_count = round(len(C['train']) * cat_min_frequency)
  105.                 rare_value = '___rare___'
  106.                 C_new = {x: [] for x in C}
  107.                 for column_idx in range(C['train'].shape[1]):
  108.                     counter = Counter(C['train'][:, column_idx].tolist())
  109.                     popular_categories = {k for k, v in counter.items() if v >= min_count}
  110.                     for part in C_new:
  111.                         C_new[part].append(
  112.                             [
  113.                                 (x if x in popular_categories else rare_value)
  114.                                 for x in C[part][:, column_idx].tolist()
  115.                             ]
  116.                         )
  117.                 C = {k: np.array(v).T for k, v in C_new.items()}
  118.  
  119.             unknown_value = np.iinfo('int64').max - 3
  120.             encoder = sklearn.preprocessing.OrdinalEncoder(
  121.                 handle_unknown='use_encoded_value',  # type: ignore[code]
  122.                 unknown_value=unknown_value,  # type: ignore[code]
  123.                 dtype='int64',  # type: ignore[code]
  124.             ).fit(C['train'])
  125.             C = {k: encoder.transform(v) for k, v in C.items()}
  126.             max_values = C['train'].max(axis=0)
  127.             for part in ['val', 'test']:
  128.                 for column_idx in range(C[part].shape[1]):
  129.                     C[part][C[part][:, column_idx] == unknown_value, column_idx] = (
  130.                         max_values[column_idx] + 1
  131.                     )
  132.  
  133.             if cat_policy == 'indices':
  134.                 result = (N, C)
  135.             elif cat_policy == 'ohe':
  136.                 ohe = sklearn.preprocessing.OneHotEncoder(
  137.                     handle_unknown='ignore', sparse=False, dtype='float32'  # type: ignore[code]
  138.                 )
  139.                 ohe.fit(C['train'])
  140.                 C = {k: ohe.transform(v) for k, v in C.items()}
  141.                 result = (N, C)
  142.                 #result = C if N is None else {x: np.hstack((N[x], C[x])) for x in N}
  143.             elif cat_policy == 'counter':
  144.                 assert seed is not None
  145.                 loo = LeaveOneOutEncoder(sigma=0.1, random_state=seed, return_df=False)
  146.                 loo.fit(C['train'], self.y['train'])
  147.                 C = {k: loo.transform(v).astype('float32') for k, v in C.items()}  # type: ignore[code]
  148.                 if not isinstance(C['train'], np.ndarray):
  149.                     C = {k: v.values for k, v in C.items()}  # type: ignore[code]
  150.                 if normalization:
  151.                     C = normalize(C, normalization, seed, inplace=True)  # type: ignore[code]
  152.                 result = (N, C)
  153.                 #result = C if N is None else {x: np.hstack((N[x], C[x])) for x in N}
  154.             else:
  155.                 raise ValueError('Unknow categorical policy')
  156.  
  157.             save_result(result)
  158.             self.x = result
  159.         self.X_num,self.X_cat = self.x
  160.         self.X_num = None if self.X_num is None else self.X_num[data_part]
  161.         self.X_cat = None if self.X_cat is None else self.X_cat[data_part]
  162.         
  163.         # build Y
  164.         
  165.         if self.info['task_type'] == 'regression':
  166.             assert policy == 'mean_std'
  167.         y = deepcopy(self.y)
  168.         if y_poicy:
  169.             if not self.info['task_type'] == 'regression':
  170.                 warnings.warn('y_policy is not None, but the task is NOT regression')
  171.                 info = None
  172.             elif y_poicy == 'mean_std':
  173.                 mean, std = self.y['train'].mean(), self.y['train'].std()
  174.                 y = {k: (v - mean) / std for k, v in y.items()}
  175.                 info = {'policy': policy, 'mean': mean, 'std': std}
  176.             else:
  177.                 raise ValueError('Unknow y policy')
  178.         else:
  179.             info = None
  180.         
  181.         self.y = y[data_part]
  182.         if len(self.y.shape)==1:
  183.             self.y = self.y.reshape((self.y.shape[0],1))
  184.         self.y_info = info
  185.         
  186.     def __len__(self):
  187.         X = self.X_num if self.X_num is not None else self.X_cat
  188.         return len(X)
  189.     
  190.     def __getitem__(self,idx):
  191.         return torch.FloatTensor(self.X_num[idx]).to(device),torch.IntTensor(self.X_cat[idx]).to(device),torch.FloatTensor(self.y[idx]).to(device)
  192.  
  193. ##读设定文件
  194. import pytomlpp as toml
  195.  
  196. xgboost_config = toml.load("xgboost.toml")
  197. lightGBM_config = toml.load("lightgbm.toml")
  198. ft_transformer_config = toml.load("FT_TRANSFORMER.toml")
  199. ft_transformer_mine_config = toml.load("FT_TRANSFORMER_MINE.toml")
  200. resNet_config = toml.load("resnet.toml")
  201. LassoNet_config = toml.load("LassoNet.toml")
  202.  
  203. def needs_wd(name):
  204.     return all(x not in name for x in ['tokenizer', '.norm', '.bias'])
  205.  
  206. def make_optimizer(
  207.     optimizer: str,
  208.     parameter_groups,
  209.     lr: float,
  210.     weight_decay: float,
  211. ) -> optim.Optimizer:
  212.     Optimizer = {
  213.         'adam': optim.Adam,
  214.         'adamw': optim.AdamW,
  215.         'sgd': optim.SGD,
  216.     }[optimizer]
  217.     momentum = (0.9,) if Optimizer is optim.SGD else ()
  218.     return Optimizer(parameter_groups, lr, *momentum, weight_decay=weight_decay)
  219.  
  220. def train_model_xgboost(model,fit_kwargs,dataset_train,dataset_valid,dataset_test,seed):
  221.     model_state_dict_path = os.path.join(os.getcwd(),f"xgboost_state_seed_{seed}.pickle")
  222.     model_result_records = "xgboost_Result.txt"
  223.     feature_importance_record_path = f"xgboost_feature_importance_{seed}.npy"
  224.     
  225.     if os.path.exists(model_state_dict_path):
  226.         return
  227.     
  228.     X_train = dataset_train.X_cat if dataset_train.X_num is None else np.hstack((dataset_train.X_num, dataset_train.X_cat))
  229.     Y_train = dataset_train.y
  230.     X_valid = dataset_valid.X_cat if dataset_valid.X_num is None else np.hstack((dataset_valid.X_num, dataset_valid.X_cat))
  231.     Y_valid = dataset_valid.y
  232.     X_test = dataset_test.X_cat if dataset_test.X_num is None else np.hstack((dataset_test.X_num, dataset_test.X_cat))
  233.     Y_test = dataset_test.y
  234.     
  235.     fit_kwargs['eval_set'] = [(X_valid,Y_valid)]
  236.     model.fit(X_train, Y_train, **fit_kwargs)
  237.     prediction = model.predict(X_test)
  238.     result = skm.classification_report(Y_test, prediction, output_dict=True)
  239.     model.save_model(model_state_dict_path)
  240.     recall = result["1"]["recall"]
  241.     acc = result['accuracy']
  242.     with open(model_result_records,"a") as f:
  243.         f.write(f"seed{seed} accuracy is:{acc} and the recall is :{recall}\n")
  244.     np.save(feature_importance_record_path, model.feature_importances_)
  245.     
  246. def train_model_lightGBM(model,fit_kwargs,dataset_train,dataset_valid,dataset_test,seed):
  247.     model_state_dict_path = os.path.join(os.getcwd(),f"lightGBM_state_seed_{seed}.pickle")
  248.     model_result_records = "lightGBM_Result.txt"
  249.     feature_importance_record_path = f"lightGBM_feature_importance_{seed}.npy"
  250.     if os.path.exists(model_state_dict_path):
  251.         return
  252.     X_train = dataset_train.X_cat if dataset_train.X_num is None else np.hstack((dataset_train.X_num, dataset_train.X_cat))
  253.     Y_train = dataset_train.y
  254.     X_valid = dataset_valid.X_cat if dataset_valid.X_num is None else np.hstack((dataset_valid.X_num, dataset_valid.X_cat))
  255.     Y_valid = dataset_valid.y
  256.     X_test = dataset_test.X_cat if dataset_test.X_num is None else np.hstack((dataset_test.X_num, dataset_test.X_cat))
  257.     Y_test = dataset_test.y
  258.     n_num_features = dataset_train.X_num.shape[1]
  259.     n_features = dataset_train.X_num.shape[1]+dataset_train.X_cat.shape[1]
  260.     fit_kwargs['categorical_feature'] = list(range(n_num_features, n_features))
  261.     model.fit(X_train, Y_train, **fit_kwargs,eval_set=(X_valid, Y_valid))
  262.     prediction = model.predict(X_test)
  263.     result = skm.classification_report(Y_test, prediction, output_dict=True)
  264.     recall = result["1"]["recall"]
  265.     acc = result['accuracy']
  266. #     joblib.dump(model, model_state_dict_path)
  267.     with open(model_result_records,"a") as f:
  268.         f.write(f"seed{seed} accuracy is:{acc} and the recall is :{recall}\n")
  269.     np.save(feature_importance_record_path, model.feature_importances_)
  270.  
  271. def train_model(model,config,dl_train,dl_valid,dl_test,seed,model_type,is_mine=False):
  272.     model_state_dict_path = os.path.join(os.getcwd(),f"{model_type}_state_seed_{seed}.pickle")
  273.     model_result_records = f"{model_type}_Result.txt"
  274.     
  275.     if is_mine:
  276.         model_state_dict_path = model_state_dict_path.replace(".pickle","_mine.pickle")
  277.         model_result_records = model_result_records.replace(".txt","_mine.txt")
  278.     
  279.     if os.path.exists(model_state_dict_path):
  280.         return
  281.     
  282.     parameters_with_wd = [v for k, v in model.named_parameters() if needs_wd(k)]
  283.     parameters_without_wd = [v for k, v in model.named_parameters() if not needs_wd(k)]
  284.     
  285.     loss_fn = F.binary_cross_entropy_with_logits
  286.  
  287.     optimizer = make_optimizer(
  288.         config["training"]["optimizer"],
  289.         (
  290.             [
  291.                 {'params': parameters_with_wd},
  292.                 {'params': parameters_without_wd, 'weight_decay': 0.0},
  293.             ]
  294.         ),
  295.         config["training"]["lr"],#to be trained in optuna
  296.         config["training"]["weight_decay"]#to be trained in optuna
  297.     )
  298.     
  299.     loss_best = np.nan
  300.     best_epoch = -1
  301.     patience = 0
  302.     
  303.     def save_state():
  304.         torch.save(
  305.             model.state_dict(),model_state_dict_path
  306.         )
  307.     
  308.     #dl_train.batch_size=config['training']['batch_size']
  309.     for epoch in range(config['training']['n_epochs']):
  310.         model.train()
  311.         for i,(x_num,x_cat,y) in enumerate(dl_train):
  312.             optimizer.zero_grad()
  313.             y_batch = model(x_num,x_cat)
  314.             loss = loss_fn(y_batch.reshape((y_batch.shape[0],1)),y)
  315.             loss.backward()
  316.             optimizer.step()
  317.             
  318.         model.eval()
  319.         with torch.no_grad():
  320.             for i,(x_num,x_cat,y) in enumerate(dl_valid): #只有1个迭代
  321.                 y_batch = model(x_num,x_cat)
  322.                 loss = loss_fn(y_batch.reshape((y_batch.shape[0],1)),y)
  323.                 new_loss = loss.detach()
  324.                 if np.isnan(loss_best) or new_loss.cpu().numpy() < loss_best:
  325.                     patience = 0
  326.                     best_epoch = epoch
  327.                     loss_best = new_loss.cpu().numpy()
  328.                     save_state()
  329.                 else:
  330.                     patience+=1
  331.         if patience>= config['training']['patience']:
  332.             break
  333.     
  334.     #读取state_dict
  335.     model.load_state_dict(
  336.         torch.load(model_state_dict_path)
  337.                     )
  338.     model.eval()
  339.     with torch.no_grad():
  340.         for i,(x_num,x_cat,y) in enumerate(dl_test): #只有1个迭代
  341.             y_batch = model(x_num,x_cat)
  342.             prediction = y_batch.cpu().numpy()
  343.             prediction = np.round(scipy.special.expit(prediction)).astype('int64')
  344.             result = skm.classification_report(y.cpu().numpy(), prediction, output_dict=True)
  345.     recall = result["1.0"]["recall"]
  346.     acc = result['accuracy']
  347.     with open(model_result_records,"a") as f:
  348.         f.write(f"seed{seed} accuracy is:{acc} and the recall is :{recall}\n")
  349.         
  350. def train_model_LassoNet(model,config,dl_train,dl_valid,dl_test,seed):
  351.     model_state_dict_path = os.path.join(os.getcwd(),f"lassoNet_state_seed_{seed}.pickle")
  352.     model_result_records = f"lassoNet_Result.txt"
  353.     if os.path.exists(model_state_dict_path):
  354.         return
  355.     parameters_with_wd = [v for k, v in model.named_parameters() if needs_wd(k)]
  356.     parameters_without_wd = [v for k, v in model.named_parameters() if not needs_wd(k)]
  357.     
  358.     loss_fn = F.binary_cross_entropy_with_logits
  359.  
  360.     optimizer = make_optimizer(
  361.         config["training"]["optimizer"],
  362.         (
  363.             [
  364.                 {'params': parameters_with_wd},
  365.                 {'params': parameters_without_wd, 'weight_decay': 0.0},
  366.             ]
  367.         ),
  368.         config["training"]["lr"],#to be trained in optuna
  369.         config["training"]["weight_decay"]#to be trained in optuna
  370.     )
  371.     
  372.     loss_best = np.nan
  373.     best_epoch = -1
  374.     patience = 0
  375.     
  376.     def save_state():
  377.         torch.save(
  378.             model.state_dict(),model_state_dict_path
  379.         )
  380.     
  381.     #dl_train.batch_size=config['training']['batch_size']
  382.     for epoch in range(config['training']['n_epochs']):
  383.         model.train()
  384.         for i,(x_num,x_cat,y) in enumerate(dl_train):
  385.             optimizer.zero_grad()
  386. #             y_batch = model(x_num,x_cat)
  387.             loss = 0
  388.             def closure():
  389.                 nonlocal loss
  390.                 optimizer.zero_grad()
  391.                 ans = (
  392.                     loss_fn(model(x_num,x_cat), y)
  393.                     + model.gamma * model.l2_regularization()
  394.                     + model.gamma_skip * model.l2_regularization_skip()
  395.                 )
  396.                 ans.backward() #相当于第7行Compute gradient of the loss
  397.                 loss += ans.item()# * len(batch) / n_train
  398.                 return ans
  399.             optimizer.step(closure)
  400.             model.prox(lambda_=model.lambda_ * optimizer.param_groups[0]["lr"], M=model.M) #Hier-Prox算法
  401.             
  402.         model.eval()
  403.         with torch.no_grad():
  404.             for i,(x_num,x_cat,y) in enumerate(dl_valid): #只有1个迭代
  405.                 y_batch = model(x_num,x_cat)#.reshape((y_batch.shape[0],1))
  406.                 loss = (
  407.                     loss_fn(y_batch.reshape((y_batch.shape[0],1)), y).item()
  408.                     + model.gamma * model.l2_regularization().item()
  409.                     + model.gamma_skip * model.l2_regularization_skip().item()
  410.                     + model.lambda_ * model.l1_regularization_skip().item()
  411.                 )
  412.                 new_loss = loss#.detach()
  413.                 if np.isnan(loss_best) or new_loss < loss_best:
  414.                     patience = 0
  415.                     best_epoch = epoch
  416.                     loss_best = new_loss#.cpu().numpy()
  417.                     save_state()
  418.                 else:
  419.                     patience+=1
  420.         if patience>= config['training']['patience']:
  421.             break
  422.     
  423.     #读取state_dict
  424.     model.load_state_dict(
  425.         torch.load(model_state_dict_path)
  426.                     )
  427.     model.eval()
  428.     with torch.no_grad():
  429.         for i,(x_num,x_cat,y) in enumerate(dl_test): #只有1个迭代
  430.             y_batch = model(x_num,x_cat)
  431.             prediction = y_batch.cpu().numpy()
  432.             prediction = np.round(scipy.special.expit(prediction)).astype('int64')
  433.             result = skm.classification_report(y.cpu().numpy(), prediction, output_dict=True)
  434.     recall = result["1.0"]["recall"]
  435.     acc = result['accuracy']
  436.     with open(model_result_records,"a") as f:
  437.         f.write(f"seed{seed} accuracy is:{acc} and the recall is :{recall}\n")
  438.  
  439.  
  440. import sklearn.metrics as skm
  441.  
  442. ##  设定训练参数
  443. def train_by_model(configs,seed,model_type):
  444.     zero.set_randomness(seed)
  445.     
  446.     def build_dataloaders(configs):
  447.         data_configs = configs["data"]
  448.  
  449.         D_train = CustomDataset(
  450.             data_path_father+"adult"
  451.             ,data_part="train"
  452.             ,normalization=data_configs.get("normalization")
  453.             ,num_nan_policy="mean"
  454.             ,cat_nan_policy="new"
  455.             ,cat_policy=data_configs.get("cat_policy", 'indices')
  456.             ,seed=seed
  457.             ,y_poicy=data_configs.get("y_policy"),cat_min_frequency=0
  458.         )
  459.  
  460.         D_valid = CustomDataset(
  461.             data_path_father+"adult"
  462.             ,data_part="val"
  463.             ,normalization=data_configs.get("normalization")
  464.             ,num_nan_policy="mean"
  465.             ,cat_nan_policy="new"
  466.             ,cat_policy=data_configs.get("cat_policy", 'indices')
  467.             ,seed=seed
  468.             ,y_poicy=data_configs.get("y_policy"),cat_min_frequency=0
  469.         )
  470.  
  471.         D_test = CustomDataset(
  472.             data_path_father+"adult"
  473.             ,data_part="test"
  474.             ,normalization=data_configs.get("normalization")
  475.             ,num_nan_policy="mean"
  476.             ,cat_nan_policy="new"
  477.             ,cat_policy=data_configs.get("cat_policy", 'indices')
  478.             ,seed=seed
  479.             ,y_poicy=data_configs.get("y_policy"),cat_min_frequency=0
  480.         )
  481.         dl_train = DataLoader(D_train,batch_size=256)
  482.         dl_val = DataLoader(D_valid,batch_size=len(D_valid))
  483.         dl_test = DataLoader(D_test,batch_size=len(D_test))
  484.  
  485.         return D_train,D_valid,D_test,dl_train,dl_val,dl_test
  486.     
  487.     D_train,D_valid,D_test,dl_train,dl_valid,dl_test = build_dataloaders(configs)
  488.     if "FT_Transformer" in model_type:
  489.         configs["model"].setdefault('token_bias', True)
  490.         configs["model"].setdefault('kv_compression', None)
  491.         configs["model"].setdefault('kv_compression_sharing', None)
  492.         model = Transformer(d_numerical=0 if D_train.X_num is None else D_train.X_num.shape[1],
  493.                             categories = None if D_train.X_cat is None else [len(set(D_train.X_cat[:, i].tolist())) for i in range(D_train.X_cat.shape[1])],
  494.                            d_out=D_train.info['n_classes'] if D_train.info["task_type"]=="multiclass" else 1
  495.                             ,**configs['model']).to(device)
  496.         is_mine = model_type.endswith("_mine")
  497.         train_model(model,configs,dl_train,dl_valid,dl_test,seed,"FT_Transformer",is_mine=is_mine)
  498.     elif model_type == "LassoNet":
  499.         model = LassoNet(
  500.             d_numerical=0 if D_train.X_num is None else D_train.X_num.shape[1],
  501.             categories = None if D_train.X_cat is None else [len(set(D_train.X_cat[:, i].tolist())) for i in range(D_train.X_cat.shape[1])],
  502.             d_out=D_train.info['n_classes'] if D_train.info["task_type"]=="multiclass" else 1
  503.             ,**configs['model']
  504.             ).to(device)
  505.         train_model_LassoNet(model,configs,dl_train,dl_valid,dl_test,seed)
  506.     elif model_type == "ResNet":
  507.         model = ResNet(
  508.             d_numerical=0 if D_train.X_num is None else D_train.X_num.shape[1],
  509.             categories = None if D_train.X_cat is None else [len(set(D_train.X_cat[:, i].tolist())) for i in range(D_train.X_cat.shape[1])],
  510.             d_out=D_train.info['n_classes'] if D_train.info["task_type"]=="multiclass" else 1,
  511.             **configs['model'],
  512.         ).to(device)
  513.         train_model(model,configs,dl_train,dl_valid,dl_test,seed,"ResNet",is_mine=False)
  514.     elif model_type == "xgboost":
  515.         fit_kwargs = deepcopy(configs["fit"])
  516.         configs["model"]['random_state'] = seed
  517.         fit_kwargs['eval_metric'] = 'error'
  518.         model = XGBClassifier(**configs["model"])
  519.         train_model_xgboost(model,fit_kwargs,D_train,D_valid,D_test,seed)
  520.     elif model_type == "lightGBM":
  521.         model_kwargs = deepcopy(configs['model'])
  522.         model_kwargs['random_state'] = seed
  523.         fit_kwargs = deepcopy(configs['fit'])
  524.         early_stop_rounds = fit_kwargs.get("early_stopping_rounds")
  525.         del fit_kwargs["early_stopping_rounds"]
  526.         del fit_kwargs["verbose"]
  527.         fit_kwargs['eval_metric'] = 'binary_error'
  528.         ES = early_stopping(early_stop_rounds)
  529.         verbose = log_evaluation(10**8)
  530.         model = LGBMClassifier(**model_kwargs,callbacks = [ES,verbose])
  531.         train_model_lightGBM(model,fit_kwargs,D_train,D_valid,D_test,seed)
  532.     else:
  533.         raise ValueError("model_type not recognized")
  534.     
  535.  
  536. seeds=[6368,1658,8366,8641,7052,7600,297,5829,9295,1698,2157,3318,8312,7741,9570]
  537. for i,seed in enumerate(seeds):
  538.     train_by_model(xgboost_config,seed,"xgboost")
  539.     train_by_model(lightGBM_config,seed,"lightGBM")
  540.     train_by_model(ft_transformer_config,seed,"FT_Transformer")
  541.     train_by_model(ft_transformer_mine_config,seed,"FT_Transformer_mine")
  542.     train_by_model(resNet_config,seed,"ResNet")
  543.     train_by_model(LassoNet_config,seed,"LassoNet")

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/秋刀鱼在做梦/article/detail/781905
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号