devbox
在线问答5
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

fasttext 文本分类_step by step带你fastText文本分类

作者：在线问答5 | 2024-08-12 05:06:17

赞

踩

fasttext 向量 分类

本文参考原文-http://bjbsair.com/2020-03-25/tech-info/6300/ 写在前面

今天的教程是基于FAIR的Bag of Tricks for Efficient Text Classification[1]。也就是我们常说的fastText。

最让人欣喜的这篇论文配套提供了fasttext工具包。这个工具包代码质量非常高，论文结果一键还原，目前已经是包装地非常专业了，这是fastText官网和其github代码库，以及提供了python接口，可以直接通过pip安装。这样准确率高又快的模型绝对是实战利器。

为了更好地理解fasttext原理，我们现在直接复现来一遍，但是代码中仅仅实现了最简单的基于单词的词向量求平均，并未使用b-gram的词向量，所以自己实现的文本分类效果会低于facebook开源的库。

论文概览

❝
We can train fastText on more than one billion words in less than ten minutes using a standard multicore CPU, and classify half a million sentences among 312K classes in less than a minute.

首先引用论文中的一段话来看看作者们是怎么评价fasttext模型的表现的。

这篇论文的模型非常之简单，之前了解过word2vec的同学可以发现这跟CBOW的模型框架非常相似。

bce57b223b4a70ff414221aa8b6222bb.png

对应上面这个模型，比如输入是一句话，到就是这句话的单词或者是n-gram。每一个都对应一个向量，然后对这些向量取平均就得到了文本向量，然后用这个平均向量取预测标签。当类别不多的时候，就是最简单的softmax；当标签数量巨大的时候，就要用到「hierarchical softmax」了。

模型真的很简单，也没什么可以说的了。下面提一下论文中的两个tricks：

  • 「hierarchical softmax」
    类别数较多时，通过构建一个霍夫曼编码树来加速softmax layer的计算，和之前word2vec中的trick相同
  • 「N-gram features」
    只用unigram的话会丢掉word order信息，所以通过加入N-gram features进行补充 用hashing来减少N-gram的存储

看了论文的实验部分，如此简单的模型竟然能取得这么好的效果 ！

但是也有人指出论文中选取的数据集都是对句子词序不是很敏感的数据集，所以得到文中的试验结果并不奇怪。

代码实现

看完阉割版代码大家记得去看看源码噢~ 跟之前系列的一样，定义一个fastTextModel类，然后写网络框架，输入输出placeholder，损失，训练步骤等。

  1. class fastTextModel(BaseModel):  
  2.     """  
  3.     A simple implementation of fasttext for text classification  
  4.     """  
  5.     def __init__(self, sequence_length, num_classes, vocab_size,  
  6.                  embedding_size, learning_rate, decay_steps, decay_rate,  
  7.                  l2_reg_lambda, is_training=True,  
  8.                  initializer=tf.random_normal_initializer(stddev=0.1)):  
  9.         self.vocab_size = vocab_size  
  10.         self.embedding_size = embedding_size  
  11.         self.num_classes = num_classes  
  12.         self.sequence_length = sequence_length  
  13.         self.learning_rate = learning_rate  
  14.         self.decay_steps = decay_steps  
  15.         self.decay_rate = decay_rate  
  16.         self.is_training = is_training  
  17.         self.l2_reg_lambda = l2_reg_lambda  
  18.         self.initializer = initializer  
  19.         self.input_x = tf.placeholder(tf.int32, [None, self.sequence_length], name='input_x')  
  20.         self.input_y = tf.placeholder(tf.int32, [None, self.num_classes], name='input_y')  
  21.         self.global_step = tf.Variable(0, trainable=False, name='global_step')  
  22.         self.instantiate_weight()  
  23.         self.logits = self.inference()  
  24.         self.loss_val = self.loss()  
  25.         self.train_op = self.train()  
  26.         self.predictions = tf.argmax(self.logits, axis=1, name='predictions')  
  27.         correct_prediction = tf.equal(self.predictions, tf.argmax(self.input_y, 1))  
  28.         self.accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'), name='accuracy')  
  29.     def instantiate_weight(self):  
  30.         with tf.name_scope('weights'):  
  31.             self.Embedding = tf.get_variable('Embedding', shape=[self.vocab_size, self.embedding_size],  
  32.                                              initializer=self.initializer)  
  33.             self.W_projection = tf.get_variable('W_projection', shape=[self.embedding_size, self.num_classes],  
  34.                                                 initializer=self.initializer)  
  35.             self.b_projection = tf.get_variable('b_projection', shape=[self.num_classes])  
  36.     def inference(self):  
  37.         """  
  38.         1. word embedding  
  39.         2. average embedding  
  40.         3. linear classifier  
  41.         :return:  
  42.         """  
  43.         # embedding layer  
  44.         with tf.name_scope('embedding'):  
  45.             words_embedding = tf.nn.embedding_lookup(self.Embedding, self.input_x)  
  46.             self.average_embedding = tf.reduce_mean(words_embedding, axis=1)  
  47.         logits = tf.matmul(self.average_embedding, self.W_projection) +self.b_projection  
  48.         return logits  
  49.     def loss(self):  
  50.         # loss  
  51.         with tf.name_scope('loss'):  
  52.             losses = tf.nn.softmax_cross_entropy_with_logits(labels=self.input_y, logits=self.logits)  
  53.             data_loss = tf.reduce_mean(losses)  
  54.             l2_loss = tf.add_n([tf.nn.l2_loss(cand_var) for cand_var in tf.trainable_variables()  
  55.                                 if 'bias' not in cand_var.name]) * self.l2_reg_lambda  
  56.             data_loss += l2_loss * self.l2_reg_lambda  
  57.             return data_loss  
  58.     def train(self):  
  59.         with tf.name_scope('train'):  
  60.             learning_rate = tf.train.exponential_decay(self.learning_rate, self.global_step,  
  61.                                                        self.decay_steps, self.decay_rate,  
  62.                                                        staircase=True)  
  63.             train_op = tf.contrib.layers.optimize_loss(self.loss_val, global_step=self.global_step,  
  64.                                                       learning_rate=learning_rate, optimizer='Adam')  
  65.         return train_op
  66. def prepocess():  
  67.     """  
  68.     For load and process data  
  69.     :return:  
  70.     """  
  71.     print("Loading data...")  
  72.     x_text, y = data_process.load_data_and_labels(FLAGS.positive_data_file, FLAGS.negative_data_file)  
  73.     # bulid vocabulary  
  74.     max_document_length = max(len(x.split(' ')) for x in x_text)  
  75.     vocab_processor = learn.preprocessing.VocabularyProcessor(max_document_length)  
  76.     x = np.array(list(vocab_processor.fit_transform(x_text)))  
  77.     # shuffle  
  78.     np.random.seed(10)  
  79.     shuffle_indices = np.random.permutation(np.arange(len(y)))  
  80.     x_shuffled = x[shuffle_indices]  
  81.     y_shuffled = y[shuffle_indices]  
  82.     # split train/test dataset  
  83.     dev_sample_index = -1 * int(FLAGS.dev_sample_percentage * float(len(y)))  
  84.     x_train, x_dev = x_shuffled[:dev_sample_index], x_shuffled[dev_sample_index:]  
  85.     y_train, y_dev = y_shuffled[:dev_sample_index], y_shuffled[dev_sample_index:]  
  86.     del x, y, x_shuffled, y_shuffled  
  87.     print('Vocabulary Size: {:d}'.format(len(vocab_processor.vocabulary_)))  
  88.     print('Train/Dev split: {:d}/{:d}'.format(len(y_train), len(y_dev)))  
  89.     return x_train, y_train, vocab_processor, x_dev, y_dev  
  90. def train(x_train, y_train, vocab_processor, x_dev, y_dev):  
  91.     with tf.Graph().as_default():  
  92.         session_conf = tf.ConfigProto(  
  93.             # allows TensorFlow to fall back on a device with a certain operation implemented  
  94.             allow_soft_placement= FLAGS.allow_soft_placement,  
  95.             # allows TensorFlow log on which devices (CPU or GPU) it places operations  
  96.             log_device_placement=FLAGS.log_device_placement  
  97.         )  
  98.         sess = tf.Session(config=session_conf)  
  99.         with sess.as_default():  
  100.             # initialize cnn  
  101.             fasttext = fastTextModel(sequence_length=x_train.shape[1],  
  102.                       num_classes=y_train.shape[1],  
  103.                       vocab_size=len(vocab_processor.vocabulary_),  
  104.                       embedding_size=FLAGS.embedding_size,  
  105.                       l2_reg_lambda=FLAGS.l2_reg_lambda,  
  106.                       is_training=True,  
  107.                       learning_rate=FLAGS.learning_rate,  
  108.                       decay_steps=FLAGS.decay_steps,  
  109.                       decay_rate=FLAGS.decay_rate  
  110.                     )  
  111.             # output dir for models and summaries  
  112.             timestamp = str(time.time())  
  113.             out_dir = os.path.abspath(os.path.join(os.path.curdir, 'run', timestamp))  
  114.             if not os.path.exists(out_dir):  
  115.                 os.makedirs(out_dir)  
  116.             print('Writing to {} n'.format(out_dir))  
  117.             # checkpoint dir. checkpointing – saving the parameters of your model to restore them later on.  
  118.             checkpoint_dir = os.path.abspath(os.path.join(out_dir, FLAGS.ckpt_dir))  
  119.             checkpoint_prefix = os.path.join(checkpoint_dir, 'model')  
  120.             if not os.path.exists(checkpoint_dir):  
  121.                 os.makedirs(checkpoint_dir)  
  122.             saver = tf.train.Saver(tf.global_variables(), max_to_keep=FLAGS.num_checkpoints)  
  123.             # Write vocabulary  
  124.             vocab_processor.save(os.path.join(out_dir, 'vocab'))  
  125.             # Initialize all  
  126.             sess.run(tf.global_variables_initializer())  
  127.             def train_step(x_batch, y_batch):  
  128.                 """  
  129.                 A single training step  
  130.                 :param x_batch:  
  131.                 :param y_batch:  
  132.                 :return:  
  133.                 """  
  134.                 feed_dict = {  
  135.                     fasttext.input_x: x_batch,  
  136.                     fasttext.input_y: y_batch,  
  137.                 }  
  138.                 _, step, loss, accuracy = sess.run(  
  139.                     [fasttext.train_op, fasttext.global_step, fasttext.loss_val, fasttext.accuracy],  
  140.                     feed_dict=feed_dict  
  141.                 )  
  142.                 time_str = datetime.datetime.now().isoformat()  
  143.                 print("{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy))  
  144.             def dev_step(x_batch, y_batch):  
  145.                 """  
  146.                 Evaluate model on a dev set  
  147.                 Disable dropout  
  148.                 :param x_batch:  
  149.                 :param y_batch:  
  150.                 :param writer:  
  151.                 :return:  
  152.                 """  
  153.                 feed_dict = {  
  154.                     fasttext.input_x: x_batch,  
  155.                     fasttext.input_y: y_batch,  
  156.                 }  
  157.                 step, loss, accuracy = sess.run(  
  158.                     [fasttext.global_step, fasttext.loss_val, fasttext.accuracy],  
  159.                     feed_dict=feed_dict  
  160.                 )  
  161.                 time_str = datetime.datetime.now().isoformat()  
  162.                 print("dev results:{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy))  
  163.             # generate batches  
  164.             batches = data_process.batch_iter(list(zip(x_train, y_train)), FLAGS.batch_size, FLAGS.num_epochs)  
  165.             # training loop  
  166.             for batch in batches:  
  167.                 x_batch, y_batch = zip(*batch)  
  168.                 train_step(x_batch, y_batch)  
  169.                 current_step = tf.train.global_step(sess, fasttext.global_step)  
  170.                 if current_step % FLAGS.validate_every == 0:  
  171.                     print('n Evaluation:')  
  172.                     dev_step(x_dev, y_dev)  
  173.                     print('')  
  174.             path = saver.save(sess, checkpoint_prefix, global_step=current_step)  
  175.             print('Save model checkpoint to {} n'.format(path))  
  176. def main(argv=None):  
  177.     x_train, y_train, vocab_processor, x_dev, y_dev = prepocess()  
  178.     train(x_train, y_train, vocab_processor, x_dev, y_dev)  
  179. if __name__ == '__main__':  
  180.     tf.app.run()

本文参考资料

[1] Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759

The End

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/在线问答5/article/detail/967857
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号