从零开始深度学习0510——RNN+LSTM基本知识+LSTM做回归任务实例_从零实现rnn和lstm

0510

 

RNN的东西不会去忘记，直接一串子全部保留下来

 

Lstm 长短时记忆  

可以控制

这个参数也是需要去训练 逐渐优化 得到的

门单元

ft 遗忘门   it 保留门

输入经过了一个遗忘门，经过了一个保留门

选择性的遗忘和保留  是随时更新ct的状态的 从开始更新到结束

 

Classification and Location 分类与回归

分类就是 输入图像 输出类别

回归就是 输入图像 输出这个图像的(x,y,w,h) 就是定位，找到坐标和长宽高

将回归的任务  加在哪里

 

L2 distance 欧式距离

L2 regularliaztion  L2正则化惩罚项   

L2 loss  均方误差 常用来做回归任务  MSE

 

用活动窗口的做法  去进行回归任务

 

其他深度神经网络  在达到一定的层数后，层数越深，不一定效果越好

但是ResNet 是层数越深，效果越好

 

 

 

 

 

 

 

 

 

 

 

 

 

设计网络的技巧

 

 Data augmentation 数据增强

 

1. Horizontal flips  水平翻转  镜面操作

1. 随机裁剪
2. 

Translation 平移变换

Rotation  角度变换

Stretching 拉伸

Shearing  修剪

Lens distortions

 

 

 

Rnn中time_step

https://www.zhihu.com/question/271774530/answer/364711129

 

https://blog.csdn.net/program_developer/article/details/84794556

 

每个minibatch中的每个样本应该是成上下文相关的，什么意思呢？比如做诗的生成，每首诗作为一个单独的样本。它的内部的字的上下文是这首诗，而与另外一首诗没有关系。在做梯度更新的时候，每个样本做前向传播初始隐状态h是0，并不会考虑前面的诗的信息。一个样本中构成了一个序列，time_step就是这个序列的长度，time_step就是说要指定多长的序列能够构成一个上下文相关的序列。诗歌这个例子，很明确就是一首诗的长度。如果我们把time_step设置成每个诗句而不是整首诗，那么结果产生的只是孤立的一句句诗而不是一整首诗歌。可能存在每首诗的长度不一样长，于是就有了max_time这个参数，设定最长的诗的长度，其他不够长的诗补长到max_time的大小。我在stackoverflow看到，实际上我们可以不设置time_step这个参数或者说设置为None，我没有具体看是keras还是tensorflow的实现。我认为如果可以不设置，那么最好就这样处理。那么什么时候需要关心time_step这个参数呢？比如文章的生成。设置time_step就是要关心一个单独的序列样本应该多长，比如你认为一个段落就是一个样本，太远的句子并不会多当前的段落影响太多。做预测的时候与time_step有什么关系呢？训练时样本的序列长度是多少生成是就应该多少这样最为合理。当然也可以无休止的生成这样只是可能效果不会怎么好。

 

Lstm，做回归任务

 详细参考  ComputerVision ---->  Neural_NetWorks ---->  RNN-easy-àrnn_full_code.py

 

 
 
# View more python learning tutorial on my Youtube and Youku channel!!!
 
# Youtube video tutorial: https://www.youtube.com/channel/UCdyjiB5H8Pu7aDTNVXTTpcg
# Youku video tutorial: http://i.youku.com/pythontutorial
 
"""
Please note, this code is only for python 3+. If you are using python 2+, please modify the code accordingly.
Run this script on tensorflow r0.10. Errors appear when using lower versions.
lstm  回归问题的预测
"""
 
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
 
 
BATCH_START = 0 #
TIME_STEPS = 20 #每20个作为一个有上下文的时间序列
BATCH_SIZE = 50 #50个一组训练
INPUT_SIZE = 1 #
OUTPUT_SIZE = 1 #
CELL_SIZE = 10 #10个cell
LR = 0.006 #学习率
 
#生成数据
def get_batch():
    global BATCH_START, TIME_STEPS
    # xs shape (50batch, 20steps)
    xs = np.arange(BATCH_START, BATCH_START+TIME_STEPS*BATCH_SIZE).reshape((BATCH_SIZE, TIME_STEPS)) / (10*np.pi)
    seq = np.sin(xs)  # b
    res = np.cos(xs)  # r
    BATCH_START += TIME_STEPS
    # plt.plot(xs[0, :], res[0, :], 'r', xs[0, :], seq[0, :], 'b--')
    # plt.show()
    # returned seq, res and xs: shape (batch, step, input)
    return [seq[:, :, np.newaxis], res[:, :, np.newaxis], xs] # 返回生成的数据  seq res xs
 
 
class LSTMRNN(object):
    def __init__(self, n_steps, input_size, output_size, cell_size, batch_size):
        self.n_steps = n_steps
        self.input_size = input_size
        self.output_size = output_size
        self.cell_size = cell_size
        self.batch_size = batch_size
        with tf.name_scope('inputs'):
            self.xs = tf.placeholder(tf.float32, [None, n_steps, input_size], name='xs') # 定义占位符是3维 [batch_size, time_step, word_dim]
            self.ys = tf.placeholder(tf.float32, [None, n_steps, output_size], name='ys') #
        with tf.variable_scope('in_hidden'):
            self.add_input_layer()
        with tf.variable_scope('LSTM_cell'):
            self.add_cell()
        with tf.variable_scope('out_hidden'):
            self.add_output_layer()
        with tf.name_scope('cost'):
            self.compute_cost()
        with tf.name_scope('train'):
            self.train_op = tf.train.AdamOptimizer(LR).minimize(self.cost)
 
    def add_input_layer(self,):
        l_in_x = tf.reshape(self.xs, [-1, self.input_size], name='2_2D')  # (batch*n_step, in_size)  将三维的数据reshape成二维
        # Ws (in_size, cell_size)
        Ws_in = self._weight_variable([self.input_size, self.cell_size])
        # bs (cell_size, )
        bs_in = self._bias_variable([self.cell_size,])
        # l_in_y = (batch * n_steps, cell_size)
        with tf.name_scope('Wx_plus_b'):
            l_in_y = tf.matmul(l_in_x, Ws_in) + bs_in
        # reshape l_in_y ==> (batch, n_steps, cell_size)
        self.l_in_y = tf.reshape(l_in_y, [-1, self.n_steps, self.cell_size], name='2_3D') #再将二维的结果 reshape 成三维
 
    def add_cell(self):
        lstm_cell = tf.contrib.rnn.BasicLSTMCell(self.cell_size, forget_bias=1.0, state_is_tuple=True)
        with tf.name_scope('initial_state'):
            self.cell_init_state = lstm_cell.zero_state(self.batch_size, dtype=tf.float32)
        self.cell_outputs, self.cell_final_state = tf.nn.dynamic_rnn(
            lstm_cell, self.l_in_y, initial_state=self.cell_init_state, time_major=False)
 
    def add_output_layer(self):
        # shape = (batch * steps, cell_size)
        l_out_x = tf.reshape(self.cell_outputs, [-1, self.cell_size], name='2_2D')
        Ws_out = self._weight_variable([self.cell_size, self.output_size])
        bs_out = self._bias_variable([self.output_size, ])
        # shape = (batch * steps, output_size)
        with tf.name_scope('Wx_plus_b'):
            self.pred = tf.matmul(l_out_x, Ws_out) + bs_out
 
    def compute_cost(self):
        losses = tf.contrib.legacy_seq2seq.sequence_loss_by_example(
            [tf.reshape(self.pred, [-1], name='reshape_pred')],
            [tf.reshape(self.ys, [-1], name='reshape_target')],
            [tf.ones([self.batch_size * self.n_steps], dtype=tf.float32)],
            average_across_timesteps=True,
            softmax_loss_function=self.ms_error,
            name='losses'
        )
        with tf.name_scope('average_cost'):
            self.cost = tf.div(
                tf.reduce_sum(losses, name='losses_sum'),
                self.batch_size,
                name='average_cost')
            tf.summary.scalar('cost', self.cost)
 
    @staticmethod
    def ms_error(labels, logits):
        return tf.square(tf.subtract(labels, logits))
 
    def _weight_variable(self, shape, name='weights'):
        initializer = tf.random_normal_initializer(mean=0., stddev=1.,) #均方误差为1
        return tf.get_variable(shape=shape, initializer=initializer, name=name)
 
    def _bias_variable(self, shape, name='biases'):
        initializer = tf.constant_initializer(0.1) #常量初始化
        return tf.get_variable(name=name, shape=shape, initializer=initializer)
 
 
if __name__ == '__main__':
    model = LSTMRNN(TIME_STEPS, INPUT_SIZE, OUTPUT_SIZE, CELL_SIZE, BATCH_SIZE)
    sess = tf.Session()
    merged = tf.summary.merge_all()  #用来 tensorbord可视化
    writer = tf.summary.FileWriter("./logs/", sess.graph) #保存到log文件夹
    # tf.initialize_all_variables() no long valid from
    # 2017-03-02 if using tensorflow >= 0.12
    if int((tf.__version__).split('.')[1]) < 12 and int((tf.__version__).split('.')[0]) < 1:
        init = tf.initialize_all_variables()
    else:
        init = tf.global_variables_initializer()
    sess.run(init)
    # relocate to the local dir and run this line to view it on Chrome (http://0.0.0.0:6006/):
    # $ tensorboard --logdir='logs'
 
    plt.ion() #启动交互模式 实现动图
    plt.show()
    for i in range(200):
        seq, res, xs = get_batch()
        if i == 0:
            feed_dict = {
                    model.xs: seq,
                    model.ys: res,
                    # create initial state
            }
        else:
            feed_dict = {
                model.xs: seq,
                model.ys: res,
                model.cell_init_state: state    # use last state as the initial state for this run
            }
 
        _, cost, state, pred = sess.run(
            [model.train_op, model.cost, model.cell_final_state, model.pred],
            feed_dict=feed_dict)
 
        # plotting
        plt.plot(xs[0, :], res[0].flatten(), 'r', xs[0, :], pred.flatten()[:TIME_STEPS], 'b--')
        plt.ylim((-1.2, 1.2))
        plt.draw()
        plt.pause(0.3)
 
        if i % 20 == 0:
            print('cost: ', round(cost, 4))
            result = sess.run(merged, feed_dict)
            writer.add_summary(result, i)
 

 

 

 

 

 

Autoencoder 自编码  非监督学习

类似PCA主成分分析   可以达到降维的目的 拿更优质特征更多的来代替原来图像中无用的信息

就是将图片打码，再将打码后的图片还原

 

Batch Normalization --------------BN