P13 Deep Learning_define a set of function

P13 Deep Learning

Three Steps for Deep Learning

• Step 1：define a set of function
• Step 2：goodness of function
• Step 3：pick the best function

Step 1：define a set of function

• 如何连接Neural

  • Fully Connect Feedforward Network
    • Input：1 和 -1

  

• 计算出全部的结果

• Input：0 和 0

• 一个Network给定了参数就是一个Function

• Given network structure,define a function set

Fully Connect Feedforward Network

• 两两相互连接
• Input Layer
• Hidden Layers
• Output Layer

• Deep = Many hidden layers

Matrix Operation

Neural Network

• 一连串的Matrix Operation
• Using parallel computing techniques to speed up matrix operation

• 前一个Layer的Output当作Feature

• Softmax归一化

Example Application

• Handwriting Digit Recognition

• 判断一下属于每个数字的概率

• Input：256-dimension vector
• Output：10-dimension vector
• need a function，this function is Neural Network

• Layer是自己设计的，要有几个Layer，每个Layer有几个Neuron，就决定了Function Set长什么样子

• Q：How many layers？How many neurons for each layer?
• A：Trial and Error + Intuition
• Q：Can the structure be automatically determined
• A：E.g. Evolutiuonary Artificial Neural Networks
• Q：Can we design the network structure
• A：Convolutional Neural Network(CNN)卷积神经网络

Step2：goodness of function

• Loss for an Example
• Cross Entropy：交叉熵，不断地调整参数，使Cross Entropy越小越好

• Total Loss

• 利用Gradient Descent找到一组好的参数

• Backpropagation：an efficient way to compute 微分 in neural network

• more parameters,better performance是很正常的，more parameters cover function set更广