强化学习（四）—— Actor-Critic_actor-critic网络架构

1. 网络结构

• 状态价值函数：
  $$V_{\pi }(s_t)=\sum _a{Q}_{\pi }(s_t,a)\cdot \pi (a|s_t)$$

• 通过策略网络近似策略函数:
  $$\pi (a|s)\approx \pi (a|s;\theta )$$
  

• 通过价值网络近似动作价值函数:
  $$q(s,a;W)\approx Q(s,a)$$
  

• 神经网络近似后的状态价值函数:
  $$V(s;\theta ,W)=\sum _{a}q(s,a;W)\ast \pi (a|s;\theta )$$

• 通过对策略网络不断更新以增加状态价值函数值。

• 通过对价值网络不断更新来更好的预测所获得的回报。

2. 网络函数

Policy Network

• 通过策略网络近似策略函数
  $$\pi (a|s_t)\approx \pi (a|s_t;\theta )$$
• 状态价值函数及其近似
  $$V_{\pi }(s_t)=\sum _a\pi (a|s_t)Q_{\pi }(s_t,a)$$
  $$V(s_t;\theta )=\sum _a\pi (a|s_t;\theta )\cdot Q_{\pi }(s_t,a)$$
• 策略学习最大化的目标函数
  $$J(\theta )=E_S[V(S;\theta )]$$
• 依据策略梯度上升进行
  $$\theta \leftarrow \theta +\beta \cdot \frac{\partial V(s;\theta )}{\partial \theta }$$

3. 策略网络的更新-策略梯度

Policy Network

• 策略梯度为：
  $$\begin{gathered} g(a,\theta )=\frac{\partial ln\pi (a|s;\theta )}{\partial \theta }\cdot q(s,a;W) \\ \frac{\partial V(s;\theta ,W)}{\partial \theta }=E[g(A,\theta )] \end{gathered}$$
• 可采用随机策略梯度，（无偏估计）
  $$\begin{gathered} a\sim \pi (\cdot |s_t;\theta ) \\ {\theta }_{t+1}={\theta }_t+\beta \cdot g(a,{\theta }_t) \end{gathered}$$

4. 价值网络的更新-时序差分（TD）

• TD的目标：
  $$y_t=r_t+\gamma q(s_{t+1},a_{t+1};W_t)$$
• 损失函数为：
  $$loss=\frac{1}{2}[q(s_t,a_t;W_t)-y_t{]}^2$$
• 采用梯度下降进行更新：
  $$W_{t+1}=W_t-\alpha \cdot \frac{\partial loss}{\partial W}{|}_{W=W_t}$$

5. 网络训练流程

一次更新中，Agent执行一次动作，获得一次奖励。

1. 获得状态s_t并随机采样动作：$$a_t\sim \pi (\cdot |s_t;\theta )$$
2. Agent执行动作，并获得环境的新状态和奖励：$$\begin{gathered} s_{t+1} \\ {r}_t \end{gathered}$$
3. 依据新状态再次随机采样动作（该动作在本次迭代中并不执行）：$${\tilde{a}}_{t+1}\sim \pi (\cdot |s_{t+1};\theta )$$
4. 依据价值网络，分别计算两个动作和状态的价值：$$\begin{gathered} q_t=q(s_t,a_t;W_t) \\ {q}_{t+1}=q(s_{t+1},{\tilde{a}}_{t+1};W_t) \end{gathered}$$
5. 计算TD误差：$${\delta }_t=q_t-(r_t+\gamma q_{t+1})$$
6. 计算价值网络的导数：$$d_{W,t}=\frac{\partial q(s_t,a_t;W)}{\partial W}{|}_{W=W_t}$$
7. 对价值网络进行梯度更新：$$W_{t+1}=W_t-\alpha \cdot {\delta }_t\cdot d_{W,t}$$
8. 计算策略网络的梯度：$$d_{\theta ,t}=\frac{\partial ln[\pi (\cdot |s_t;\theta )]}{\partial \theta }{|}_{\theta ={\theta }_t}$$
9. 对策略网络进行梯度更新,式（2）为式（1）对动作价值函数值使用了baseline，目标函数的期望一致，但是方差减小，网络更容易收敛。$$\begin{gathered} {\theta }_{t+1}=\theta +\beta \cdot q_t\cdot d_{\theta ,t} \\ {\theta }_{t+1}=\theta +\beta \cdot {\delta }_t\cdot d_{\theta ,t} \end{gathered}$$

6. 案例

该网络的收敛对于模型大小、激活函数等参数较敏感。

# -*- coding: utf-8 -*-
# @Time : 2022/3/29 21:51
# @Author : CyrusMay WJ
# @FileName: AC.py
# @Software: PyCharm
# @Blog ：https://blog.csdn.net/Cyrus_May

import tensorflow as tf
import numpy as np
import logging
import sys
import gym


class Critic():
    def __init__(self,logger=None,input_dim=6,gamma=0.9):
        self.logger = logger
        self.__build_model(input_dim)
        self.gamma = gamma
        self.optimizer = tf.optimizers.Adam(learning_rate=0.001)

    def __build_model(self,input_dim):
        self.model = tf.keras.Sequential([
            tf.keras.layers.Dense(32, activation="relu"),
            tf.keras.layers.Dense(1)
        ])
        self.model.build(input_shape=[None,input_dim])

    def predict(self,action,state):
        action = tf.one_hot([action],depth=2)
        state = tf.convert_to_tensor([state])
        x = tf.concat([action,state],axis=1)
        return self.model(x)[0][0]

    def train(self,state,state_,action,action_,reward,done):
        action = tf.one_hot([action], depth=2)
        state = tf.convert_to_tensor([state])
        action_ = tf.one_hot([action_], depth=2)
        state_ = tf.convert_to_tensor([state_])
        x = tf.concat([action, state], axis=1)
        x_ = tf.concat([action_, state_], axis=1)
        done = 0 if done else 1
        with tf.GradientTape() as tape:
            q = self.model(x)
            q_ = self.model(x_)
            Td_error = (reward + done * self.gamma * q_ - q)
            loss = tf.square(Td_error)
            dt = tape.gradient(loss,self.model.trainable_variables)
        self.optimizer.apply_gradients(zip(dt,self.model.trainable_variables))
        return Td_error


class Actor():
    def __init__(self,logger=None,input_dim=4,gamma=0.9,output_dim=2):
        self.logger = logger
        self.__build_model(input_dim,output_dim)
        self.gamma = gamma
        self.optimizer = tf.optimizers.Adam(learning_rate=0.001)
        self.output_dim = output_dim

    def __build_model(self,input_dim,output_dim=2):
        self.model = tf.keras.Sequential([
            tf.keras.layers.Dense(32, activation="relu"),
            tf.keras.layers.Dense(output_dim)
        ])
        self.model.build(input_shape=[None,input_dim])
    def predict(self,state):
        state = tf.convert_to_tensor([state])
        logits = self.model(state)
        prob = tf.nn.softmax(logits).numpy()
        action = np.random.choice([i for i in range(self.output_dim)],p=prob.ravel())
        return action

    def train(self,state,action,TD_error,done):
        state = tf.convert_to_tensor([state])
        with tf.GradientTape() as tape:
            logits = self.model(state)
            loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels = [action], logits=logits)
            loss = tf.reduce_sum(tf.multiply(TD_error,loss))
            dt = tape.gradient(loss,self.model.trainable_variables)
            self.optimizer.apply_gradients(zip(dt,self.model.trainable_variables))

class Agent():
    def __init__(self,gamma=0.9,logger=None):
        self.gamma = gamma
        self.logger = logger
        self.env = gym.make("CartPole-v0")
        self.actor = Actor(logger=logger,input_dim=4,gamma=self.gamma,output_dim=2)
        self.critic = Critic(logger = logger,input_dim=6,gamma=self.gamma)

    def train(self,tran_epochs=1000,max_act=100):
        history_returns = []
        for epoch in range(tran_epochs):
            single_returns = 0
            state = self.env.reset()
            for iter in range(max_act):
                self.env.render()
                action = self.actor.predict(state)
                state_,reward,done,info = self.env.step(action)
                action_  = self.actor.predict(state_)
                TD_error = self.critic.train(state,state_,action,action_,reward,done)
                self.actor.train(state,action,TD_error,done)
                single_returns+=(reward)
                state = state_
                if done:
                    break
            if history_returns:
                history_returns.append(history_returns[-1]*0.9+0.1*single_returns)
            else:
                history_returns.append( single_returns)
            self.logger.info("epoch:{}\{} || epoch return:{:,.4f} || history return:{:,.4f}".format(tran_epochs,epoch+1,single_returns,history_returns[-1]))
        self.env.close()



    def test(self,max_act=1000):
        state = self.env.reset()
        single_returns = 0
        for iter in range(max_act):
            self.env.render()
            action = self.actor.predict(state)
            state_, reward, done, info = self.env.step(action)
            single_returns += (reward)
            if done:
                self.logger.info("End in {} iterations".format(iter+1))
                break
        if not done:
            self.logger.info("success and return is {}".format(single_returns))


if __name__ == '__main__':
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)
    screen_handler = logging.StreamHandler(sys.stdout)
    screen_handler.setLevel(logging.INFO)
    formatter = logging.Formatter('%(asctime)s - %(module)s.%(funcName)s:%(lineno)d - %(levelname)s - %(message)s')
    screen_handler.setFormatter(formatter)
    logger.addHandler(screen_handler)

    agent = Agent(logger=logger)
    agent.train(tran_epochs=2000,max_act=500)
    agent.test()



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145

本文部分内容为参考B站学习视频书写的笔记！

by CyrusMay 2022 03 29

摸不到的颜色 是否叫彩虹
看不到的拥抱 是否叫做微风
————五月天（星空）————