强化学习之PPO_ppo优势函数 - wpsshop博客

查看

编辑修改

首页
UNITY
NODEJS
PYTHON
AI
GIT
PHP
GO
CEF3
JAVA
HTML
CSS

知新_RL

这个屌丝很懒，什么也没留下！

热门标签

jquery
HTML
CSS
PHP
ASP
PYTHON
GO
AI
C
C++
C#
PHOTOSHOP
UNITY
iOS
android
vue
xml
爬虫
SEO
LINUX
WINDOWS
JAVA
MFC
CEF3
CAD
NODEJS
GIT
Pyppeteer
article

热门文章

1手把手系列！使用 Zilliz Cloud 和 AWS Bedrock 搭建 RAG 应用
2kafka-消费者组-点对点测试
3华为OD机试 - 生成哈夫曼树（Java & JS & Python & C & C++）_生成哈夫曼树华为od
4计算机专业近几年的参考文献,近几年计算机专业发参考文献计算机专业发专著类参考文献哪里找...
5人工智能轨道交通行业周刊-第29期（2023.1.2-1.8）_昆明南站小红车机器人
6清览云题库--数据可视化
7消息队列:RabbitMQ_消息队列rabbitmq
8小华最多能得到多少克黄金【华为OD机试】(JAVA&Python&C++&JS题解)_小华按照地图去寻宝
9Makefile入门_makefile学习
10html标签“”中文本对齐实现，让人蛋疼的知识_html textarea跟前两行数据对齐</a></li></ul></div></div></div><link rel="stylesheet" href="https://cdn.wpsshop.cn/public/blog/css/mid_detail.css?v=890" type="text/css"><link rel="stylesheet" href="https://cdn.wpsshop.cn/public/blog/css/mid_clk.css?v=890" type="text/css"><script type="text/javascript" src="https://cdn.wpsshop.cn/public/blog/js/mid_.js?v=890"></script><link rel="stylesheet" href="https://cdn.wpsshop.cn/public/study/js/j/vs.css?v=890" type="text/css"><script type="text/javascript" src="https://cdn.wpsshop.cn/public/study/js/j/highlight.pack.js?v=890" ></script><link rel="stylesheet" href="https://cdn.wpsshop.cn/public/blog/css/index_contenthljs.css?v=890" type="text/css"><script type="text/javascript" src="https://cdn.wpsshop.cn/public/wpsshopindex/js/detail01tj.js?v=531890" ></script><link rel="stylesheet" href="https://cdn.wpsshop.cn/public/wpsshopindex/css/mid_detail_last.css?v=3890" type="text/css"><script type="text/javascript" src="https://cdn.wpsshop.cn/public/wpsshopindex/js/mid_detail_last.js?v=33890"></script><style type="text/css"></style><script type="text/javascript"></script><div class="interactive_info" id="body-data-0098" attrv="50130"></div><div class="article_box" style="overflow: hidden;"><div class="article_channel"><div class="channel_top_line"><div class="cn_t_l"></div><div class="cn_t_r"></div></div><div class="article_channel_info">当前位置: <a href="/article/list/1">article</a> > 正文 </div></div><div class="article_title"><h1>强化学习之PPO_ppo优势函数</h1></div><div class="article_info"><div class="article_c_info"> 作者：知新_RL | 2024-05-31 03:30:56</div><div class="article_click" style="width: 140px;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div class="article_intro">ppo优势函数</div><div class="article_content article_contentheight"> <link rel="stylesheet" href="https://csdnimg.cn/release/blogv2/dist/mdeditor/css/editerView/kdoc_html_views-1a98987dfd.css"> <link rel="stylesheet" href="https://csdnimg.cn/release/blogv2/dist/mdeditor/css/editerView/ck_htmledit_views-044f2cf1dc.css"> <div id="content_views" class="markdown_views prism-atom-one-dark"> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path> </svg> <p>阅读本文前先了解TRPO算法有助于理解，我对此也写过博客：https://blog.csdn.net/tianjuewudi/article/details/120191097</p> <p>参考李宏毅老师的视频：https://www.bilibili.com/video/BV1Wv411h7kN?p=80</p> <p>PPO，全名Proximal Policy Optimization，近端策略优化算法。</p> <p>PPO算法是一种新型的Policy Gradient算法，Policy Gradient算法对步长十分敏感，但是又难以选择合适的步长，在训练过程中新旧策略的的变化差异如果过大则不利于学习。PPO提出了新的目标函数可以再多个训练步骤实现小批量的更新，解决了Policy Gradient算法中步长难以确定的问题。其实TRPO也是为了解决这个思想但是直接求解TRPO这种带约束的问题是十分复杂的，他与PPO算法的效果差不多，但是PPO将KL散度作为惩罚项，更加容易求解。</p> <h3><a name="t0"></a><a id="PPO_8"></a>PPO</h3> <p>PPO也分为Actor的部分和Critic的部分。</p> <p>首先定义优势函数：<br> <span class="katex--display"><span class="katex-display"><span class="katex"><span class="katex-mathml"> A t ^ = ∑ t ′ > t γ t ′ − t r t ′ − V ϕ ( s t ) \hat{A_t} = \sum_{t'>t} \gamma^{t'-t}r_{t'} - V_{\phi}(s_t) </span><span class="katex-html"><span class="base"><span class="strut" style="height: 1.09677em; vertical-align: -0.15em;"></span><span class="mord accent"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.94677em;"><span class="" style="top: -3em;"><span class="pstrut" style="height: 3em;"></span><span class="mord"><span class="mord"><span class="mord mathdefault">A</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span></span></span><span class="" style="top: -3.25233em;"><span class="pstrut" style="height: 3em;"></span><span class="accent-body" style="left: -0.25em;">^</span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span><span class="mspace" style="margin-right: 0.277778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right: 0.277778em;"></span></span><span class="base"><span class="strut" style="height: 2.37136em; vertical-align: -1.32136em;"></span><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 1.05001em;"><span class="" style="top: -1.85601em; margin-left: 0em;"><span class="pstrut" style="height: 3.05em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.682829em;"><span class="" style="top: -2.786em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span><span class="mrel mtight">></span><span class="mord mathdefault mtight">t</span></span></span></span><span class="" style="top: -3.05em;"><span class="pstrut" style="height: 3.05em;"></span><span class=""><span class="mop op-symbol large-op">∑</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 1.32136em;"><span class=""></span></span></span></span></span><span class="mspace" style="margin-right: 0.166667em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.05556em;">γ</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.99248em;"><span class="" style="top: -3.113em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.827829em;"><span class="" style="top: -2.931em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span><span class="mbin mtight">−</span><span class="mord mathdefault mtight">t</span></span></span></span></span></span></span></span></span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.02778em;">r</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.32798em;"><span class="" style="top: -2.55em; margin-left: -0.02778em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.682829em;"><span class="" style="top: -2.786em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mspace" style="margin-right: 0.222222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right: 0.222222em;"></span></span><span class="base"><span class="strut" style="height: 1.03611em; vertical-align: -0.286108em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.22222em;">V</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.336108em;"><span class="" style="top: -2.55em; margin-left: -0.22222em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">ϕ</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.286108em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord"><span class="mord mathdefault">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mclose">)</span></span></span></span></span></span><br> 更新Actor的部分我们把奖励函数设置成：<br> <span class="katex--display"><span class="katex-display"><span class="katex"><span class="katex-mathml"> J P P O ( θ ) = ∑ t = 1 T π θ ( a t ∣ s t ) π θ o l d ( a t ∣ s t ) A ^ t − λ K L [ π θ o l d ∣ π θ ] J_{PPO}(\theta) = \sum_{t=1}^T \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}\hat{A}_t - \lambda KL[\pi_{\theta_{old}}|\pi_{\theta}] </span><span class="katex-html"><span class="base"><span class="strut" style="height: 1em; vertical-align: -0.25em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.09618em;">J</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.328331em;"><span class="" style="top: -2.55em; margin-left: -0.09618em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.13889em;">P</span><span class="mord mathdefault mtight" style="margin-right: 0.13889em;">P</span><span class="mord mathdefault mtight" style="margin-right: 0.02778em;">O</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathdefault" style="margin-right: 0.02778em;">θ</span><span class="mclose">)</span><span class="mspace" style="margin-right: 0.277778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right: 0.277778em;"></span></span><span class="base"><span class="strut" style="height: 3.09545em; vertical-align: -1.26711em;"></span><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 1.82834em;"><span class="" style="top: -1.88289em; margin-left: 0em;"><span class="pstrut" style="height: 3.05em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span class="" style="top: -3.05001em;"><span class="pstrut" style="height: 3.05em;"></span><span class=""><span class="mop op-symbol large-op">∑</span></span></span><span class="" style="top: -4.30001em; margin-left: 0em;"><span class="pstrut" style="height: 3.05em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight" style="margin-right: 0.13889em;">T</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 1.26711em;"><span class=""></span></span></span></span></span><span class="mspace" style="margin-right: 0.166667em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 1.427em;"><span class="" style="top: -2.314em;"><span class="pstrut" style="height: 3em;"></span><span class="mord"><span class="mord"><span class="mord mathdefault" style="margin-right: 0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.336108em;"><span class="" style="top: -2.55em; margin-left: -0.03588em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.02778em;">θ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.3448em;"><span class="" style="top: -2.34877em; margin-left: -0.02778em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">o</span><span class="mord mathdefault mtight" style="margin-right: 0.01968em;">l</span><span class="mord mathdefault mtight">d</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.151229em;"><span class=""></span></span></span></span></span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.25586em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord"><span class="mord mathdefault">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mord">∣</span><span class="mord"><span class="mord mathdefault">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mclose">)</span></span></span><span class="" style="top: -3.23em;"><span class="pstrut" style="height: 3em;"></span><span class="frac-line" style="border-bottom-width: 0.04em;"></span></span><span class="" style="top: -3.677em;"><span class="pstrut" style="height: 3em;"></span><span class="mord"><span class="mord"><span class="mord mathdefault" style="margin-right: 0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.336108em;"><span class="" style="top: -2.55em; margin-left: -0.03588em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.02778em;">θ</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord"><span class="mord mathdefault">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mord">∣</span><span class="mord"><span class="mord mathdefault">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mclose">)</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.94186em;"><span class=""></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mord"><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.94677em;"><span class="" style="top: -3em;"><span class="pstrut" style="height: 3em;"></span><span class="mord"><span class="mord mathdefault">A</span></span></span><span class="" style="top: -3.25233em;"><span class="pstrut" style="height: 3em;"></span><span class="accent-body" style="left: -0.11111em;">^</span></span></span></span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mspace" style="margin-right: 0.222222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right: 0.222222em;"></span></span><span class="base"><span class="strut" style="height: 1.00586em; vertical-align: -0.25586em;"></span><span class="mord mathdefault">λ</span><span class="mord mathdefault" style="margin-right: 0.07153em;">K</span><span class="mord mathdefault">L</span><span class="mopen">[</span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.336108em;"><span class="" style="top: -2.55em; margin-left: -0.03588em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.02778em;">θ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.3448em;"><span class="" style="top: -2.34877em; margin-left: -0.02778em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">o</span><span class="mord mathdefault mtight" style="margin-right: 0.01968em;">l</span><span class="mord mathdefault mtight">d</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.151229em;"><span class=""></span></span></span></span></span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.25586em;"><span class=""></span></span></span></span></span></span><span class="mord">∣</span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.336108em;"><span class="" style="top: -2.55em; margin-left: -0.03588em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.02778em;">θ</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mclose">]</span></span></span></span></span></span></p> <ul><li> <p>这里的A其实应该用新策略采样出来计算的Advantage函数，但由于参数改变的不多，我们可以用旧策略的A来近似新策略的A。</p> </li><li> <p>前面一项的设置是因为我们只能充旧策略中采样而不能从新策略中采样所以做了important Sampling。同时这样做可以把我们的策略从on-policy转变为off-policy，旧策略可以收集很多数据之后，就能用这些数据训练很多次的网络，然后再重新采样。</p> </li><li> <p>一开始新旧策略一样因此<span class="katex--inline"><span class="katex"><span class="katex-mathml"> π θ ( a t ∣ s t ) π θ o l d ( a t ∣ s t ) = 1 \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}=1 </span><span class="katex-html"><span class="base"><span class="strut" style="height: 1.63568em; vertical-align: -0.62568em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 1.01em;"><span class="" style="top: -2.655em;"><span class="pstrut" style="height: 3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.3448em;"><span class="" style="top: -2.34877em; margin-left: -0.03588em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.02778em;">θ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.3448em;"><span class="" style="top: -2.3448em; margin-left: -0.02778em; margin-right: 0.1em;"><span class="pstrut" style="height: 2.69444em;"></span><span class="mord mtight"><span class="mord mathdefault mtight">o</span><span class="mord mathdefault mtight" style="margin-right: 0.01968em;">l</span><span class="mord mathdefault mtight">d</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.34964em;"><span class=""></span></span></span></span></span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.400971em;"><span class=""></span></span></span></span></span></span><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathdefault mtight">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.296343em;"><span class="" style="top: -2.357em; margin-left: 0em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.143em;"><span class=""></span></span></span></span></span></span><span class="mord mtight">∣</span><span class="mord mtight"><span class="mord mathdefault mtight">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.296343em;"><span class="" style="top: -2.357em; margin-left: 0em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.143em;"><span class=""></span></span></span></span></span></span><span class="mclose mtight">)</span></span></span></span><span class="" style="top: -3.23em;"><span class="pstrut" style="height: 3em;"></span><span class="frac-line" style="border-bottom-width: 0.04em;"></span></span><span class="" style="top: -3.485em;"><span class="pstrut" style="height: 3em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.3448em;"><span class="" style="top: -2.34877em; margin-left: -0.03588em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.02778em;">θ</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.151229em;"><span class=""></span></span></span></span></span></span><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathdefault mtight">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.296343em;"><span class="" style="top: -2.357em; margin-left: 0em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.143em;"><span class=""></span></span></span></span></span></span><span class="mord mtight">∣</span><span class="mord mtight"><span class="mord mathdefault mtight">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.296343em;"><span class="" style="top: -2.357em; margin-left: 0em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.143em;"><span class=""></span></span></span></span></span></span><span class="mclose mtight">)</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.62568em;"><span class=""></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mspace" style="margin-right: 0.277778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right: 0.277778em;"></span></span><span class="base"><span class="strut" style="height: 0.64444em; vertical-align: 0em;"></span><span class="mord">1</span></span></span></span></span>，经过多次公式迭代后这个值就会越来越不一样。</p> </li><li> <p>这样当我们新策略的值远大于旧概率时，更新会快一些，但由于后面一项KL散度的存在，不会让概率分布差距过大的新旧策略更新太快，限制了新策略的更新幅度。</p> </li></ul> <p>而对于Critic部分，我们构建一个输出状态价值的网络，训练网络接近预定值：<br> <span class="katex--display"><span class="katex-display"><span class="katex"><span class="katex-mathml"> L B L ( ϕ ) = − ∑ t = 1 T ( ∑ t ′ > t γ t ′ − t r t ′ − V ϕ ( s t ) ) 2 L_{BL}(\phi) = -\sum_{t=1}^{T}(\sum_{t'>t} \gamma^{t'-t}r_{t'}-V_{\phi}(s_t))^2 </span><span class="katex-html"><span class="base"><span class="strut" style="height: 1em; vertical-align: -0.25em;"></span><span class="mord"><span class="mord mathdefault">L</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.328331em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.05017em;">B</span><span class="mord mathdefault mtight">L</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathdefault">ϕ</span><span class="mclose">)</span><span class="mspace" style="margin-right: 0.277778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right: 0.277778em;"></span></span><span class="base"><span class="strut" style="height: 3.14969em; vertical-align: -1.32136em;"></span><span class="mord">−</span><span class="mspace" style="margin-right: 0.166667em;"></span><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 1.82834em;"><span class="" style="top: -1.88289em; margin-left: 0em;"><span class="pstrut" style="height: 3.05em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span class="" style="top: -3.05001em;"><span class="pstrut" style="height: 3.05em;"></span><span class=""><span class="mop op-symbol large-op">∑</span></span></span><span class="" style="top: -4.30001em; margin-left: 0em;"><span class="pstrut" style="height: 3.05em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.13889em;">T</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 1.26711em;"><span class=""></span></span></span></span></span><span class="mopen">(</span><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 1.05001em;"><span class="" style="top: -1.85601em; margin-left: 0em;"><span class="pstrut" style="height: 3.05em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.682829em;"><span class="" style="top: -2.786em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span><span class="mrel mtight">></span><span class="mord mathdefault mtight">t</span></span></span></span><span class="" style="top: -3.05em;"><span class="pstrut" style="height: 3.05em;"></span><span class=""><span class="mop op-symbol large-op">∑</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 1.32136em;"><span class=""></span></span></span></span></span><span class="mspace" style="margin-right: 0.166667em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.05556em;">γ</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.99248em;"><span class="" style="top: -3.113em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.827829em;"><span class="" style="top: -2.931em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span><span class="mbin mtight">−</span><span class="mord mathdefault mtight">t</span></span></span></span></span></span></span></span></span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.02778em;">r</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.32798em;"><span class="" style="top: -2.55em; margin-left: -0.02778em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.682829em;"><span class="" style="top: -2.786em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mspace" style="margin-right: 0.222222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right: 0.222222em;"></span></span><span class="base"><span class="strut" style="height: 1.15022em; vertical-align: -0.286108em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.22222em;">V</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.336108em;"><span class="" style="top: -2.55em; margin-left: -0.22222em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">ϕ</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.286108em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord"><span class="mord mathdefault">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mclose">)</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.864108em;"><span class="" style="top: -3.113em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span></span></span><br> 通过梯度递减来训练这个Loss为0.</p> <p><img src="https://img-blog.csdnimg.cn/83f2cf2059154f14ac0004bb48df0ddf.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBA5b6u56yR5bCP5pif,size_20,color_FFFFFF,t_70,g_se,x_16#pic_center" alt="在这里插入图片描述"></p> <h4><a name="t1"></a><a id="PPO_with_Adaptive_KL_Penalty_37"></a><strong>PPO with Adaptive KL Penalty</strong></h4> <p>我们还发现KL散度还要乘以一个变量，该版本PPO的核心想法就是利用自适应的beta值（adaptive beta）：</p> <p><img src="https://img-blog.csdnimg.cn/c59820445b3448a08ed6d4c043d7ac9f.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBA5b6u56yR5bCP5pif,size_20,color_FFFFFF,t_70,g_se,x_16#pic_center" alt="在这里插入图片描述"></p> <p>当KL散度大于我们的设定值时，我们需要加大处罚力度，而小于一个设定值时，减小处罚力度，这样使得新旧策略的差距既不会偏大，也不会过小。</p> <h4><a name="t2"></a><a id="PPO_with_Clipped_Objective_48"></a><strong>PPO with Clipped Objective</strong></h4> <p>这是基于上面PPO的改进，相比于用KL散度来限制策略的更新幅度，我们不如直接限制新旧策略的比例来实现防止其差距过大。这样还减少了KL散度的运算，提升了算法性能。<br> <span class="katex--display"><span class="katex-display"><span class="katex"><span class="katex-mathml"> L C L I P ( θ ) = E t ^ [ m i n ( r t ( θ ) A t ^ , c l i p ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A t ^ ) ] L^{CLIP}(\theta) = \hat{E_t}[min(r_t(\theta)\hat{A_t},clip(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A_t})] </span><span class="katex-html"><span class="base"><span class="strut" style="height: 1.14133em; vertical-align: -0.25em;"></span><span class="mord"><span class="mord mathdefault">L</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height: 0.891331em;"><span class="" style="top: -3.113em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.07153em;">C</span><span class="mord mathdefault mtight">L</span><span class="mord mathdefault mtight" style="margin-right: 0.07847em;">I</span><span class="mord mathdefault mtight" style="margin-right: 0.13889em;">P</span></span></span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathdefault" style="margin-right: 0.02778em;">θ</span><span class="mclose">)</span><span class="mspace" style="margin-right: 0.277778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right: 0.277778em;"></span></span><span class="base"><span class="strut" style="height: 1.19677em; vertical-align: -0.25em;"></span><span class="mord accent"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.94677em;"><span class="" style="top: -3em;"><span class="pstrut" style="height: 3em;"></span><span class="mord"><span class="mord"><span class="mord mathdefault" style="margin-right: 0.05764em;">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: -0.05764em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span></span></span><span class="" style="top: -3.25233em;"><span class="pstrut" style="height: 3em;"></span><span class="accent-body" style="left: -0.25em;">^</span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span><span class="mopen">[</span><span class="mord mathdefault">m</span><span class="mord mathdefault">i</span><span class="mord mathdefault">n</span><span class="mopen">(</span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.02778em;">r</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: -0.02778em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathdefault" style="margin-right: 0.02778em;">θ</span><span class="mclose">)</span><span class="mord accent"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.94677em;"><span class="" style="top: -3em;"><span class="pstrut" style="height: 3em;"></span><span class="mord"><span class="mord"><span class="mord mathdefault">A</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span></span></span><span class="" style="top: -3.25233em;"><span class="pstrut" style="height: 3em;"></span><span class="accent-body" style="left: -0.25em;">^</span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right: 0.166667em;"></span><span class="mord mathdefault">c</span><span class="mord mathdefault" style="margin-right: 0.01968em;">l</span><span class="mord mathdefault">i</span><span class="mord mathdefault">p</span><span class="mopen">(</span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.02778em;">r</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: -0.02778em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathdefault" style="margin-right: 0.02778em;">θ</span><span class="mclose">)</span><span class="mpunct">,</span><span class="mspace" style="margin-right: 0.166667em;"></span><span class="mord">1</span><span class="mspace" style="margin-right: 0.222222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right: 0.222222em;"></span></span><span class="base"><span class="strut" style="height: 0.83888em; vertical-align: -0.19444em;"></span><span class="mord mathdefault">ϵ</span><span class="mpunct">,</span><span class="mspace" style="margin-right: 0.166667em;"></span><span class="mord">1</span><span class="mspace" style="margin-right: 0.222222em;"></span><span class="mbin">+</span><span class="mspace" style="margin-right: 0.222222em;"></span></span><span class="base"><span class="strut" style="height: 1.19677em; vertical-align: -0.25em;"></span><span class="mord mathdefault">ϵ</span><span class="mclose">)</span><span class="mord accent"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.94677em;"><span class="" style="top: -3em;"><span class="pstrut" style="height: 3em;"></span><span class="mord"><span class="mord"><span class="mord mathdefault">A</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span></span></span><span class="" style="top: -3.25233em;"><span class="pstrut" style="height: 3em;"></span><span class="accent-body" style="left: -0.25em;">^</span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span><span class="mclose">)</span><span class="mclose">]</span></span></span></span></span></span></p> <p><span class="katex--display"><span class="katex-display"><span class="katex"><span class="katex-mathml"> r t ( θ ) = π θ ( a t ∣ s t ) π θ o l d ( a t ∣ s t ) r_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)} </span><span class="katex-html"><span class="base"><span class="strut" style="height: 1em; vertical-align: -0.25em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right: 0.02778em;">r</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: -0.02778em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathdefault" style="margin-right: 0.02778em;">θ</span><span class="mclose">)</span><span class="mspace" style="margin-right: 0.277778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right: 0.277778em;"></span></span><span class="base"><span class="strut" style="height: 2.36886em; vertical-align: -0.94186em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 1.427em;"><span class="" style="top: -2.314em;"><span class="pstrut" style="height: 3em;"></span><span class="mord"><span class="mord"><span class="mord mathdefault" style="margin-right: 0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.336108em;"><span class="" style="top: -2.55em; margin-left: -0.03588em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.02778em;">θ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.3448em;"><span class="" style="top: -2.34877em; margin-left: -0.02778em; margin-right: 0.0714286em;"><span class="pstrut" style="height: 2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">o</span><span class="mord mathdefault mtight" style="margin-right: 0.01968em;">l</span><span class="mord mathdefault mtight">d</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.151229em;"><span class=""></span></span></span></span></span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.25586em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord"><span class="mord mathdefault">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mord">∣</span><span class="mord"><span class="mord mathdefault">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mclose">)</span></span></span><span class="" style="top: -3.23em;"><span class="pstrut" style="height: 3em;"></span><span class="frac-line" style="border-bottom-width: 0.04em;"></span></span><span class="" style="top: -3.677em;"><span class="pstrut" style="height: 3em;"></span><span class="mord"><span class="mord"><span class="mord mathdefault" style="margin-right: 0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.336108em;"><span class="" style="top: -2.55em; margin-left: -0.03588em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right: 0.02778em;">θ</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mopen">(</span><span class="mord"><span class="mord mathdefault">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mord">∣</span><span class="mord"><span class="mord mathdefault">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height: 0.280556em;"><span class="" style="top: -2.55em; margin-left: 0em; margin-right: 0.05em;"><span class="pstrut" style="height: 2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.15em;"><span class=""></span></span></span></span></span></span><span class="mclose">)</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height: 0.94186em;"><span class=""></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span></span></span></p> <p>这里我们把新旧策略的比例限制在1-ε和1+ε直接，说白了就是在超过这个范围时把新策略直接设定为边界值。</p> <p><img src="https://img-blog.csdnimg.cn/ad414f62eb3c4d59b1c2059fd3652680.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBA5b6u56yR5bCP5pif,size_20,color_FFFFFF,t_70,g_se,x_16#pic_center" alt="在这里插入图片描述"></p> <p>在A为正数的情况下，我们倾向于采取这种行为，因此新策略该行为的概率会慢慢增大，但是又不能大于我们的界限。在A为负数的情况下，说明这时我们的上一次更新的梯度方向相反了，我们倾向于摒弃这种行为，因此我们要加大更新力度去弥补上一次的失败，为了反转这个优势函数为负数的趋势，因此设置了一个下限而没有设置上限。</p> </div><div data-report-view="{"mod":"1585297308_001","spm":"1001.2101.3001.6548","dest":"https://blog.csdn.net/tianjuewudi/article/details/120212234","extend1":"pc","ab":"new"}"><div></div></div> <link href="https://csdnimg.cn/release/blogv2/dist/mdeditor/css/editerView/markdown_views-f23dff6052.css" rel="stylesheet"> <link href="https://csdnimg.cn/release/blogv2/dist/mdeditor/css/style-c216769e99.css" rel="stylesheet"> <div><div><span>声明：</span>本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：<a href="https://www.wpsshop.cn/w/知新_RL/article/detail/650130">https://www.wpsshop.cn/w/知新_RL/article/detail/650130</a></div></div></div><div class="article_recommend"><div class="list_top">推荐阅读</div><div class="ListItems"><ul class="NewsList"><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="One PUNCH Man——半监督学习_“基于分歧的方法”" href="/w/不正经/article/detail/649145" target="_blank">One <em>PUNCH</em> Man——半<em>监督</em><em>学习</em>_“<em>基于</em><em>分歧</em><em>的</em><em>方法</em>”...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="One PUNCH Man——半监督学习_“基于分歧的方法”" href="/w/不正经/article/detail/649145" target="_blank">文章目录半<em>监督</em><em>学习</em>介绍半<em>监督</em>SVM<em>基于</em><em>分歧</em><em>的</em><em>方法</em>半<em>监督</em>聚类半<em>监督</em><em>学习</em>这一块<em>的</em>算法原理，由于有<em>的</em>内容大学数学没学到<em>的</em>缘故，... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="面试时如何做好5分钟自我介绍？_入职面试自我介绍必看" href="/w/我家小花儿/article/detail/649137" target="_blank"><em>面试</em>时如何做好5<em>分钟</em><em>自我介绍</em><em>？</em><em>_</em><em>入职</em><em>面试</em><em>自我介绍</em>必看...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="面试时如何做好5分钟自我介绍？_入职面试自我介绍必看" href="/w/我家小花儿/article/detail/649137" target="_blank">一个常规的<em>面试</em>，寒暄之后<em>面试</em>官提出的第一个问题几乎千篇一律：“请您简单地做一下<em>自我介绍</em>”。有些被<em>面试</em>者都会问：简历中情况... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="JAVA多线程-线程安全问题_java多线程安全" href="/w/我家小花儿/article/detail/650131" target="_blank"><em>JAVA</em>多<em>线程</em>-<em>线程</em><em>安全</em>问题_<em>java</em>多<em>线程</em><em>安全</em>...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="JAVA多线程-线程安全问题_java多线程安全" href="/w/我家小花儿/article/detail/650131" target="_blank">一、CPU多核缓存架构CPU分为三级缓存：每个CPU都有L1,L2缓存，但是L3缓存是多核公用的。CPU查找数据的顺序... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="todesk一直显示连接本地服务问题_todesk连接本地服务" href="/w/从前慢现在也慢/article/detail/650035" target="_blank"><em>todesk</em>一直显示<em>连接</em><em>本地</em><em>服务</em>问题_<em>todesk</em><em>连接</em><em>本地</em><em>服务</em>...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="todesk一直显示连接本地服务问题_todesk连接本地服务" href="/w/从前慢现在也慢/article/detail/650035" target="_blank">关闭<em>todesk</em>终端输入 sudo systemctl restart <em>todesk</em>d.service重新打开todes... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="python解释器pycharm安装及环境变量配置教程图文详解_pycharm的numpy环境变量" href="/w/凡人多烦事01/article/detail/649921" target="_blank"><em>python</em><em>解释器</em><em>pycharm</em><em>安装</em>及<em>环境变量</em>配置教程图文详解_<em>pycharm</em>的<em>numpy</em><em>环境变量</em>...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="python解释器pycharm安装及环境变量配置教程图文详解_pycharm的numpy环境变量" href="/w/凡人多烦事01/article/detail/649921" target="_blank">@本文来源于公众号：csdn2299，喜欢可以关注公众号程序员学府文章目录1.<em>python</em><em>解释器</em><em>安装</em>2.配置<em>环境变量</em>3... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="中国AI开发者真实现状：写代码这条路，会走多久？" href="/w/2023面试高手/article/detail/649591" target="_blank"><em>中国</em><em>AI</em><em>开发者</em>真实现状<em>：</em>写代码这条路<em>，</em>会走多久？...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="中国AI开发者真实现状：写代码这条路，会走多久？" href="/w/2023面试高手/article/detail/649591" target="_blank">2016 年起<em>，</em>人工智能成为<em>中国</em><em>开发者</em>重点关注的技术领域<em>，</em>以深度学习驱动的计算机视觉、自然语言处理、语音相关技术成为渗透... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="Python python-barcode模块（生成条形码）_python barcode" href="/w/羊村懒王/article/detail/649439" target="_blank"><em>Python</em> <em>python</em>-<em>barcode</em>模块（<em>生成</em><em>条形码</em>）_<em>python</em> <em>barcode</em>...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="Python python-barcode模块（生成条形码）_python barcode" href="/w/羊村懒王/article/detail/649439" target="_blank"><em>Python</em> <em>python</em>-<em>barcode</em>模块（<em>生成</em><em>条形码</em>）一、安装<em>python</em>-<em>barcode</em>二、<em>生成</em><em>条形码</em>三种方法a... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="在Linux/Ubuntu/Debian上安装TensorFlow 2.14.0" href="/w/笔触狂放9/article/detail/649317" target="_blank">在<em>Linux</em>/<em>Ubuntu</em>/<em>Debian</em>上<em>安装</em><em>TensorFlow</em> <em>2.14</em>.0...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="在Linux/Ubuntu/Debian上安装TensorFlow 2.14.0" href="/w/笔触狂放9/article/detail/649317" target="_blank">在<em>Ubuntu</em>上<em>安装</em><em>TensorFlow</em> <em>2.14</em>.0在<em>Linux</em>/<em>Ubuntu</em>/<em>Debian</em>上<em>安装</em><em>TensorFlow</em>... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="使用UOS安装器安装deb包后直接启动界面程序_uos 用户登录后启动程序" href="/w/不正经/article/detail/649288" target="_blank">使用<em>UOS</em><em>安装</em>器<em>安装</em>deb包后直接<em>启动</em><em>界面</em><em>程序</em>_uos 用户登录后<em>启动</em><em>程序</em>...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="使用UOS安装器安装deb包后直接启动界面程序_uos 用户登录后启动程序" href="/w/不正经/article/detail/649288" target="_blank">最近开发了一款<em>界面</em><em>程序</em>，想实现<em>安装</em>完deb包后可以直接<em>启动</em><em>界面</em><em>程序</em>。我用dpkg -i 和 apt install 都可... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="Spark启动：WARN util.Utils: Your hostname, ... resolv" href="/w/IT小白/article/detail/649242" target="_blank"><em>Sp<em>a</em>rk</em>启动：<em>WARN</em> <em>util</em>.<em>Utils</em>: Your <em>hostn<em>a</em>me</em>, ... resolv...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="Spark启动：WARN util.Utils: Your hostname, ... resolv" href="/w/IT小白/article/detail/649242" target="_blank">完整警告输出如下：2021-12-09 23:49:14,625 <em>WARN</em> <em>util</em>.<em>Utils</em>: Your hostn... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="java多线程的安全_Java多线程安全问题" href="/w/从前慢现在也慢/article/detail/650121" target="_blank"><em>java</em><em>多线程</em>的<em>安全</em>_Java<em>多线程</em><em>安全</em><em>问题</em>...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="java多线程的安全_Java多线程安全问题" href="/w/从前慢现在也慢/article/detail/650121" target="_blank">线程<em>安全</em><em>问题</em>.因为线程的随机性，有可能会导致<em>多线程</em>在操作数据时发生数据错误的情况产生。线程<em>安全</em><em>问题</em>产生的原因：当线程中多... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="【Unity Shader】纹理实践3.0：切线空间下使用法线纹理_unity shader tbn" href="/w/小蓝xlanll/article/detail/649770" target="_blank">【<em>Unity</em> <em>Shader</em>】<em>纹理</em>实践3.0：<em>切线</em>空间下使用法线<em>纹理</em>_<em>unity</em> <em>shader</em> tbn...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="【Unity Shader】纹理实践3.0：切线空间下使用法线纹理_unity shader tbn" href="/w/小蓝xlanll/article/detail/649770" target="_blank">记录了法线<em>纹理</em>的使用过程，同时对在<em>切线</em>空间下进行光照计算的<em>Shader</em>代码编写过程做了一个相对详细的记录。_<em>unity</em> ... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="为什么说深耕AI领域绕不开知识图谱？" href="/w/2023面试高手/article/detail/649511" target="_blank"><em>为什么</em>说深耕<em>AI</em>领域绕不开<em>知识</em><em>图谱</em>？...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="为什么说深耕AI领域绕不开知识图谱？" href="/w/2023面试高手/article/detail/649511" target="_blank">所有在 <em>AI</em> 领域深耕的人，最终都会发现语义鸿沟仍是一个非常具有挑战性的问题，这最终还需要借助<em>知识</em><em>图谱</em>等技术，来帮助将整... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="Python使用opencv获取视频每帧的图片_python opencv获取视频每一帧的图片" href="/w/Cpp五条/article/detail/649173" target="_blank"><em>Python</em>使用<em>opencv</em><em>获取</em><em>视频</em>每帧的<em>图片</em>_<em>python</em> <em>opencv</em><em>获取</em><em>视频</em>每一帧的<em>图片</em>...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="Python使用opencv获取视频每帧的图片_python opencv获取视频每一帧的图片" href="/w/Cpp五条/article/detail/649173" target="_blank"><em>Python</em>使用<em>opencv</em><em>获取</em><em>视频</em>每帧的<em>图片</em>import osimport cv2cap = cv2.VideoCap... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="UE4 后期材质节点学习_ue4饱和度节点" href="/w/Cpp五条/article/detail/649847" target="_blank"><em>UE4</em> 后期材质<em>节点</em>学习_<em>ue4</em><em>饱和度</em><em>节点</em>...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="UE4 后期材质节点学习_ue4饱和度节点" href="/w/Cpp五条/article/detail/649847" target="_blank">因为SceneTexture中的Color是一个Vector4，3维和4维是不能相乘，所以Multiply报错。Frac... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="大数据技术的知识图谱_大数据知识图谱" href="/w/小蓝xlanll/article/detail/649958" target="_blank"><em>大</em><em>数据</em><em>技术</em>的<em>知识</em>图谱_<em>大</em><em>数据</em><em>知识</em>图谱...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="大数据技术的知识图谱_大数据知识图谱" href="/w/小蓝xlanll/article/detail/649958" target="_blank">当今及将来很长的時期内，互联网<em>大</em><em>数据</em>将变成开发设计和运用的重中之重。对比于别的行业的互联网<em>大</em><em>数据</em>，比如交通出行互联网<em>大</em>数... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="基于Keras、DenseNet模型微调、参数冻结、数据增强、模型训练、模型验证全流程记录（模型微调" href="/w/不正经/article/detail/649596" target="_blank">基于<em>Keras</em>、<em>DenseNet</em><em>模型</em>微调、<em>参数</em><em>冻结</em>、数据增强、<em>模型</em><em>训练</em>、<em>模型</em>验证全流程记录（<em>模型</em>微调...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="基于Keras、DenseNet模型微调、参数冻结、数据增强、模型训练、模型验证全流程记录（模型微调" href="/w/不正经/article/detail/649596" target="_blank">使用keras搭建<em>模型</em>，用imagenet的权重进行预<em>训练</em>。<em>densenet</em>169的layers数量未595，<em>冻结</em><em>模型</em>前... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="Python中opencv的使用" href="/w/盐析白兔/article/detail/649228" target="_blank"><em>Python</em>中<em>opencv</em>的使用...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="Python中opencv的使用" href="/w/盐析白兔/article/detail/649228" target="_blank">1 <em>Python</em>的<em>opencv</em>安装OpenCV-<em>Python</em> Tutorials：https://docs.<em>opencv</em>... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="数据研发学习笔记4.4：大数据之NoSQL数据库_实验4 nosql和关系数据库的操作比较" href="/w/Monodyee/article/detail/649710" target="_blank"><em>数据</em>研发学习笔记4.4：大<em>数据</em>之<em>NoSQL</em><em><em>数据</em>库</em>_实验4 <em>nosql</em>和关系<em><em>数据</em>库</em>的<em>操作</em>比较...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="数据研发学习笔记4.4：大数据之NoSQL数据库_实验4 nosql和关系数据库的操作比较" href="/w/Monodyee/article/detail/649710" target="_blank">文章目录1 <em>NoSQL</em>简介2 <em>NoSQL</em>兴起的原因3 <em>NoSQL</em>与关系<em><em>数据</em>库</em>的比较4 <em>NoSQL</em>的四大类型4.1 键值数... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li><li><div class="NewTitle"><a title="article" class="cat" href="/article/list/1" target="_blank">article<i></i></a><a title="One PUNCH Man——线性回归算法_说明变量目的变量" href="/w/我家自动化/article/detail/649218" target="_blank">One <em>PUNCH</em> Man——<em>线性</em><em>回归</em>算法_<em>说明</em><em>变量</em> <em>目的</em><em>变量</em>...</a></div><div class="NewsInfo"><div class="NewsDesc" style=""><a title="One PUNCH Man——线性回归算法_说明变量目的变量" href="/w/我家自动化/article/detail/649218" target="_blank">文章目录<em>回归</em>分析的<em>目的</em><em>线性</em><em>回归</em>步骤一元<em>线性</em><em>回归</em>操作和解释<em>回归</em>分析的<em>目的</em><em>回归</em>分析的<em>目的</em>大致可分为两种：第一，预测。预测目标... </a><div class="article_click rice1" style="width: 240px;float: left;"><p class="operation-b-img operation-b-img-active"><i class="img-up txclick" attc="upclick" attn="0"></i><span class="num"> 赞</span></p><p class="operation-b-img operation-b-img-active"><i class="img-down txclick" attc="downclick" attn="0"></i><span class="num">踩</span></p></div></div><div style="clear: both;"></div></div></li></ul><div class="list_tools_top">相关标签</div><div class="list_tools_box"><ul><li><a title="一拳超人从不秃头" rel="nofollow" href="/s?w=一拳超人从不秃头" target="_self">一拳超人从不秃头</a></li><li><a title="java" rel="nofollow" href="/s?w=java" target="_self">java</a></li><li><a title="安全" rel="nofollow" href="/s?w=安全" target="_self">安全</a></li><li><a title="缓存" rel="nofollow" href="/s?w=缓存" target="_self">缓存</a></li><li><a title="经验分享" rel="nofollow" href="/s?w=经验分享" target="_self">经验分享</a></li><li><a title="python" rel="nofollow" href="/s?w=python" target="_self">python</a></li><li><a title="windows" rel="nofollow" href="/s?w=windows" target="_self">windows</a></li><li><a title="linux" rel="nofollow" href="/s?w=linux" target="_self">linux</a></li><li><a title="ubuntu" rel="nofollow" href="/s?w=ubuntu" target="_self">ubuntu</a></li><li><a title="debian" rel="nofollow" href="/s?w=debian" target="_self">debian</a></li><li><a title="spark" rel="nofollow" href="/s?w=spark" target="_self">spark</a></li><li><a title="大数据" rel="nofollow" href="/s?w=大数据" target="_self">大数据</a></li><li><a title="java多线程的安全" rel="nofollow" href="/s?w=java多线程的安全" target="_self">java多线程的安全</a></li><li><a title="unity" rel="nofollow" href="/s?w=unity" target="_self">unity</a></li><li><a title="游戏" rel="nofollow" href="/s?w=游戏" target="_self">游戏</a></li><li><a title="技术美术" rel="nofollow" href="/s?w=技术美术" target="_self">技术美术</a></li><li><a title="ue4" rel="nofollow" href="/s?w=ue4" target="_self">ue4</a></li><li><a title="材质" rel="nofollow" href="/s?w=材质" target="_self">材质</a></li><li><a title="学习" rel="nofollow" href="/s?w=学习" target="_self">学习</a></li><li><a title="大数据开发" rel="nofollow" href="/s?w=大数据开发" target="_self">大数据开发</a></li><li><a title="大数据学习" rel="nofollow" href="/s?w=大数据学习" target="_self">大数据学习</a></li><li><a title="大数据知识" rel="nofollow" href="/s?w=大数据知识" target="_self">大数据知识</a></li><div style="clear: both;"></div></ul></div><div class="list_tools_top"></div></div></div></div><style type="text/css"></style><link rel="stylesheet" href="https://cdn.wpsshop.cn/public/blog/css/phone.css?v=17352890" type="text/css"><script type="text/javascript"></script></div></div></div></div></div></div><style type="text/css"> /* footer css */ .index006-cover-page-foot { background-color: #F7F7F7; } </style><div class="cppui-row-1_100 cppui-row-view-1 row_1_100 ui-draggable" style="display: block;"><div class="cppui-column-1_100-0-1 column-view-com cppui-column-1_100-0-view-1 ui-sortable"><div class="drag-item-show-view-1 footermsg_view_pic ui-draggable" style="display: block;"><div class="index006-cover-page-foot drag-item-hide-view-1"><p class="copyright">Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。 </p><div style="background-color:#F7F7F7; text-align:center; height:20px;padding-top:5px;"><a target="_blank" href="#" style="display:inline-block;text-decoration:none;height:20px;line-height:20px;"><img src="https://cdnimages.wpsshop.cn/57public/skin/index/default/ui/images/beian_ghs.png" style="float:left;"> <p style="float:left;height:20px;line-height:20px;margin: 0px 0px 0px 5px; color:#939393;">闽ICP备14008679号</p></a> <a href="https://www.wpsshop.cn/xml/w/%E7%9F%A5%E6%96%B0_RL/article/detail/650130.xml" target="_blank" style="display:inline-block;text-decoration:none;height:20px;line-height:20px;"><img src="https://cdn.wpsshop.cn/public/blog/images/site.png" style="float:left;"></a> <a href="/xml/w/g007/article/detail/new.xml" target="_blank" style="display:inline-block;text-decoration:none;height:20px;line-height:20px;"><img src="https://cdn.wpsshop.cn/public/blog/images/site.png" style="float:left;"></a> <a href="/site.xml" target="_blank" style="display:inline-block;text-decoration:none;height:20px;line-height:20px;"><img src="https://cdn.wpsshop.cn/public/blog/images/site.png" style="float:left;"></a></div></div></div></div></div><script type="text/javascript">var domain = document.domain;var hr=encodeURIComponent(window.location.href+'=='+document.referrer); document.write('<script src="https://ad.wpsshop.cn/admin.appflux?s=ad_flu_ind&fluxuserauto=yes&hr='+hr+'&p57weburl='+domain+'&p57usercode1='+domain+'&u=Vy1RJQhqAXUEMQNnAm0HaVVu&i='+p57ref("id")+'&r='+escape(document.referrer)+'" language="JavaScript"><\/script>');function p57ref(id){var reg=new RegExp("(^|&)"+id+"=([^&]*)(&|$)");var ref=window.location.search.substr(1).match(reg);if(ref!=null)return unescape(ref[2]);return null;} </script></body></html>