devbox
不正经
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

计算卸载论文阅读01-理论梳理_计算卸载会议

作者：不正经 | 2024-03-23 15:21:32

赞

踩

计算卸载会议

标题：When Learning Joins Edge: Real-time Proportional Computation Offloading via Deep Reinforcement Learning

会议：ICPADS 2019

一、梳理

问题：在任务进行卸载时，往往忽略了任务的特定的卸载比例。

模型：针对上述问题，我们提出了一种创新的强化学习（RL）方法来解决比例计算问题。我们考虑了一种常见的卸载场景，该场景具有时变带宽和异构设备，并且设备不断生成应用程序。对于每个应用程序，客户端必须选择本地或远程执行该应用程序，并确定要卸载的比例。我们将该问题制定为一个长期优化问题，然后提出一种基于RL的算法来解决该问题。基本思想是估计可能决策的收益，其中选择收益最大的决策。我们没有采用原来的深度Q网络（DQN），而是通过添加优先级缓冲机制和专家缓冲机制，提出了Advanced DQN（ADQN），分别提高了样本的利用率和克服了冷启动问题。

目的：最小化时延和能耗的权值和(优化参数：用户卸载策略，卸载的比例)。

算法：ADQN。

二、模型细节与算法

2.1系统模型

计算卸载主要包含: 本地计算和边缘计算

1、本地计算：用户U_i在时隙t_j时的计算时延(计算时延和排队时延)和能耗为

T_{ij}^{l}=D_{ij}^{l}+t_{q}^{l}=\frac{\omega B_{ij}Q_{y_{ij}}}{f_{ij}^{l}}+t_{q}^{l}

E_{ij}^{l}=e_lB_{ij}Q_{y_{ij}}

其中B_{ij}表示任务大小，Q_{y_{ij}}表示本地计算任务的比例

2、边缘计算：包含时延(上下行传输时延+执行时延+排队时延)和能耗
T_{ij}^{r}=D_{ij}^{r}+t_q^r=\frac{\omega B_{ij}P_{y_{ij}}}{F_{ij}^r}+\frac{B_{ij}P_{y_{ij}}}{r_{ul}^{ij}}+\frac{\omega^s B_{ij}}{r_{dl}^{ij}} + t_q^r

E_{ij}^r=e_r B_{ij}P_{y_{ij}}

其中P_{y_{ij}}表示卸载到边缘服务器的比例，\omega^s表示每单位计算任务处理后得到的结果大小。

3、总的时延与能耗成本

W_{ij}=\lambda \max (T_{ij}^r,T_{ij}^l)+\beta(E_{ij}^r+E_{ij}^l)

2.2优化目标

\min \sum_{t_j=1}^{T}[W_{ij}]

       {\tt s.t.}   f_{x_{ij}}^r \leqslant F_{x_{ij}}^r, f_{x_{ij}}^l \leqslant F_{x_{ij}}^l

                r_{ul}^{ij} \leqslant R_{ul}^i, r_{dl}^{ij} \leqslant R_{dl}^i

                x_{ij} \in {1,2,...,m},y_{ij} \in {1,2,...,n}

2.3 State、Action和Reward

1、状态空间：

状态空间包含：（数据大小、本地CPU频率，local可用计算资源，上行速率，下行速率，ES可用计算资源)

2、动作空间：

动作空间包含：（用户选择ES的策略，任务卸载比例）

3、奖励：

奖励：R=\frac{W_l - W(s,a)}{W_l}

W_l表示本地计算成本，W(s,a)表示部分卸载产生的成本。

2.4代码

计算卸载环境代码：

  1. import math
  2. import copy
  3. import numpy as np
  4.  
  5.  
  6. class ENV():
  7.     def __init__(self, UEs=3, MECs=7, k=33, lam=0.5):
  8.         self.UEs = UEs
  9.         self.MECs = MECs
  10.         self.k = k
  11.  
  12.         q = np.full((k, 1), 0.)
  13.         p = np.linspace(0, 1, k).reshape((k, 1))
  14.         # Create Action
  15.         for i in range(MECs - 1):
  16.             a = np.full((k, 1), float(i + 1))
  17.             b = np.linspace(0, 1, k).reshape((k, 1))
  18.             q = np.append(q, a, axis=0)
  19.             p = np.append(p, b, axis=0)  # 231(33 * 7) * 33
  20.  
  21.         self.actions = np.hstack((q, p))  # 231 * 2
  22.         self.n_actions = len(self.actions)  # 231
  23.         self.n_features = 3 + MECs * 3  # 3 + 7 * 3
  24.         self.discount = 0.01
  25.  
  26.         # 基本参数
  27.         # 频率
  28.         self.Hz = 1
  29.         self.kHz = 1000 * self.Hz
  30.         self.mHz = 1000 * self.kHz
  31.         self.GHz = 1000 * self.mHz
  32.         self.nor = 10 ** (-7)
  33.         self.nor1 = 10 ** 19
  34.  
  35.         # 数据大小
  36.         self.bit = 1
  37.         self.B = 8 * self.bit
  38.         self.KB = 1024 * self.B
  39.         self.MB = 1024 * self.KB
  40.  
  41.         # self.task_cpu_cycle = np.random.randint(2 * 10**9, 3* 10**9)
  42.  
  43.         self.UE_f = np.random.randint(1.5 * self.GHz * self.nor, 2 * self.GHz * self.nor)  # UE的计算能力
  44.         self.MEC_f = np.random.randint(5 * self.GHz * self.nor, 7 * self.GHz * self.nor)  # MEC的计算能力
  45.         # self.UE_f = 500 * self.mHz     # UE的计算能力
  46.         # self.MEC_f = np.random.randint(5.2 * self.GHz, 24.3 * self.GHz)  # MEC的计算能力
  47.         self.tr_energy = 1  # 传输能耗
  48.         self.r = 40 * math.log2(1 + (16 * 10)) * self.MB * self.nor  # 传输速率
  49.         # self.r = 800 # 传输速率
  50.         self.ew, self.lw = 10 ** (-26), 3 * 10 ** (-26)  # 能耗系数
  51.         # self.ew, self.lw = 0.3, 0.15 # 能耗系数
  52.         self.et, self.lt = 1, 1
  53.         self.local_core_max, self.local_core_min = 1.3 * self.UE_f, 0.7 * self.UE_f
  54.         self.server_core_max, self.server_core_min = 1.3 * self.MEC_f, 0.7 * self.MEC_f
  55.         self.uplink_max, self.uplink_min = 1.3 * self.r, 0.7 * self.r
  56.         self.downlink_max, self.downlink_min = 1.3 * self.r, 0.7 * self.r
  57.         self.lam = lam
  58.         self.e = 1
  59.  
  60.     def reset(self):
  61.         # 初始化环境，状态空间
  62.         obs = []
  63.         servers_cap = []
  64.         new_cap = True
  65.         for i in range(self.UEs):
  66.             uplink, downlink = [], []
  67.             # np.random.seed(np.random.randint(1, 1000))
  68.             # task_size = np.random.randint(2 * 10**8 * self.nor, 3 * 10**8 * self.nor) #   任务大小
  69.             task_size = np.random.randint(1.5 * self.mHz, 2 * self.mHz)  # 任务大小
  70.             # self.task_size = self.task_size * self.task_cpu_cycle                     # 处理一个任务所需要的cpu频率
  71.             # task_cpu_cycle = np.random.randint(2 * 10**9 * self.nor, 3 * 10**9 * self.nor)
  72.             task_cpu_cycle = np.random.randint(10 ** 3, 10 ** 5)
  73.             local_comp = np.random.randint(0.9 * self.UE_f, 1.1 * self.UE_f)  # UE的计算能力
  74.             for i in range(self.MECs):
  75.                 up = np.random.randint(0.9 * self.r, 1.1 * self.r)
  76.                 down = np.random.randint(0.9 * self.r, 1.1 * self.r)
  77.                 if new_cap:
  78.                     cap = np.random.randint(0.9 * self.MEC_f, 1.1 * self.MEC_f)  # MEC计算能力
  79.                     servers_cap.append(cap)
  80.                 uplink.append(up)
  81.                 downlink.append(down)
  82.             observation = np.array([task_size, task_cpu_cycle, local_comp])
  83.             observation = np.hstack((observation, servers_cap, uplink, downlink))
  84.             obs.append(observation)
  85.             new_cap = False
  86.         return obs
  87.  
  88.     def choose_action(self, prob):
  89.         """
  90.         根据概率选择动作
  91.         :param prob:
  92.         :return:
  93.         """
  94.         action_choice = np.linspace(0, 1, self.k)
  95.         actions = []
  96.         for i in range(self.UEs):
  97.             a = np.random.choice(a=(self.MECs * self.k), p=prob[i])
  98.             target_server = int(a / self.k)
  99.             percen = action_choice[a % self.k]
  100.             action = [target_server, percen]
  101.             actions.append(action)
  102.         return actions
  103.  
  104.     def step(self, observation, actions_prob, is_prob=True, is_compared=True):
  105.         if is_prob:
  106.             actions = self.choose_action(actions_prob)
  107.         else:
  108.             actions = actions_prob
  109.         new_cap = False
  110.         obs_ = []
  111.         rew, local, ran, mec = [], [], [], []
  112.         dpg_times, local_times, ran_times, mec_times = [], [], [], []
  113.         dpg_energys, local_energys, ran_energys, mec_energys = [], [], [], []
  114.         total = []
  115.         a, b, c, d = 0, 0, 0, 0
  116.         for i in range(self.UEs):
  117.             if i == self.UEs - 1:
  118.                 new_cap = True
  119.             # 提取信息
  120.             task_size, task_cpu_cycle, local_comp, servers_cap, uplink, downlink = \
  121.                 observation[i][0], observation[i][1], observation[i][2], observation[i][3:3+self.MECs], observation[i][3+self.MECs:3+self.MECs*2], observation[i][3+self.MECs*2:3+self.MECs*3]
  122.  
  123.             action = actions[i]
  124.             target_server, percen = int(action[0]), action[1]
  125.  
  126.             # 计算奖励
  127.             # 1=======部分卸载==========
  128.             # 卸载及回传数据产生的时延和能耗
  129.             tr_time = (percen * task_size) / uplink[target_server] + self.discount * (percen * task_size) / downlink[
  130.                 target_server]
  131.             tr_energy = (self.tr_energy * percen * task_size) / uplink[target_server] + self.discount * (
  132.                         self.tr_energy * percen * task_size) / downlink[target_server]
  133.  
  134.             # 本地计算时延和能耗
  135.             comp_local_time = task_cpu_cycle * (1 - percen) / (local_comp)
  136.             comp_local_energy = self.lw * task_cpu_cycle * (1 - percen) * local_comp ** 2
  137.  
  138.             # 边缘计算时延和能耗
  139.             comp_mec_time = (percen * task_cpu_cycle) / servers_cap[target_server]
  140.             comp_mec_energy = self.ew * percen * task_cpu_cycle * servers_cap[target_server] ** 2
  141.  
  142.             # 最大计算时延
  143.             comp_time = max(comp_local_time, comp_mec_time)
  144.             time_cost = (comp_time + tr_time) * self.et
  145.             # 能耗成本
  146.             energy_cost = (tr_energy + comp_local_energy + comp_mec_energy) * self.e
  147.             # 总成本
  148.             total_cost = self.lam * time_cost + (1 - self.lam) * energy_cost
  149.  
  150.             # 2、=======完全本地计算==========
  151.             local_only_time = task_cpu_cycle / (local_comp) * self.et
  152.             local_only_energy = self.lw * task_cpu_cycle * local_comp ** 2 * self.e
  153.             # local_only_energy = task_size * local_comp
  154.             local_only = self.lam * local_only_time + (1 - self.lam) * local_only_energy
  155.  
  156.             # 3、=======完全边缘计算==========
  157.             mec_only_tr_time = task_size / uplink[target_server] + self.discount * task_size / downlink[target_server]
  158.             mec_only_tr_energy = self.tr_energy * task_size / uplink[
  159.                 target_server] + self.discount * self.tr_energy * task_size / downlink[target_server]
  160.             # print("mec_only_tr_time:", mec_only_tr_time)
  161.             # print("mec_only_tr_energy:", mec_only_tr_energy)
  162.  
  163.             mec_only_comp_time = task_cpu_cycle / servers_cap[target_server]
  164.             mec_only_comp_energy = self.ew * task_cpu_cycle * servers_cap[target_server] ** 2
  165.             # mec_only_comp_energy = task_size * servers_cap[target_server]
  166.             # print("mec_only_comp_time:", mec_only_comp_time)
  167.             # print("mec_only_comp_energy:", mec_only_comp_energy)
  168.  
  169.             mec_only_time_cost = (mec_only_tr_time + mec_only_comp_time) * self.et
  170.             mec_only_energy_cost = (mec_only_tr_energy + mec_only_comp_energy) * self.e
  171.  
  172.             mec_only = self.lam * mec_only_time_cost + (1 - self.lam) * mec_only_energy_cost
  173.  
  174.             # 4、=======随机卸载==========
  175.             percen_ran = np.random.uniform()  # 随机卸载比例
  176.             mec_ran = np.random.randint(self.MECs)  # 随机选择一个服务器进行卸载
  177.  
  178.             random_tr_time = (percen_ran * task_size) / uplink[mec_ran] + (self.discount * percen_ran * task_size) / \
  179.                              downlink[mec_ran]
  180.             random_tr_energy = (self.tr_energy * percen_ran * task_size) / uplink[mec_ran] + self.discount * (
  181.                         self.tr_energy * percen_ran * task_size) / downlink[mec_ran]
  182.  
  183.             random_comp_local_time = (1 - percen_ran) * task_cpu_cycle / local_comp
  184.             random_comp_local_energy = self.lw * (1 - percen_ran) * task_cpu_cycle * local_comp ** 2
  185.             # random_comp_local_energy = (1 - percen_ran) * task_size * local_comp
  186.  
  187.             random_comp_mec_time = percen_ran * task_cpu_cycle / servers_cap[mec_ran]
  188.             random_comp_mec_energy = self.ew * percen_ran * task_cpu_cycle * servers_cap[mec_ran] ** 2
  189.             # random_comp_mec_energy = percen_ran * task_size * servers_cap[mec_ran]
  190.  
  191.             random_comp_time = max(random_comp_local_time, random_comp_mec_time)
  192.             random_time_cost = (random_comp_time + random_tr_time) * self.et
  193.             random_energy_cost = (random_tr_energy + random_comp_local_energy + random_comp_mec_energy) * self.e
  194.  
  195.             random_total = self.lam * random_time_cost + (1 - self.lam) * random_energy_cost
  196.             random_total_cost2 = random_energy_cost
  197.  
  198.             reward = -total_cost
  199.  
  200.             # 得到下一个observation
  201.             x = np.random.uniform()
  202.             y = 0.5
  203.             if x > y:
  204.                 local_comp = min(local_comp + np.random.randint(0, 0.2 * self.UE_f), self.local_core_max)
  205.                 for j in range(self.MECs):
  206.                     cap = min(servers_cap[j] + np.random.randint(0, 0.3 * self.UE_f), self.server_core_max)
  207.                     # MEC容量保持一致
  208.                     if new_cap:
  209.                         for x in range(self.UEs):
  210.                             observation[x][2 + j] = cap
  211.                     downlink[j] = min(downlink[j] + np.random.randint(0, 0.2 * self.r), self.downlink_max)
  212.                     uplink[j] = min(uplink[j] + np.random.randint(0, 0.2 * self.r), self.uplink_max)
  213.             else:
  214.                 local_comp = max(local_comp + np.random.randint(-0.2 * self.UE_f, 0), self.local_core_min)
  215.                 for j in range(self.MECs):
  216.                     # MEC容量保持一致
  217.                     if new_cap:
  218.                         cap = max(servers_cap[j] + np.random.randint(0, 0.3 * self.UE_f), self.server_core_max)
  219.                         for x in range(self.UEs):
  220.                             observation[x][2 + j] = cap
  221.                     downlink[j] = max(downlink[j] - np.random.randint(0, 0.2 * self.r), self.downlink_min)
  222.                     uplink[j] = max(uplink[j] - np.random.randint(0, 0.2 * self.r), self.uplink_min)
  223.  
  224.             task_size = np.random.randint(10, 50)
  225.             task_cpu_cycle = np.random.randint(10 ** 3, 10 ** 5)  # 处理任务所需要的CPU频率
  226.             observation_ = np.array([task_size, task_cpu_cycle, local_comp])
  227.             observation_ = np.hstack((observation_, servers_cap, uplink, downlink))
  228.             obs_.append(observation_)
  229.  
  230.             rew.append(reward)
  231.             local.append(local_only)
  232.             mec.append(mec_only)
  233.             ran.append(random_total)
  234.  
  235.             dpg_times.append(time_cost)
  236.             local_times.append(local_only_time)
  237.             mec_times.append(mec_only_time_cost)
  238.             ran_times.append(random_time_cost)
  239.  
  240.             dpg_energys.append(energy_cost)
  241.             local_energys.append(local_only_energy)
  242.             mec_energys.append(mec_only_energy_cost)
  243.             ran_energys.append(random_energy_cost)
  244.  
  245.             total.append(total_cost)
  246.  
  247.         if is_compared:
  248.             return obs_, rew, local, mec, ran, dpg_times, local_times, mec_times, ran_times, dpg_energys, local_energys, mec_energys, ran_energys, total
  249.         else:
  250.             return obs_, rew, dpg_times, dpg_energys
  251.

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/不正经/article/detail/296036
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号