当前位置:   article > 正文

Hadoop-3.1.1 Yarn 对GPU的调度和管理_yarn gpu

yarn gpu

GPU ON YARN上的配置

hadoop-3.1.1版本的yarn支持对gpu的调度和管理,主要分为两种模式:

  • yarn自动获取gpu的资源,进行分配;
  • 由用户指定使用哪些gpu资源;

具体的配置方法详见文档 Using GPU On YARN.md

GPU ON YARN 上的执行

执行命令:

yarn jar /home/nht/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar \
  -jar /home/nht/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar \
  -shell_script /root/smitest.sh \
  -container_memory 1024 \
  -container_resources memory-mb=1024,vcores=1,yarn.io/gpu=1 \
  -num_containers 2
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

或者直接执行

yarn jar /home/nht/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar \
  -jar /home/nht/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar \
  -shell_script /root/smitest.sh 
  • 1
  • 2
  • 3

脚本smitest.sh里面的内容:

# !/bin/bash

/usr/bin/nvidia-smi > /tmp/smitest
  • 1
  • 2
  • 3

输出文件smitest里面的内容:

Fri Sep 28 14:43:09 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.66                 Driver Version: 384.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0 Off |                  N/A |
| 27%   33C    P8    12W / 180W |     10MiB /  8112MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1080    Off  | 00000000:02:00.0 Off |                  N/A |
| 27%   33C    P8    12W / 180W |     10MiB /  8114MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Process
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/寸_铁/article/detail/741577
推荐阅读
相关标签
  

闽ICP备14008679号