赞
踩
也是为了复现最原始的 Attention is all you need 实验过程;
下载cuda 10.0 for ubuntu18.04
CUDA Toolkit 10.0 Download | NVIDIA Developer
安装cuda:
预备工作
创建blacklist***.conf具体如下,加入两行内容
-
-
- sudo vim /etc/modprobe.d/blacklist-nouveau.conf
- 内容:
- blacklist nouveau
- options nouveau modeset=0
-
- sudo update-initramfs -u
-
-
-
-
-
安装cuda 10.0:
1. 手动下载 cuda 10.0 for ubuntu18.04,并安装:
-
- sudo apt-get install linux-headers-$(uname -r) \
- && sudo apt-key del 7fa2af80 \
- && sudo dpkg -i cuda-repo-ubuntu1804-10-0-local-10.0.130-410.48_1.0-1_amd64.deb \
- && sudo apt-key add /var/cuda-repo-10-0-local-10.0.130-410.48/7fa2af80.pub \
- && sudo apt-get update \
- && sudo apt-get -y install cuda
注册nvidia开发者,并下载对应 ***cudnn***.deb,安装命令:
- #注意安装顺序:
-
-
- sudo dpkg -i libcudnn7_7.6.5.32-1+cuda10.0_amd64.deb && \
- sudo dpkg -i libcudnn7-dev_7.6.5.32-1+cuda10.0_amd64.deb && \
- sudo dpkg -i libcudnn7-doc_7.6.5.32-1+cuda10.0_amd64.deb
预先安装合适版本的python3 module:
- sudo apt install python3.7*
-
- sudo rm /usr/bin/python3
-
- sudo ln /usr/bin/python3.7 /usr/bin/python3
-
- sudo apt install python3-pip
-
- sudo pip3 install ipython
-
- sudo pip3 install Cython
-
- sudo apt-get install -y glibc-doc manpages-posix-dev
-
- sudo pip3 install numpy==1.15.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/
-
- sudo pip3 install pkgconfig==1.4.0
-
- cd /usr/lib/python3/dist-packages
- sudo ln -s apt_pkg.cpython-36m-x86_64-linux-gnu.so apt_pkg.so
-
- # install HDF5:具体方法在文末
-
- sudo pip3 install h5py==2.10.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/
-
- sudo pip3 install setuptools==57.5.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/
sudo pip3 install tensorflow-gpu==1.14.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/
-
-
- sudo apt-get update
- sudo apt-get install build-essential
- sudo apt-get build-dep hdf5
-
- mkdir ~/Software
- cd ~/Software
- wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.10/src/hdf5-1.10.10.tar.gz
-
- tar -xf hdf5-1.8.10.tar.gz
- cd hdf5-1.8.10/
- ./configure
- make -j9
- sudo make install
demo 测试源码 hello_tf114.py
- import os
- os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
-
- def tf114_demo():
- a = 3
- b = 4
- c = a + b
- print("a + b in py =",c)
-
- a_t = tf.constant(3)
- b_t = tf.constant(4)
- c_t = a_t + b_t
- print("TensorFlow add a_t + b_t =", c_t)
-
- with tf.Session() as sess:
- c_t_value = sess.run(c_t)
- print("c_t_value= ", c_t_value)
-
- return None
-
- if __name__ == "__main__":
- tf114_demo()
运行结果:
rtx 2080ti 是2018年5月发布,cuda 10.2 首版是2019年发布的,故可以推断,cuda 10.2也支持2080ti,并可以在 2080ti上 运行 tensorflow 1.14.0;待更深入的网络训练的测试
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
实测发现,在ubuntu18.04中,使用cuda10.2并不能直接支持tensorflow 1.14.0,因为tensorflow 1.14.0会默认 load cuda相关lib的10.0版本,例如 libcudart.so.10.0 libcublas.so.10.0 等;
如果安装cuda10.2后不做任何改动,将出现如下错误:
所以,要尝试用10.2的so充当 10.0 的 so文件:
- cd /usr/lib/x86_64-linux-gnu \
- && sudo ln -s libcublas.so.10.2.2.89 libcublas.so.10.0 \
- && sudo ln -s libcublasLt.so.10.2.2.89 libcublasLt.so.10.0
-
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/targets/x86_64-linux/lib/
-
- cd /usr/local/cuda-10.2/targets/x86_64-linux/lib/ \
- && sudo ln -s libcufft.so.10.1.2.89 libcufft.so.10.0 \
- && sudo ln -s libcufftw.so.10.1.2.89 libcufftw.so.10.0 \
- && sudo ln -s libcurand.so.10.1.2.89 libcurand.so.10.0 \
- && sudo ln -s libcusolverMg.so.10.3.0.89 libcusolverMg.so.10.0 \
- && sudo ln -s libcusolver.so.10.3.0.89 libcusolver.so.10.0 \
- && sudo ln -s libcusparse.so.10.3.1.89 libcusparse.so.10.0 \
- && sudo ln -s libnppc.so.10.2.1.89 libnppc.so.10.0 \
- && sudo ln -s libnppial.so.10.2.1.89 libnppial.so.10.0 \
- && sudo ln -s libnppicc.so.10.2.1.89 libnppicc.so.10.0 \
- && sudo ln -s libnppicom.so.10.2.1.89 libnppicom.so.10.0 \
- && sudo ln -s libnppidei.so.10.2.1.89 libnppidei.so.10.0 \
- && sudo ln -s libnppif.so.10.2.1.89 libnppif.so.10.0 \
- && sudo ln -s libnppig.so.10.2.1.89 libnppig.so.10.0 \
- && sudo ln -s libnppim.so.10.2.1.89 libnppim.so.10.0 \
- && sudo ln -s libnppist.so.10.2.1.89 libnppist.so.10.0 \
- && sudo ln -s libnppisu.so.10.2.1.89 libnppisu.so.10.0 \
- && sudo ln -s libnppitc.so.10.2.1.89 libnppitc.so.10.0 \
- && sudo ln -s libnpps.so.10.2.1.89 libnpps.so.10.0 \
- && sudo ln -s libnvgraph.so.10.2.89 libnvgraph.so.10.0 \
- && sudo ln -s libnvjpeg.so.10.3.1.89 libnvjpeg.so.10.0 \
- && sudo ln -s libcudart.so.10.2.89 libcudart.so.10.0 \
- && sudo ln -s libaccinj64.so.10.2.89 libaccinj64.so.10.0 \
- && sudo ln -s libcuinj64.so.10.2.89 libcuinj64.so.10.0 \
- && sudo ln -s libcupti.so.10.2.75 libcupti.so.10.0 \
- && sudo ln -s libnvrtc-builtins.so.10.2.89 libnvrtc-builtins.so.10.0 \
- && sudo ln -s libnvrtc.so.10.2.89 libnvrtc.so.10.0
-
-
-
- #别忘了这个:
- export LD_LIBRARY_PATH=/usr/local/cuda-10.2/targets/x86_64-linux/lib/
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/targets/x86_64-linux/lib/
浅测可以:
cuda10.0 应该是调用不动2080TI卡,因为发布时间的先后;而 cuda10.2发布的时间晚于2080ti的发布时间,所以是可以支持到的;
bidirectional_rnn.py
- """ Bi-directional Recurrent Neural Network.
- A Bi-directional Recurrent Neural Network (LSTM) implementation example using
- TensorFlow library. This example is using the MNIST database of handwritten
- digits (http://yann.lecun.com/exdb/mnist/)
- Links:
- [Long Short Term Memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf)
- [MNIST Dataset](http://yann.lecun.com/exdb/mnist/).
- Author: Aymeric Damien
- Project: https://github.com/aymericdamien/TensorFlow-Examples/
- """
-
- from __future__ import print_function
-
- import tensorflow as tf
- from tensorflow.contrib import rnn
- import numpy as np
-
- # Import MNIST data
- from tensorflow.examples.tutorials.mnist import input_data
- mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
-
- '''
- To classify images using a bidirectional recurrent neural network, we consider
- every image row as a sequence of pixels. Because MNIST image shape is 28*28px,
- we will then handle 28 sequences of 28 steps for every sample.
- '''
-
- # Training Parameters
- learning_rate = 0.001
- training_steps = 10000
- batch_size = 128
- display_step = 200
-
- # Network Parameters
- num_input = 28 # MNIST data input (img shape: 28*28)
- timesteps = 28 # timesteps
- num_hidden = 128 # hidden layer num of features
- num_classes = 10 # MNIST total classes (0-9 digits)
-
- # tf Graph input
- X = tf.placeholder("float", [None, timesteps, num_input])
- Y = tf.placeholder("float", [None, num_classes])
-
- # Define weights
- weights = {
- # Hidden layer weights => 2*n_hidden because of forward + backward cells
- 'out': tf.Variable(tf.random_normal([2*num_hidden, num_classes]))
- }
- biases = {
- 'out': tf.Variable(tf.random_normal([num_classes]))
- }
-
-
- def BiRNN(x, weights, biases):
-
- # Prepare data shape to match `rnn` function requirements
- # Current data input shape: (batch_size, timesteps, n_input)
- # Required shape: 'timesteps' tensors list of shape (batch_size, num_input)
-
- # Unstack to get a list of 'timesteps' tensors of shape (batch_size, num_input)
- x = tf.unstack(x, timesteps, 1)
-
- # Define lstm cells with tensorflow
- # Forward direction cell
- lstm_fw_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)
- # Backward direction cell
- lstm_bw_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)
-
- # Get lstm cell output
- try:
- outputs, _, _ = rnn.static_bidirectional_rnn(lstm_fw_cell, lstm_bw_cell, x,
- dtype=tf.float32)
- except Exception: # Old TensorFlow version only returns outputs not states
- outputs = rnn.static_bidirectional_rnn(lstm_fw_cell, lstm_bw_cell, x,
- dtype=tf.float32)
-
- # Linear activation, using rnn inner loop last output
- return tf.matmul(outputs[-1], weights['out']) + biases['out']
-
- logits = BiRNN(X, weights, biases)
- prediction = tf.nn.softmax(logits)
-
- # Define loss and optimizer
- loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
- logits=logits, labels=Y))
- optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
- train_op = optimizer.minimize(loss_op)
-
- # Evaluate model (with test logits, for dropout to be disabled)
- correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))
- accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
-
- # Initialize the variables (i.e. assign their default value)
- init = tf.global_variables_initializer()
-
- # Start training
- with tf.Session() as sess:
-
- # Run the initializer
- sess.run(init)
-
- for step in range(1, training_steps+1):
- batch_x, batch_y = mnist.train.next_batch(batch_size)
- # Reshape data to get 28 seq of 28 elements
- batch_x = batch_x.reshape((batch_size, timesteps, num_input))
- # Run optimization op (backprop)
- sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
- if step % display_step == 0 or step == 1:
- # Calculate batch loss and accuracy
- loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,
- Y: batch_y})
- print("Step " + str(step) + ", Minibatch Loss= " + \
- "{:.4f}".format(loss) + ", Training Accuracy= " + \
- "{:.3f}".format(acc))
-
- print("Optimization Finished!")
-
- # Calculate accuracy for 128 mnist test images
- test_len = 128
- test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))
- test_label = mnist.test.labels[:test_len]
- print("Testing Accuracy:", \
- sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))
$ python3 bidirectional_rnn.py
训练效果:
nvidia-smi gpu 使用效果:
convolutional_network_raw_deviceInfo.py
- """ Convolutional Neural Network.
- Build and train a convolutional neural network with TensorFlow.
- This example is using the MNIST database of handwritten digits
- (http://yann.lecun.com/exdb/mnist/)
- Author: Aymeric Damien
- Project: https://github.com/aymericdamien/TensorFlow-Examples/
- """
-
- from __future__ import division, print_function, absolute_import
-
- import tensorflow as tf
-
- # Import MNIST data
- from tensorflow.examples.tutorials.mnist import input_data
- mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
-
- # Training Parameters
- learning_rate = 0.001
- num_steps = 200
- batch_size = 128
- display_step = 10
-
- # Network Parameters
- num_input = 784 # MNIST data input (img shape: 28*28)
- num_classes = 10 # MNIST total classes (0-9 digits)
- dropout = 0.75 # Dropout, probability to keep units
-
- # tf Graph input
- X = tf.placeholder(tf.float32, [None, num_input])
- Y = tf.placeholder(tf.float32, [None, num_classes])
- keep_prob = tf.placeholder(tf.float32) # dropout (keep probability)
-
-
- # Create some wrappers for simplicity
- def conv2d(x, W, b, strides=1):
- # Conv2D wrapper, with bias and relu activation
- x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')
- x = tf.nn.bias_add(x, b)
- return tf.nn.relu(x)
-
-
- def maxpool2d(x, k=2):
- # MaxPool2D wrapper
- return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],
- padding='SAME')
-
-
- # Create model
- def conv_net(x, weights, biases, dropout):
- # MNIST data input is a 1-D vector of 784 features (28*28 pixels)
- # Reshape to match picture format [Height x Width x Channel]
- # Tensor input become 4-D: [Batch Size, Height, Width, Channel]
- x = tf.reshape(x, shape=[-1, 28, 28, 1])
-
- # Convolution Layer
- conv1 = conv2d(x, weights['wc1'], biases['bc1'])
- # Max Pooling (down-sampling)
- conv1 = maxpool2d(conv1, k=2)
-
- # Convolution Layer
- conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
- # Max Pooling (down-sampling)
- conv2 = maxpool2d(conv2, k=2)
-
- # Fully connected layer
- # Reshape conv2 output to fit fully connected layer input
- fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])
- fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
- fc1 = tf.nn.relu(fc1)
- # Apply Dropout
- fc1 = tf.nn.dropout(fc1, dropout)
-
- # Output, class prediction
- out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
- return out
-
- # Store layers weight & bias
- weights = {
- # 5x5 conv, 1 input, 32 outputs
- 'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32])),
- # 5x5 conv, 32 inputs, 64 outputs
- 'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])),
- # fully connected, 7*7*64 inputs, 1024 outputs
- 'wd1': tf.Variable(tf.random_normal([7*7*64, 1024])),
- # 1024 inputs, 10 outputs (class prediction)
- 'out': tf.Variable(tf.random_normal([1024, num_classes]))
- }
-
- biases = {
- 'bc1': tf.Variable(tf.random_normal([32])),
- 'bc2': tf.Variable(tf.random_normal([64])),
- 'bd1': tf.Variable(tf.random_normal([1024])),
- 'out': tf.Variable(tf.random_normal([num_classes]))
- }
-
- # Construct model
- logits = conv_net(X, weights, biases, keep_prob)
- prediction = tf.nn.softmax(logits)
-
- # Define loss and optimizer
- loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
- logits=logits, labels=Y))
- optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
- train_op = optimizer.minimize(loss_op)
-
-
- # Evaluate model
- correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))
- accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
-
- # Initialize the variables (i.e. assign their default value)
- init = tf.global_variables_initializer()
-
- # Start training
- with tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)) as sess:
-
- # Run the initializer
- sess.run(init)
-
- for step in range(1, num_steps+1):
- batch_x, batch_y = mnist.train.next_batch(batch_size)
- # Run optimization op (backprop)
- sess.run(train_op, feed_dict={X: batch_x, Y: batch_y, keep_prob: 0.8})
- if step % display_step == 0 or step == 1:
- # Calculate batch loss and accuracy
- loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,
- Y: batch_y,
- keep_prob: 1.0})
- print("Step " + str(step) + ", Minibatch Loss= " + \
- "{:.4f}".format(loss) + ", Training Accuracy= " + \
- "{:.3f}".format(acc))
-
- print("Optimization Finished!")
-
- # Calculate accuracy for 256 MNIST test images
- print("Testing Accuracy:", \
- sess.run(accuracy, feed_dict={X: mnist.test.images[:256],
- Y: mnist.test.labels[:256],
- keep_prob: 1.0}))
$ python3 convolutional_network_raw_deviceInfo.py
训练效果:
$nvidia-smi
————————————————————————————————————————
- sudo apt install python3-pip \
- && sudo pip3 install Cython \
- && sudo pip3 install ipython \
- && sudo apt-get install -y glibc-doc manpages-posix-dev \
- && sudo pip3 install numpy==1.15.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/ \
- && sudo pip3 install pkgconfig==1.4.0 \
- && cd /usr/lib/python3/dist-packages \
- && sudo ln -s apt_pkg.cpython-36m-x86_64-linux-gnu.so apt_pkg.so
-
-
-
-
- sudo apt-get update \
- && sudo apt-get install build-essential \
- && sudo apt-get build-dep hdf5 \
- && mkdir ~/Software \
- && cd ~/Software \
- && wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.10/src/hdf5-1.10.10.tar.gz \
- && tar -xf hdf5-1.10.10.tar.gz \
- && cd hdf5-1.10.10/ \
- && ./configure \
- && make -j9 \
- && sudo make install
-
-
-
- sudo pip3 install h5py==2.10.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/ \
- && sudo pip3 install setuptools==57.5.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/
-
- cd /usr/lib/x86_64-linux-gnu \
- && sudo ln -s libcublas.so.10.2.2.89 libcublas.so.10.0 \
- && sudo ln -s libcublasLt.so.10.2.2.89 libcublasLt.so.10.0
-
-
-
-
-
-
-
-
-
-
-
- cd /usr/local/cuda-10.2/targets/x86_64-linux/lib/ \
- && sudo ln -s libcufft.so.10.1.2.89 libcufft.so.10.0 \
- && sudo ln -s libcufftw.so.10.1.2.89 libcufftw.so.10.0 \
- && sudo ln -s libcurand.so.10.1.2.89 libcurand.so.10.0 \
- && sudo ln -s libcusolverMg.so.10.3.0.89 libcusolverMg.so.10.0 \
- && sudo ln -s libcusolver.so.10.3.0.89 libcusolver.so.10.0 \
- && sudo ln -s libcusparse.so.10.3.1.89 libcusparse.so.10.0 \
- && sudo ln -s libnppc.so.10.2.1.89 libnppc.so.10.0 \
- && sudo ln -s libnppial.so.10.2.1.89 libnppial.so.10.0 \
- && sudo ln -s libnppicc.so.10.2.1.89 libnppicc.so.10.0 \
- && sudo ln -s libnppicom.so.10.2.1.89 libnppicom.so.10.0 \
- && sudo ln -s libnppidei.so.10.2.1.89 libnppidei.so.10.0 \
- && sudo ln -s libnppif.so.10.2.1.89 libnppif.so.10.0 \
- && sudo ln -s libnppig.so.10.2.1.89 libnppig.so.10.0 \
- && sudo ln -s libnppim.so.10.2.1.89 libnppim.so.10.0 \
- && sudo ln -s libnppist.so.10.2.1.89 libnppist.so.10.0 \
- && sudo ln -s libnppisu.so.10.2.1.89 libnppisu.so.10.0 \
- && sudo ln -s libnppitc.so.10.2.1.89 libnppitc.so.10.0 \
- && sudo ln -s libnpps.so.10.2.1.89 libnpps.so.10.0 \
- && sudo ln -s libnvgraph.so.10.2.89 libnvgraph.so.10.0 \
- && sudo ln -s libnvjpeg.so.10.3.1.89 libnvjpeg.so.10.0 \
- && sudo ln -s libcudart.so.10.2.89 libcudart.so.10.0 \
- && sudo ln -s libaccinj64.so.10.2.89 libaccinj64.so.10.0 \
- && sudo ln -s libcuinj64.so.10.2.89 libcuinj64.so.10.0 \
- && sudo ln -s libcupti.so.10.2.75 libcupti.so.10.0 \
- && sudo ln -s libnvrtc-builtins.so.10.2.89 libnvrtc-builtins.so.10.0 \
- && sudo ln -s libnvrtc.so.10.2.89 libnvrtc.so.10.0
-
-
-
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/targets/x86_64-linux/lib/
测试Quadro P600 4GB显卡也可以, cuda 10.2 + cudnn 7.6.5 + tensorflow 1.14.0
成功训练:
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。