当前位置:   article > 正文

Ubuntu16.04+cuda10.0+cudnn7.6配置tensorflow1.x环境

cuda10.0+cudnn7.6

现阶段有多种深度学习框架,应用较多的主要是tensorflow、pytorch以及百度公司的paddlepaddle。个人刚开始使用的深度学习框架是基于pytorch的mmdetection,来自于商汤科技公司,学习和使用了接近一年,利用其跑工程,学习网络等,主要是目标检测方向的应用,整体有较好把握,收获较大。近期,老板事物不多,个人时间较充裕,于是开始学习使用tensorflow,进一步了解其他的深度学习框架。下面分享一下我的环境安装过程:

tensorflow的环境配置不比mmdetection简单,但是由于有了后者的配环境经验,所以摸索起来即使遇到了困难,也能凭经验一步一步解决。我的系统是Ubuntu16.04,之前就装好了cuda,cudatoolkit和cudnn都已经版本对应了,所以少了一些前期的准备工作。

但是实际安装tensorflow的时候还是遇到了很多困难,原先计划安装tensorflow2.x版本,也从tensorflow的官网去找一些官方的安装教程,想避免走一些弯路,但最后下来,我发现我按照官网的教程倒是走了不少弯路,而且还没有成功。按照官网的教程cuda10.0是可以安装2.x版本的,但个人实践经验后还是觉得不靠谱,也和自己不熟悉有关吧,比如docker、bazel构建tensorflow等,操作起来不方便,而且也没有必要,官网把这些放在一起是真的浪费我的时间。

我开始装的是2.0版本,开始按照官网教程来的,有些指令执行不了,换方式也没解决。最后放弃了官方教程,从网上找了一些和自己环境差不多的博客进行参考,指令倒是都顺利装完了,但是跑实例的时候跑不出来,试着修改问题,但找不到源码位置,所以这次的尝试又夭折了。我后面把出现的问题找了一下:

(tf2) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf2/tensorflow/workspace/traning_demo# python model_main_tf2.py --model_dir=models/my_ssd_mobilenet_v2 --pipeline_config_path=models/my_ssd_mobilenet_v2/pipeline.config 
WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
W0408 14:14:37.727346 140687491278656 cross_device_ops.py:1321] There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
I0408 14:14:37.727849 140687491278656 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: None
I0408 14:14:37.742078 140687491278656 config_util.py:552] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0408 14:14:37.742265 140687491278656 config_util.py:552] Maybe overwriting use_bfloat16: False
WARNING:tensorflow:From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/object_detection/model_lib_v2.py:546: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
W0408 14:14:37.806042 140687491278656 deprecation.py:339] From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/object_detection/model_lib_v2.py:546: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
INFO:tensorflow:Reading unweighted datasets: ['images/train.record']
I0408 14:14:37.813343 140687491278656 dataset_builder.py:163] Reading unweighted datasets: ['images/train.record']
INFO:tensorflow:Reading record datasets for input file: ['images/train.record']
I0408 14:14:37.814830 140687491278656 dataset_builder.py:80] Reading record datasets for input file: ['images/train.record']
INFO:tensorflow:Number of filenames to read: 1
I0408 14:14:37.814923 140687491278656 dataset_builder.py:81] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0408 14:14:37.814991 140687491278656 dataset_builder.py:88] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/object_detection/builders/dataset_builder.py:105: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
W0408 14:14:37.817644 140687491278656 deprecation.py:339] From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/object_detection/builders/dataset_builder.py:105: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
WARNING:tensorflow:From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/object_detection/builders/dataset_builder.py:237: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W0408 14:14:37.847065 140687491278656 deprecation.py:339] From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/object_detection/builders/dataset_builder.py:237: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W0408 14:14:43.827106 140687491278656 deprecation.py:339] From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
W0408 14:14:47.471886 140687491278656 deprecation.py:339] From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/object_detection/inputs.py:282: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0408 14:14:49.574233 140687491278656 deprecation.py:339] From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/object_detection/inputs.py:282: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
/root/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/keras/backend.py:434: UserWarning: `tf.keras.backend.set_learning_phase` is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the `training` argument of the `__call__` method of your layer or model.
  warnings.warn('`tf.keras.backend.set_learning_phase` is deprecated and '
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 14:15:02.898844 140671520339712 convolutional_keras_box_predictor.py:154] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 14:15:02.899355 140671520339712 convolutional_keras_box_predictor.py:154] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 14:15:02.899594 140671520339712 convolutional_keras_box_predictor.py:154] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 14:15:02.899832 140671520339712 convolutional_keras_box_predictor.py:154] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 14:15:02.900039 140671520339712 convolutional_keras_box_predictor.py:154] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 14:15:02.900240 140671520339712 convolutional_keras_box_predictor.py:154] depth of additional conv before box predictor: 0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._groundtruth_lists
W0408 14:15:23.303248 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._groundtruth_lists
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor
W0408 14:15:23.303692 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._batched_prediction_tensor_names
W0408 14:15:23.303777 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._batched_prediction_tensor_names
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads
W0408 14:15:23.303838 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._sorted_head_names
W0408 14:15:23.303881 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._sorted_head_names
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._shared_nets
W0408 14:15:23.303931 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._shared_nets
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings
W0408 14:15:23.303973 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background
W0408 14:15:23.304014 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.0
W0408 14:15:23.304055 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.1
W0408 14:15:23.304095 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.2
W0408 14:15:23.304135 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.2
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.3
W0408 14:15:23.304175 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.3
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.4
W0408 14:15:23.304216 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.4
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.5
W0408 14:15:23.304257 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._shared_nets.5
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0
W0408 14:15:23.304313 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1
W0408 14:15:23.304363 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2
W0408 14:15:23.304403 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3
W0408 14:15:23.304442 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4
W0408 14:15:23.304482 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5
W0408 14:15:23.304522 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0
W0408 14:15:23.304562 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1
W0408 14:15:23.304601 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2
W0408 14:15:23.304653 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3
W0408 14:15:23.304707 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4
W0408 14:15:23.304747 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5
W0408 14:15:23.304787 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0._box_encoder_layers
W0408 14:15:23.304844 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0._box_encoder_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1._box_encoder_layers
W0408 14:15:23.304888 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1._box_encoder_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2._box_encoder_layers
W0408 14:15:23.304928 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2._box_encoder_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3._box_encoder_layers
W0408 14:15:23.304969 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3._box_encoder_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4._box_encoder_layers
W0408 14:15:23.305009 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4._box_encoder_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5._box_encoder_layers
W0408 14:15:23.305050 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5._box_encoder_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers
W0408 14:15:23.305090 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1._class_predictor_layers
W0408 14:15:23.305131 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1._class_predictor_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2._class_predictor_layers
W0408 14:15:23.305172 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2._class_predictor_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3._class_predictor_layers
W0408 14:15:23.305213 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3._class_predictor_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4._class_predictor_layers
W0408 14:15:23.305254 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4._class_predictor_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5._class_predictor_layers
W0408 14:15:23.305295 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5._class_predictor_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0
W0408 14:15:23.305337 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1._box_encoder_layers.0
W0408 14:15:23.305378 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1._box_encoder_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2._box_encoder_layers.0
W0408 14:15:23.305418 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2._box_encoder_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3._box_encoder_layers.0
W0408 14:15:23.305459 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3._box_encoder_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4._box_encoder_layers.0
W0408 14:15:23.305498 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4._box_encoder_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5._box_encoder_layers.0
W0408 14:15:23.305538 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5._box_encoder_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0
W0408 14:15:23.305578 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1._class_predictor_layers.0
W0408 14:15:23.305619 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1._class_predictor_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2._class_predictor_layers.0
W0408 14:15:23.305659 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2._class_predictor_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3._class_predictor_layers.0
W0408 14:15:23.305704 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3._class_predictor_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4._class_predictor_layers.0
W0408 14:15:23.305745 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4._class_predictor_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5._class_predictor_layers.0
W0408 14:15:23.305785 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5._class_predictor_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0.kernel
W0408 14:15:23.305828 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0.bias
W0408 14:15:23.305869 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.0._box_encoder_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1._box_encoder_layers.0.kernel
W0408 14:15:23.305909 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1._box_encoder_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1._box_encoder_layers.0.bias
W0408 14:15:23.305949 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.1._box_encoder_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2._box_encoder_layers.0.kernel
W0408 14:15:23.305990 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2._box_encoder_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2._box_encoder_layers.0.bias
W0408 14:15:23.306030 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.2._box_encoder_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3._box_encoder_layers.0.kernel
W0408 14:15:23.306070 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3._box_encoder_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3._box_encoder_layers.0.bias
W0408 14:15:23.306112 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.3._box_encoder_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4._box_encoder_layers.0.kernel
W0408 14:15:23.306151 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4._box_encoder_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4._box_encoder_layers.0.bias
W0408 14:15:23.306191 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.4._box_encoder_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5._box_encoder_layers.0.kernel
W0408 14:15:23.306232 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5._box_encoder_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5._box_encoder_layers.0.bias
W0408 14:15:23.306275 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.box_encodings.5._box_encoder_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0.kernel
W0408 14:15:23.306315 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0.bias
W0408 14:15:23.306356 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.0._class_predictor_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1._class_predictor_layers.0.kernel
W0408 14:15:23.306397 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1._class_predictor_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1._class_predictor_layers.0.bias
W0408 14:15:23.306438 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.1._class_predictor_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2._class_predictor_layers.0.kernel
W0408 14:15:23.306478 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2._class_predictor_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2._class_predictor_layers.0.bias
W0408 14:15:23.306519 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.2._class_predictor_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3._class_predictor_layers.0.kernel
W0408 14:15:23.306560 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3._class_predictor_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3._class_predictor_layers.0.bias
W0408 14:15:23.306600 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.3._class_predictor_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4._class_predictor_layers.0.kernel
W0408 14:15:23.306641 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4._class_predictor_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4._class_predictor_layers.0.bias
W0408 14:15:23.306685 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.4._class_predictor_layers.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5._class_predictor_layers.0.kernel
W0408 14:15:23.306726 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5._class_predictor_layers.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5._class_predictor_layers.0.bias
W0408 14:15:23.306768 140687491278656 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background.5._class_predictor_layers.0.bias
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
W0408 14:15:23.306821 140687491278656 util.py:169] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
WARNING:tensorflow:From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py:605: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
W0408 14:15:32.760873 140644878092032 deprecation.py:537] From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py:605: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140
  • 141
  • 142
  • 143
  • 144
  • 145
  • 146
  • 147
  • 148
  • 149
  • 150
  • 151
  • 152
  • 153
  • 154
  • 155
  • 156
  • 157
  • 158
  • 159
  • 160
  • 161
  • 162
  • 163
  • 164
  • 165
  • 166
  • 167
  • 168
  • 169
  • 170
  • 171
  • 172
  • 173
  • 174
  • 175
  • 176
  • 177
  • 178
  • 179
  • 180
  • 181
  • 182
  • 183
  • 184
  • 185
  • 186
  • 187
  • 188
  • 189
  • 190
  • 191
  • 192
  • 193
  • 194
  • 195
  • 196
  • 197
  • 198
  • 199
  • 200
  • 201
  • 202
  • 203
  • 204
  • 205
  • 206
  • 207
  • 208
  • 209
  • 210
  • 211
  • 212
  • 213
  • 214
  • 215
  • 216
  • 217
  • 218
  • 219
  • 220
  • 221
  • 222
  • 223
  • 224

最后就这么卡住了,看开始报的问题,还无法调用GPU,后面我查看工程文件,还是增加了一些训练的内容,也就是环境是搭上了,但并不对。我也怀疑是版本的问题,于是下一步改为装1.x版本。

针对无法调用gpu这个问题,我上网搜了我报错的地方,发现没什么有效的解决方法,我在执行完安装tensorflow的命令后验证是否可以使用GPU上找到了一点思路,我不按官网说的linux版本使用pip install tensorflow来安装,我直接装我这个版本可以安装的tensorflow-gpu,再去执行验证指令,发现可以调用GPU,我再下一步安装对应版本的其他配置文件。这是我使用的验证指令:

 import tensorflow as tf
 tf.test.is_gpu_available()
  • 1
  • 2

发现返回的是true 说明可以调用GPU了
装完之后,我的环境是这样的:

(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# conda list
# packages in environment at /root/anaconda3/envs/tf1:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_tflow_select             2.1.0                       gpu  
absl-py                   0.12.0           py36h06a4308_0  
astor                     0.8.1            py36h06a4308_0  
blas                      1.0                         mkl  
c-ares                    1.17.1               h27cfd23_0  
ca-certificates           2021.1.19            h06a4308_1  
certifi                   2020.12.5        py36h06a4308_0  
contextlib2               0.6.0.post1              pypi_0    pypi
coverage                  5.5              py36h27cfd23_2  
cudatoolkit               10.0.130                      0  
cudnn                     7.6.5                cuda10.0_0  
cupti                     10.0.130                      0  
cycler                    0.10.0                   pypi_0    pypi
cython                    0.29.22          py36h2531618_0  
gast                      0.2.2                    py36_0  
google-pasta              0.2.0                      py_0  
grpcio                    1.36.1           py36h2157cd5_1  
h5py                      2.10.0           py36hd6299e0_1  
hdf5                      1.10.6               hb1b8bf9_0  
importlib-metadata        3.7.3            py36h06a4308_1  
intel-openmp              2020.2                      254  
keras-applications        1.0.8                      py_1  
keras-preprocessing       1.1.2              pyhd3eb1b0_0  
kiwisolver                1.3.1                    pypi_0    pypi
ld_impl_linux-64          2.33.1               h53a641e_7  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
libprotobuf               3.14.0               h8c45485_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
lvis                      0.5.3                    pypi_0    pypi
lxml                      4.6.3                    pypi_0    pypi
markdown                  3.3.4            py36h06a4308_0  
matplotlib                3.3.4                    pypi_0    pypi
mkl                       2020.2                      256  
mkl-service               2.3.0            py36he8ac12f_0  
mkl_fft                   1.3.0            py36h54f3939_0  
mkl_random                1.1.1            py36h0573a6f_0  
ncurses                   6.2                  he6710b0_1  
numpy                     1.19.2           py36h54aff64_0  
numpy-base                1.19.2           py36hfa32c7d_0  
object-detection          0.1                      pypi_0    pypi
opencv-python             4.5.1.48                 pypi_0    pypi
openssl                   1.1.1k               h27cfd23_0  
opt_einsum                3.1.0                      py_0  
pandas                    1.1.5                    pypi_0    pypi
pillow                    8.2.0                    pypi_0    pypi
pip                       21.0.1           py36h06a4308_0  
protobuf                  3.14.0           py36h2531618_1  
pycocotools               2.0.2                    pypi_0    pypi
pyparsing                 2.4.7                    pypi_0    pypi
python                    3.6.13               hdb3f193_0  
python-dateutil           2.8.1                    pypi_0    pypi
pytz                      2021.1                   pypi_0    pypi
readline                  8.1                  h27cfd23_0  
scipy                     1.5.2            py36h0b6359f_0  
setuptools                52.0.0           py36h06a4308_0  
six                       1.15.0           py36h06a4308_0  
sqlite                    3.35.4               hdfb4753_0  
tensorboard               1.15.0             pyhb230dea_0  
tensorflow                1.15.0          gpu_py36h5a509aa_0  
tensorflow-base           1.15.0          gpu_py36h9dcbed7_0  
tensorflow-estimator      1.15.1             pyh2649769_0  
tensorflow-gpu            1.15.0               h0d30ee6_0  
termcolor                 1.1.0            py36h06a4308_1  
tf-slim                   1.1.0                    pypi_0    pypi
tk                        8.6.10               hbc83047_0  
typing_extensions         3.7.4.3            pyha847dfd_0  
webencodings              0.5.1                    py36_1  
werkzeug                  0.16.1                     py_0  
wheel                     0.36.2             pyhd3eb1b0_0  
wrapt                     1.12.1           py36h7b6447c_1  
xz                        5.2.5                h7b6447c_0  
zipp                      3.4.1              pyhd3eb1b0_0  
zlib                      1.2.11               h7b6447c_3  

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81

我也把我的安装过程大致整理了一下:

(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1# conda install protobuf
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1# cd tensorflow/
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow# mkdir workspace
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow# cd workspace/
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace# mkdir training_demo
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace# cd training_demo/
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# chmod -R a=rwx ./
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# mkdir annotations
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# mkdir exported-models
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# mkdir images
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# mkdir models
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# mkdir pretrained-models
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# chmod -R a=rwx ./
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# cd ../..
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow# cd models/research/
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# protoc object_detection/protos/*.proto --python_out=.
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# git clone https://github.com/cocodataset/cocoapi.git
Cloning into 'cocoapi'...
remote: Enumerating objects: 975, done.
remote: Total 975 (delta 0), reused 0 (delta 0), pack-reused 975
Receiving objects: 100% (975/975), 11.72 MiB | 5.96 MiB/s, done.
Resolving deltas: 100% (576/576), done.
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# cd cocoapi/PythonAPI/
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research/cocoapi/PythonAPI# make
python setup.py build_ext --inplace
running build_ext
cythoning pycocotools/_mask.pyx to pycocotools/_mask.c
/root/anaconda3/envs/tf1/lib/python3.6/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /nfs/private/junnxie/tf1/tensorflow/models/research/cocoapi/PythonAPI/pycocotools/_mask.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
building 'pycocotools._mask' extension
creating build
creating build/common
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/pycocotools
gcc -pthread -B /root/anaconda3/envs/tf1/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/root/.local/lib/python3.6/site-packages/numpy/core/include -I../common -I/root/anaconda3/envs/tf1/include/python3.6m -c ../common/maskApi.c -o build/temp.linux-x86_64-3.6/../common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
../common/maskApi.c: In function ‘rleDecode’:
../common/maskApi.c:46:7: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
       for( k=0; k<R[i].cnts[j]; k++ ) *(M++)=v; v=!v; }}
       ^~~
../common/maskApi.c:46:49: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘forfor( k=0; k<R[i].cnts[j]; k++ ) *(M++)=v; v=!v; }}
                                                 ^
../common/maskApi.c: In function ‘rleFrPoly’:
../common/maskApi.c:166:3: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
   for(j=0; j<k; j++) x[j]=(int)(scale*xy[j*2+0]+.5); x[k]=x[0];
   ^~~
../common/maskApi.c:166:54: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘forfor(j=0; j<k; j++) x[j]=(int)(scale*xy[j*2+0]+.5); x[k]=x[0];
                                                      ^
../common/maskApi.c:167:3: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
   for(j=0; j<k; j++) y[j]=(int)(scale*xy[j*2+1]+.5); y[k]=y[0];
   ^~~
../common/maskApi.c:167:54: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘forfor(j=0; j<k; j++) y[j]=(int)(scale*xy[j*2+1]+.5); y[k]=y[0];
                                                      ^
../common/maskApi.c: In function ‘rleToString’:
../common/maskApi.c:212:7: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
       if(more) c |= 0x20; c+=48; s[p++]=c;
       ^~
../common/maskApi.c:212:27: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘ifif(more) c |= 0x20; c+=48; s[p++]=c;
                           ^
../common/maskApi.c: In function ‘rleFrString’:
../common/maskApi.c:220:3: warning: this ‘while’ clause does not guard... [-Wmisleading-indentation]
   while( s[m] ) m++; cnts=malloc(sizeof(uint)*m); m=0;
   ^~~~~
../common/maskApi.c:220:22: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘whilewhile( s[m] ) m++; cnts=malloc(sizeof(uint)*m); m=0;
                      ^~~~
../common/maskApi.c:228:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
     if(m>2) x+=(long) cnts[m-2]; cnts[m++]=(uint) x;
     ^~
../common/maskApi.c:228:34: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘ifif(m>2) x+=(long) cnts[m-2]; cnts[m++]=(uint) x;
                                  ^~~~
../common/maskApi.c: In function ‘rleToBbox’:
../common/maskApi.c:141:31: warning: ‘xp’ may be used uninitialized in this function [-Wmaybe-uninitialized]
       if(j%2==0) xp=x; else if(xp<x) { ys=0; ye=h-1; }
                               ^
gcc -pthread -B /root/anaconda3/envs/tf1/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/root/.local/lib/python3.6/site-packages/numpy/core/include -I../common -I/root/anaconda3/envs/tf1/include/python3.6m -c pycocotools/_mask.c -o build/temp.linux-x86_64-3.6/pycocotools/_mask.o -Wno-cpp -Wno-unused-function -std=c99
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/pycocotools
gcc -pthread -shared -B /root/anaconda3/envs/tf1/compiler_compat -L/root/anaconda3/envs/tf1/lib -Wl,-rpath=/root/anaconda3/envs/tf1/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/../common/maskApi.o build/temp.linux-x86_64-3.6/pycocotools/_mask.o -o build/lib.linux-x86_64-3.6/pycocotools/_mask.cpython-36m-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.6/pycocotools/_mask.cpython-36m-x86_64-linux-gnu.so -> pycocotools
rm -rf build
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research/cocoapi/PythonAPI# cp -r pycocotools /nfs/private/junnxie/tf1/tensorflow/models/research/
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research/cocoapi/PythonAPI# cd ../..
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# cp object_detection/packages/tf1/setup.py .
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# python
Python 3.6.13 |Anaconda, Inc.| (default, Feb 23 2021, 21:15:04) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf 
>>> tf.__version__
'1.15.0'
>>> 
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# python -m pip install .
Processing /nfs/private/junnxie/tf1/tensorflow/models/research
Collecting pillow
  Downloading Pillow-8.2.0-cp36-cp36m-manylinux1_x86_64.whl (3.0 MB)
     |????????????????????????????????| 3.0 MB 1.9 MB/s 
Collecting lxml
  Downloading lxml-4.6.3-cp36-cp36m-manylinux1_x86_64.whl (5.5 MB)
     |????????????????????????????????| 5.5 MB 18.7 MB/s 
Collecting matplotlib
  Using cached matplotlib-3.3.4-cp36-cp36m-manylinux1_x86_64.whl (11.5 MB)
Requirement already satisfied: Cython in /root/anaconda3/envs/tf1/lib/python3.6/site-packages (from object-detection==0.1) (0.29.22)
Collecting contextlib2
  Using cached contextlib2-0.6.0.post1-py2.py3-none-any.whl (9.8 kB)
Collecting tf-slim
  Using cached tf_slim-1.1.0-py2.py3-none-any.whl (352 kB)
Requirement already satisfied: six in /root/anaconda3/envs/tf1/lib/python3.6/site-packages (from object-detection==0.1) (1.15.0)
Collecting pycocotools
  Using cached pycocotools-2.0.2.tar.gz (23 kB)
Collecting lvis
  Using cached lvis-0.5.3-py3-none-any.whl (14 kB)
Requirement already satisfied: scipy in /root/anaconda3/envs/tf1/lib/python3.6/site-packages (from object-detection==0.1) (1.5.2)
Collecting pandas
  Downloading pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
     |????????????????????????????????| 9.5 MB 25.6 MB/s 
Collecting cycler>=0.10.0
  Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting pyparsing>=2.4.0
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting opencv-python>=4.1.0.25
  Using cached opencv_python-4.5.1.48-cp36-cp36m-manylinux2014_x86_64.whl (50.4 MB)
Requirement already satisfied: numpy>=1.18.2 in /root/.local/lib/python3.6/site-packages (from lvis->object-detection==0.1) (1.19.5)
Collecting kiwisolver>=1.1.0
  Using cached kiwisolver-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB)
Collecting python-dateutil>=2.8.0
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz>=2017.2
  Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Requirement already satisfied: setuptools>=18.0 in /root/anaconda3/envs/tf1/lib/python3.6/site-packages (from pycocotools->object-detection==0.1) (52.0.0.post20210125)
Requirement already satisfied: absl-py>=0.2.2 in /root/anaconda3/envs/tf1/lib/python3.6/site-packages (from tf-slim->object-detection==0.1) (0.12.0)
Building wheels for collected packages: object-detection, pycocotools
  Building wheel for object-detection (setup.py) ... done
  Created wheel for object-detection: filename=object_detection-0.1-py3-none-any.whl size=1643984 sha256=89a5fc6e88577caab4206a9a4a200eb8dba4a7e5c2b970ff9637ad300854d029
  Stored in directory: /tmp/pip-ephem-wheel-cache-zp4g9lh_/wheels/22/22/a1/fc116dd4526d44ac96cbfb85fb89f0a9c5cb88b825a6c25020
  Building wheel for pycocotools (setup.py) ... done
  Created wheel for pycocotools: filename=pycocotools-2.0.2-cp36-cp36m-linux_x86_64.whl size=273408 sha256=b9eb3f09d6addcd01d7c90fde11e61852ca1dd913c31022c5d2ec1f016a600aa
  Stored in directory: /root/.cache/pip/wheels/d8/c2/ba/8f5306f921c2e79ad7b09effdfed6bd966cfcf8c6fe55422d6
Successfully built object-detection pycocotools
Installing collected packages: python-dateutil, pyparsing, pillow, kiwisolver, cycler, pytz, opencv-python, matplotlib, tf-slim, pycocotools, pandas, lxml, lvis, contextlib2, object-detection
Successfully installed contextlib2-0.6.0.post1 cycler-0.10.0 kiwisolver-1.3.1 lvis-0.5.3 lxml-4.6.3 matplotlib-3.3.4 object-detection-0.1 opencv-python-4.5.1.48 pandas-1.1.5 pillow-8.2.0 pycocotools-2.0.2 pyparsing-2.4.7 python-dateutil-2.8.1 pytz-2021.1 tf-slim-1.1.0
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# python object_detection/builders/model_builder_tf1_test.py 
Running tests under Python 3.6.13: /root/anaconda3/envs/tf1/bin/python
[ RUN      ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params0 (True)
[       OK ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params0 (True)
[ RUN      ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params1 (False)
[       OK ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params1 (False)
[ RUN      ] ModelBuilderTF1Test.test_create_experimental_model
[       OK ] ModelBuilderTF1Test.test_create_experimental_model
[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature0 (True)
[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature0 (True)
[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature1 (False)
[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature1 (False)
[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_model_from_config_with_example_miner
[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_model_from_config_with_example_miner
[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul
[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul
[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul
[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul
[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul
[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul
[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul
[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul
[ RUN      ] ModelBuilderTF1Test.test_create_rfcn_model_from_config
[       OK ] ModelBuilderTF1Test.test_create_rfcn_model_from_config
[ RUN      ] ModelBuilderTF1Test.test_create_ssd_fpn_model_from_config
[       OK ] ModelBuilderTF1Test.test_create_ssd_fpn_model_from_config
[ RUN      ] ModelBuilderTF1Test.test_create_ssd_models_from_config
[       OK ] ModelBuilderTF1Test.test_create_ssd_models_from_config
[ RUN      ] ModelBuilderTF1Test.test_invalid_faster_rcnn_batchnorm_update
[       OK ] ModelBuilderTF1Test.test_invalid_faster_rcnn_batchnorm_update
[ RUN      ] ModelBuilderTF1Test.test_invalid_first_stage_nms_iou_threshold
[       OK ] ModelBuilderTF1Test.test_invalid_first_stage_nms_iou_threshold
[ RUN      ] ModelBuilderTF1Test.test_invalid_model_config_proto
[       OK ] ModelBuilderTF1Test.test_invalid_model_config_proto
[ RUN      ] ModelBuilderTF1Test.test_invalid_second_stage_batch_size
[       OK ] ModelBuilderTF1Test.test_invalid_second_stage_batch_size
[ RUN      ] ModelBuilderTF1Test.test_session
[  SKIPPED ] ModelBuilderTF1Test.test_session
[ RUN      ] ModelBuilderTF1Test.test_unknown_faster_rcnn_feature_extractor
[       OK ] ModelBuilderTF1Test.test_unknown_faster_rcnn_feature_extractor
[ RUN      ] ModelBuilderTF1Test.test_unknown_meta_architecture
[       OK ] ModelBuilderTF1Test.test_unknown_meta_architecture
[ RUN      ] ModelBuilderTF1Test.test_unknown_ssd_feature_extractor
[       OK ] ModelBuilderTF1Test.test_unknown_ssd_feature_extractor
----------------------------------------------------------------------
Ran 21 tests in 0.245s

OK (skipped=1)
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140
  • 141
  • 142
  • 143
  • 144
  • 145
  • 146
  • 147
  • 148
  • 149
  • 150
  • 151
  • 152
  • 153
  • 154
  • 155
  • 156
  • 157
  • 158
  • 159
  • 160
  • 161
  • 162
  • 163
  • 164
  • 165
  • 166
  • 167
  • 168
  • 169
  • 170
  • 171
  • 172
  • 173
  • 174
  • 175
  • 176
  • 177
  • 178
  • 179
  • 180
  • 181
  • 182
  • 183
  • 184
  • 185
  • 186
  • 187
  • 188
  • 189
  • 190
  • 191
  • 192
  • 193
  • 194
  • 195
  • 196
  • 197
  • 198

我把指令进一步提炼一下,主要是省去了一些中间过程和结果:

conda create -n tf1 python=3.6
conda activate tf1
conda install tensorflow-gpu=1.15.0
conda install cudatoolkit=10.0
conda install cudnn=7.6


(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1# conda install protobuf

(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow# cd models/research/
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# protoc object_detection/protos/*.proto --python_out=.
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# git clone https://github.com/cocodataset/cocoapi.git

(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# cd cocoapi/PythonAPI/
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research/cocoapi/PythonAPI# make

(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research/cocoapi/PythonAPI# cp -r pycocotools /nfs/private/junnxie/tf1/tensorflow/models/research/
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research/cocoapi/PythonAPI# cd ../..
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# cp object_detection/packages/tf1/setup.py .
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# python
Python 3.6.13 |Anaconda, Inc.| (default, Feb 23 2021, 21:15:04) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf 
>>> tf.__version__
'1.15.0'
>>> 
(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# python -m pip install .

(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/models/research# python object_detection/builders/model_builder_tf1_test.py 

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31

最后,我运行了一个目标检测工程,数据是网上找的,使用tensorflow提供的代码制作标准的数据集用于训练。
训练过程如下:

(tf1) root@k8s-deploy-jcjbxf-1609678570414-7fffc8595-hkrph:/nfs/private/junnxie/tf1/tensorflow/workspace/training_demo# python model_main.py --model_dir=models/my_ssd_inception_v2 --pipeline_config_path=models/my_ssd_inception_v2/pipeline.config 
WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.
W0408 19:36:14.746835 140553343391552 model_lib.py:813] Forced number of epochs for all eval validations to be 1.
INFO:tensorflow:Maybe overwriting train_steps: None
I0408 19:36:14.747101 140553343391552 config_util.py:552] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0408 19:36:14.747195 140553343391552 config_util.py:552] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1
I0408 19:36:14.747280 140553343391552 config_util.py:552] Maybe overwriting sample_1_of_n_eval_examples: 1
INFO:tensorflow:Maybe overwriting eval_num_epochs: 1
I0408 19:36:14.747370 140553343391552 config_util.py:552] Maybe overwriting eval_num_epochs: 1
WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
W0408 19:36:14.747480 140553343391552 model_lib.py:829] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu None
I0408 19:36:14.747575 140553343391552 model_lib.py:866] create_estimator_and_inputs: use_tpu False, export_to_tpu None
INFO:tensorflow:Using config: {'_model_dir': 'models/my_ssd_inception_v2', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fd4ed457e10>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I0408 19:36:14.748096 140553343391552 estimator.py:212] Using config: {'_model_dir': 'models/my_ssd_inception_v2', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fd4ed457e10>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7fd4ed462378>) includes params argument, but params are not passed to Estimator.
W0408 19:36:14.749072 140553343391552 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7fd4ed462378>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Not using Distribute Coordinator.
I0408 19:36:14.749960 140553343391552 estimator_training.py:186] Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
I0408 19:36:14.750178 140553343391552 training.py:612] Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
I0408 19:36:14.750450 140553343391552 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0408 19:36:14.780125 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Reading unweighted datasets: ['annotations/train.record']
I0408 19:36:14.899453 140553343391552 dataset_builder.py:163] Reading unweighted datasets: ['annotations/train.record']
INFO:tensorflow:Reading record datasets for input file: ['annotations/train.record']
I0408 19:36:14.906244 140553343391552 dataset_builder.py:80] Reading record datasets for input file: ['annotations/train.record']
INFO:tensorflow:Number of filenames to read: 1
I0408 19:36:14.906542 140553343391552 dataset_builder.py:81] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0408 19:36:14.906724 140553343391552 dataset_builder.py:88] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/builders/dataset_builder.py:105: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
W0408 19:36:14.920520 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/builders/dataset_builder.py:105: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/builders/dataset_builder.py:237: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W0408 19:36:14.985749 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/builders/dataset_builder.py:237: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/inputs.py:110: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0408 19:36:27.862425 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/inputs.py:110: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/inputs.py:94: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W0408 19:36:28.015421 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/inputs.py:94: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tensorflow_core/python/autograph/operators/control_flow.py:1004: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
W0408 19:36:34.523849 140553343391552 api.py:332] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tensorflow_core/python/autograph/operators/control_flow.py:1004: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/inputs.py:282: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0408 19:36:37.794182 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/inputs.py:282: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
INFO:tensorflow:Calling model_fn.
I0408 19:36:41.447258 140553343391552 estimator.py:1148] Calling model_fn.
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tf_slim/layers/layers.py:2802: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
W0408 19:36:41.740483 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tf_slim/layers/layers.py:2802: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:36:44.422615 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:36:44.448537 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:36:44.473237 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:36:44.497899 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:36:44.522875 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:36:44.547588 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0408 19:36:50.300761 140553343391552 deprecation.py:506] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
I0408 19:36:56.719294 140553343391552 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I0408 19:36:56.720463 140553343391552 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
I0408 19:37:00.901003 140553343391552 monitored_session.py:240] Graph was finalized.
2021-04-08 19:37:00.901417: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-04-08 19:37:00.915375: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
2021-04-08 19:37:00.921384: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55aafa799d00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-08 19:37:00.921444: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-04-08 19:37:00.923131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-04-08 19:37:00.976282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1e:00.0
2021-04-08 19:37:00.976922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-04-08 19:37:00.980881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-04-08 19:37:00.984547: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-04-08 19:37:00.985348: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-04-08 19:37:00.988312: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-04-08 19:37:00.990079: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-04-08 19:37:01.001160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-08 19:37:01.012950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-04-08 19:37:01.013081: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-04-08 19:37:01.291709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-08 19:37:01.291754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2021-04-08 19:37:01.291785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2021-04-08 19:37:01.300849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11299 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:1e:00.0, compute capability: 6.1)
2021-04-08 19:37:01.303098: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55ab0e7a4670 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-04-08 19:37:01.303132: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): TITAN Xp, Compute Capability 6.1
INFO:tensorflow:Running local_init_op.
I0408 19:37:11.309897 140553343391552 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0408 19:37:12.092763 140553343391552 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into models/my_ssd_inception_v2/model.ckpt.
I0408 19:37:24.229167 140553343391552 basic_session_run_hooks.py:606] Saving checkpoints for 0 into models/my_ssd_inception_v2/model.ckpt.
2021-04-08 19:37:42.652158: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-08 19:37:48.131007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
INFO:tensorflow:loss = 26.353672, step = 0
I0408 19:37:50.209866 140553343391552 basic_session_run_hooks.py:262] loss = 26.353672, step = 0
INFO:tensorflow:global_step/sec: 2.45856
I0408 19:38:30.884657 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 2.45856
INFO:tensorflow:loss = 7.0979624, step = 100 (40.679 sec)
I0408 19:38:30.887990 140553343391552 basic_session_run_hooks.py:260] loss = 7.0979624, step = 100 (40.679 sec)
INFO:tensorflow:global_step/sec: 3.5244
I0408 19:38:59.256149 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.5244
INFO:tensorflow:loss = 6.423036, step = 200 (28.385 sec)
I0408 19:38:59.272437 140553343391552 basic_session_run_hooks.py:260] loss = 6.423036, step = 200 (28.385 sec)
INFO:tensorflow:global_step/sec: 3.44889
I0408 19:39:28.251045 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.44889
INFO:tensorflow:loss = 6.2270827, step = 300 (28.982 sec)
I0408 19:39:28.253960 140553343391552 basic_session_run_hooks.py:260] loss = 6.2270827, step = 300 (28.982 sec)
INFO:tensorflow:global_step/sec: 3.51676
I0408 19:39:56.686080 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.51676
INFO:tensorflow:loss = 6.3535028, step = 400 (28.434 sec)
I0408 19:39:56.688244 140553343391552 basic_session_run_hooks.py:260] loss = 6.3535028, step = 400 (28.434 sec)
INFO:tensorflow:global_step/sec: 3.53058
I0408 19:40:25.010002 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.53058
INFO:tensorflow:loss = 5.5039124, step = 500 (28.324 sec)
I0408 19:40:25.011913 140553343391552 basic_session_run_hooks.py:260] loss = 5.5039124, step = 500 (28.324 sec)
INFO:tensorflow:global_step/sec: 3.52694
I0408 19:40:53.363433 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.52694
INFO:tensorflow:loss = 5.1926427, step = 600 (28.354 sec)
I0408 19:40:53.366256 140553343391552 basic_session_run_hooks.py:260] loss = 5.1926427, step = 600 (28.354 sec)
INFO:tensorflow:global_step/sec: 3.50592
I0408 19:41:21.886609 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.50592
INFO:tensorflow:loss = 5.6967077, step = 700 (28.533 sec)
I0408 19:41:21.899112 140553343391552 basic_session_run_hooks.py:260] loss = 5.6967077, step = 700 (28.533 sec)
INFO:tensorflow:global_step/sec: 3.51716
I0408 19:41:50.318632 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.51716
INFO:tensorflow:loss = 4.9808583, step = 800 (28.422 sec)
I0408 19:41:50.321131 140553343391552 basic_session_run_hooks.py:260] loss = 4.9808583, step = 800 (28.422 sec)
INFO:tensorflow:global_step/sec: 3.55013
I0408 19:42:18.486627 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.55013
INFO:tensorflow:loss = 5.414756, step = 900 (28.168 sec)
I0408 19:42:18.489443 140553343391552 basic_session_run_hooks.py:260] loss = 5.414756, step = 900 (28.168 sec)
INFO:tensorflow:global_step/sec: 3.54571
I0408 19:42:46.689790 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.54571
INFO:tensorflow:loss = 4.1702933, step = 1000 (28.203 sec)
I0408 19:42:46.692520 140553343391552 basic_session_run_hooks.py:260] loss = 4.1702933, step = 1000 (28.203 sec)
INFO:tensorflow:global_step/sec: 3.56015
I0408 19:43:14.778345 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.56015
INFO:tensorflow:loss = 4.250403, step = 1100 (28.088 sec)
I0408 19:43:14.780816 140553343391552 basic_session_run_hooks.py:260] loss = 4.250403, step = 1100 (28.088 sec)
INFO:tensorflow:global_step/sec: 3.4538
I0408 19:43:43.732146 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.4538
INFO:tensorflow:loss = 4.8844323, step = 1200 (28.964 sec)
I0408 19:43:43.744886 140553343391552 basic_session_run_hooks.py:260] loss = 4.8844323, step = 1200 (28.964 sec)
INFO:tensorflow:global_step/sec: 3.50202
I0408 19:44:12.286877 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.50202
INFO:tensorflow:loss = 5.226234, step = 1300 (28.544 sec)
I0408 19:44:12.289032 140553343391552 basic_session_run_hooks.py:260] loss = 5.226234, step = 1300 (28.544 sec)
INFO:tensorflow:global_step/sec: 3.51738
I0408 19:44:40.717238 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.51738
INFO:tensorflow:loss = 4.5052533, step = 1400 (28.431 sec)
I0408 19:44:40.719830 140553343391552 basic_session_run_hooks.py:260] loss = 4.5052533, step = 1400 (28.431 sec)
INFO:tensorflow:global_step/sec: 3.55431
I0408 19:45:08.852223 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.55431
INFO:tensorflow:loss = 4.34357, step = 1500 (28.135 sec)
I0408 19:45:08.855122 140553343391552 basic_session_run_hooks.py:260] loss = 4.34357, step = 1500 (28.135 sec)
INFO:tensorflow:global_step/sec: 3.49746
I0408 19:45:37.444391 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.49746
INFO:tensorflow:loss = 4.2605658, step = 1600 (28.592 sec)
I0408 19:45:37.447230 140553343391552 basic_session_run_hooks.py:260] loss = 4.2605658, step = 1600 (28.592 sec)
INFO:tensorflow:global_step/sec: 3.53497
I0408 19:46:05.733187 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.53497
INFO:tensorflow:loss = 3.5676794, step = 1700 (28.295 sec)
I0408 19:46:05.741827 140553343391552 basic_session_run_hooks.py:260] loss = 3.5676794, step = 1700 (28.295 sec)
INFO:tensorflow:global_step/sec: 3.55267
I0408 19:46:33.881087 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.55267
INFO:tensorflow:loss = 4.289984, step = 1800 (28.142 sec)
I0408 19:46:33.884087 140553343391552 basic_session_run_hooks.py:260] loss = 4.289984, step = 1800 (28.142 sec)
INFO:tensorflow:global_step/sec: 3.53733
I0408 19:47:02.150989 140553343391552 basic_session_run_hooks.py:692] global_step/sec: 3.53733
INFO:tensorflow:loss = 3.744223, step = 1900 (28.270 sec)
I0408 19:47:02.153777 140553343391552 basic_session_run_hooks.py:260] loss = 3.744223, step = 1900 (28.270 sec)
INFO:tensorflow:Saving checkpoints for 1997 into models/my_ssd_inception_v2/model.ckpt.
I0408 19:47:29.373254 140553343391552 basic_session_run_hooks.py:606] Saving checkpoints for 1997 into models/my_ssd_inception_v2/model.ckpt.
INFO:tensorflow:Reading unweighted datasets: ['annotations/test.record']
I0408 19:47:43.834930 140553343391552 dataset_builder.py:163] Reading unweighted datasets: ['annotations/test.record']
INFO:tensorflow:Reading record datasets for input file: ['annotations/test.record']
I0408 19:47:43.839394 140553343391552 dataset_builder.py:80] Reading record datasets for input file: ['annotations/test.record']
INFO:tensorflow:Number of filenames to read: 1
I0408 19:47:43.839524 140553343391552 dataset_builder.py:81] Number of filenames to read: 1
INFO:tensorflow:Calling model_fn.
I0408 19:47:45.471107 140553343391552 estimator.py:1148] Calling model_fn.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:47:47.676634 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:47:47.701958 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:47:47.727361 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:47:47.752024 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:47:47.777025 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:47:47.801182 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/eval_util.py:929: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0408 19:47:48.630852 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/eval_util.py:929: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
W0408 19:47:48.957098 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
INFO:tensorflow:Done calling model_fn.
I0408 19:47:49.616228 140553343391552 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-08T19:47:49Z
I0408 19:47:49.635172 140553343391552 evaluation.py:255] Starting evaluation at 2021-04-08T19:47:49Z
INFO:tensorflow:Graph was finalized.
I0408 19:47:50.092404 140553343391552 monitored_session.py:240] Graph was finalized.
2021-04-08 19:47:50.101238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1e:00.0
2021-04-08 19:47:50.101469: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-04-08 19:47:50.101504: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-04-08 19:47:50.101532: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-04-08 19:47:50.101561: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-04-08 19:47:50.101586: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-04-08 19:47:50.101615: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-04-08 19:47:50.101686: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-08 19:47:50.111780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-04-08 19:47:50.111874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-08 19:47:50.111910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2021-04-08 19:47:50.111944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2021-04-08 19:47:50.116084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11299 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:1e:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from models/my_ssd_inception_v2/model.ckpt-1997
I0408 19:47:50.128154 140553343391552 saver.py:1284] Restoring parameters from models/my_ssd_inception_v2/model.ckpt-1997
INFO:tensorflow:Running local_init_op.
I0408 19:47:52.254904 140553343391552 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0408 19:47:52.436636 140553343391552 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 25 images.
I0408 19:47:56.768712 140496349411072 coco_evaluation.py:293] Performing evaluation on 25 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0408 19:47:56.769833 140496349411072 coco_tools.py:116] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.00s)
I0408 19:47:56.772033 140496349411072 coco_tools.py:138] DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.22s).
Accumulating evaluation results...
DONE (t=0.03s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.001
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.007
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.002
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.021
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.022
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.021
INFO:tensorflow:Finished evaluation at 2021-04-08-19:47:57
I0408 19:47:57.225715 140553343391552 evaluation.py:275] Finished evaluation at 2021-04-08-19:47:57
INFO:tensorflow:Saving dict for global step 1997: DetectionBoxes_Precision/mAP = 0.0001709473, DetectionBoxes_Precision/mAP (large) = 0.00013584658, DetectionBoxes_Precision/mAP (medium) = 0.0073927394, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.00088169175, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.002352941, DetectionBoxes_Recall/AR@100 = 0.02117647, DetectionBoxes_Recall/AR@100 (large) = 0.020967742, DetectionBoxes_Recall/AR@100 (medium) = 0.02173913, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 8.233138, Loss/localization_loss = 4.0231056, Loss/regularization_loss = 0.48583883, Loss/total_loss = 12.742084, global_step = 1997, learning_rate = 0.004, loss = 12.742084
I0408 19:47:57.226226 140553343391552 estimator.py:2049] Saving dict for global step 1997: DetectionBoxes_Precision/mAP = 0.0001709473, DetectionBoxes_Precision/mAP (large) = 0.00013584658, DetectionBoxes_Precision/mAP (medium) = 0.0073927394, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.00088169175, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.002352941, DetectionBoxes_Recall/AR@100 = 0.02117647, DetectionBoxes_Recall/AR@100 (large) = 0.020967742, DetectionBoxes_Recall/AR@100 (medium) = 0.02173913, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 8.233138, Loss/localization_loss = 4.0231056, Loss/regularization_loss = 0.48583883, Loss/total_loss = 12.742084, global_step = 1997, learning_rate = 0.004, loss = 12.742084
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1997: models/my_ssd_inception_v2/model.ckpt-1997
I0408 19:47:58.198050 140553343391552 estimator.py:2109] Saving 'checkpoint_path' summary for global step 1997: models/my_ssd_inception_v2/model.ckpt-1997
INFO:tensorflow:Saving checkpoints for 2000 into models/my_ssd_inception_v2/model.ckpt.
I0408 19:47:59.095469 140553343391552 basic_session_run_hooks.py:606] Saving checkpoints for 2000 into models/my_ssd_inception_v2/model.ckpt.
INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).
I0408 19:48:01.902159 140553343391552 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
INFO:tensorflow:Reading unweighted datasets: ['annotations/test.record']
I0408 19:48:01.939406 140553343391552 dataset_builder.py:163] Reading unweighted datasets: ['annotations/test.record']
INFO:tensorflow:Reading record datasets for input file: ['annotations/test.record']
I0408 19:48:01.946810 140553343391552 dataset_builder.py:80] Reading record datasets for input file: ['annotations/test.record']
INFO:tensorflow:Number of filenames to read: 1
I0408 19:48:01.946915 140553343391552 dataset_builder.py:81] Number of filenames to read: 1
INFO:tensorflow:Calling model_fn.
I0408 19:48:02.928972 140553343391552 estimator.py:1148] Calling model_fn.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:05.048127 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:05.074190 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:05.100624 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:05.125298 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:05.150101 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:05.175326 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:Done calling model_fn.
I0408 19:48:06.351978 140553343391552 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-08T19:48:06Z
I0408 19:48:06.365622 140553343391552 evaluation.py:255] Starting evaluation at 2021-04-08T19:48:06Z
INFO:tensorflow:Graph was finalized.
I0408 19:48:06.785058 140553343391552 monitored_session.py:240] Graph was finalized.
2021-04-08 19:48:06.787840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1e:00.0
2021-04-08 19:48:06.788024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-04-08 19:48:06.788048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-04-08 19:48:06.788072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-04-08 19:48:06.788094: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-04-08 19:48:06.788114: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-04-08 19:48:06.788137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-04-08 19:48:06.788157: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-08 19:48:06.791190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-04-08 19:48:06.791261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-08 19:48:06.791276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2021-04-08 19:48:06.791332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2021-04-08 19:48:06.794789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11299 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:1e:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from models/my_ssd_inception_v2/model.ckpt-2000
I0408 19:48:06.804881 140553343391552 saver.py:1284] Restoring parameters from models/my_ssd_inception_v2/model.ckpt-2000
INFO:tensorflow:Running local_init_op.
I0408 19:48:08.983566 140553343391552 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0408 19:48:09.121775 140553343391552 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 25 images.
I0408 19:48:13.511092 140495493789440 coco_evaluation.py:293] Performing evaluation on 25 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0408 19:48:13.512457 140495493789440 coco_tools.py:116] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.00s)
I0408 19:48:13.514708 140495493789440 coco_tools.py:138] DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.25s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.001
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.009
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.027
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.017
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.031
INFO:tensorflow:Finished evaluation at 2021-04-08-19:48:13
I0408 19:48:13.930575 140553343391552 evaluation.py:275] Finished evaluation at 2021-04-08-19:48:13
INFO:tensorflow:Saving dict for global step 2000: DetectionBoxes_Precision/mAP = 0.000258673, DetectionBoxes_Precision/mAP (large) = 0.00023983713, DetectionBoxes_Precision/mAP (medium) = 0.009405941, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.001324426, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.027058823, DetectionBoxes_Recall/AR@100 (large) = 0.030645162, DetectionBoxes_Recall/AR@100 (medium) = 0.017391304, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.987997, Loss/localization_loss = 4.0830684, Loss/regularization_loss = 0.48584178, Loss/total_loss = 12.556907, global_step = 2000, learning_rate = 0.004, loss = 12.556907
I0408 19:48:13.931072 140553343391552 estimator.py:2049] Saving dict for global step 2000: DetectionBoxes_Precision/mAP = 0.000258673, DetectionBoxes_Precision/mAP (large) = 0.00023983713, DetectionBoxes_Precision/mAP (medium) = 0.009405941, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.001324426, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.027058823, DetectionBoxes_Recall/AR@100 (large) = 0.030645162, DetectionBoxes_Recall/AR@100 (medium) = 0.017391304, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.987997, Loss/localization_loss = 4.0830684, Loss/regularization_loss = 0.48584178, Loss/total_loss = 12.556907, global_step = 2000, learning_rate = 0.004, loss = 12.556907
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2000: models/my_ssd_inception_v2/model.ckpt-2000
I0408 19:48:13.956167 140553343391552 estimator.py:2109] Saving 'checkpoint_path' summary for global step 2000: models/my_ssd_inception_v2/model.ckpt-2000
INFO:tensorflow:Performing the final export in the end of training.
I0408 19:48:13.959100 140553343391552 exporter.py:410] Performing the final export in the end of training.
INFO:tensorflow:Calling model_fn.
I0408 19:48:14.372867 140553343391552 estimator.py:1148] Calling model_fn.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:16.682502 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:16.708631 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:16.733302 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:16.758555 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:16.782882 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0408 19:48:16.807321 140553343391552 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:Done calling model_fn.
I0408 19:48:17.382560 140553343391552 estimator.py:1150] Done calling model_fn.
WARNING:tensorflow:From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
W0408 19:48:17.382837 140553343391552 deprecation.py:323] From /root/anaconda3/envs/tf1/lib/python3.6/site-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
I0408 19:48:17.383411 140553343391552 export_utils.py:170] Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
I0408 19:48:17.383496 140553343391552 export_utils.py:170] Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['tensorflow/serving/predict', 'serving_default']
I0408 19:48:17.383557 140553343391552 export_utils.py:170] Signatures INCLUDED in export for Predict: ['tensorflow/serving/predict', 'serving_default']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
I0408 19:48:17.383643 140553343391552 export_utils.py:170] Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
I0408 19:48:17.383695 140553343391552 export_utils.py:170] Signatures INCLUDED in export for Eval: None
2021-04-08 19:48:17.385852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1e:00.0
2021-04-08 19:48:17.385957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-04-08 19:48:17.385978: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-04-08 19:48:17.385999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-04-08 19:48:17.386017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-04-08 19:48:17.386034: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-04-08 19:48:17.386053: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-04-08 19:48:17.386070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-08 19:48:17.389097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-04-08 19:48:17.389145: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-08 19:48:17.389159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2021-04-08 19:48:17.389171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2021-04-08 19:48:17.392436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11299 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:1e:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from models/my_ssd_inception_v2/model.ckpt-2000
I0408 19:48:17.407397 140553343391552 saver.py:1284] Restoring parameters from models/my_ssd_inception_v2/model.ckpt-2000
INFO:tensorflow:Assets added to graph.
I0408 19:48:18.091548 140553343391552 builder_impl.py:665] Assets added to graph.
INFO:tensorflow:No assets to write.
I0408 19:48:18.091738 140553343391552 builder_impl.py:460] No assets to write.
INFO:tensorflow:SavedModel written to: models/my_ssd_inception_v2/export/Servo/temp-b'1617882493'/saved_model.pb
I0408 19:48:20.169840 140553343391552 builder_impl.py:425] SavedModel written to: models/my_ssd_inception_v2/export/Servo/temp-b'1617882493'/saved_model.pb
INFO:tensorflow:Loss for final step: 3.5268488.
I0408 19:48:21.147778 140553343391552 estimator.py:371] Loss for final step: 3.5268488.

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140
  • 141
  • 142
  • 143
  • 144
  • 145
  • 146
  • 147
  • 148
  • 149
  • 150
  • 151
  • 152
  • 153
  • 154
  • 155
  • 156
  • 157
  • 158
  • 159
  • 160
  • 161
  • 162
  • 163
  • 164
  • 165
  • 166
  • 167
  • 168
  • 169
  • 170
  • 171
  • 172
  • 173
  • 174
  • 175
  • 176
  • 177
  • 178
  • 179
  • 180
  • 181
  • 182
  • 183
  • 184
  • 185
  • 186
  • 187
  • 188
  • 189
  • 190
  • 191
  • 192
  • 193
  • 194
  • 195
  • 196
  • 197
  • 198
  • 199
  • 200
  • 201
  • 202
  • 203
  • 204
  • 205
  • 206
  • 207
  • 208
  • 209
  • 210
  • 211
  • 212
  • 213
  • 214
  • 215
  • 216
  • 217
  • 218
  • 219
  • 220
  • 221
  • 222
  • 223
  • 224
  • 225
  • 226
  • 227
  • 228
  • 229
  • 230
  • 231
  • 232
  • 233
  • 234
  • 235
  • 236
  • 237
  • 238
  • 239
  • 240
  • 241
  • 242
  • 243
  • 244
  • 245
  • 246
  • 247
  • 248
  • 249
  • 250
  • 251
  • 252
  • 253
  • 254
  • 255
  • 256
  • 257
  • 258
  • 259
  • 260
  • 261
  • 262
  • 263
  • 264
  • 265
  • 266
  • 267
  • 268
  • 269
  • 270
  • 271
  • 272
  • 273
  • 274
  • 275
  • 276
  • 277
  • 278
  • 279
  • 280
  • 281
  • 282
  • 283
  • 284
  • 285
  • 286
  • 287
  • 288
  • 289
  • 290
  • 291
  • 292
  • 293
  • 294
  • 295
  • 296
  • 297
  • 298
  • 299
  • 300
  • 301
  • 302
  • 303
  • 304
  • 305
  • 306
  • 307
  • 308
  • 309
  • 310
  • 311
  • 312
  • 313
  • 314
  • 315
  • 316
  • 317
  • 318
  • 319
  • 320
  • 321
  • 322
  • 323
  • 324
  • 325
  • 326
  • 327
  • 328
  • 329
  • 330
  • 331
  • 332
  • 333
  • 334
  • 335
  • 336
  • 337
  • 338
  • 339
  • 340
  • 341
  • 342
  • 343
  • 344
  • 345
  • 346
  • 347
  • 348
  • 349
  • 350
  • 351
  • 352
  • 353
  • 354
  • 355
  • 356
  • 357
  • 358
  • 359
  • 360
  • 361
  • 362
  • 363
  • 364
  • 365
  • 366
  • 367
  • 368
  • 369
  • 370
  • 371
  • 372
  • 373
  • 374
  • 375
  • 376
  • 377
  • 378
  • 379
  • 380
  • 381
  • 382
  • 383
  • 384
  • 385
  • 386
  • 387
  • 388
  • 389
  • 390
  • 391
  • 392
  • 393
  • 394
  • 395
  • 396
  • 397
  • 398
  • 399
  • 400
  • 401
  • 402
  • 403
  • 404
  • 405
  • 406
  • 407
  • 408
  • 409
  • 410
  • 411
  • 412
  • 413
  • 414
  • 415
  • 416
  • 417
  • 418
  • 419
  • 420
  • 421
  • 422
  • 423
  • 424
  • 425
  • 426
  • 427
  • 428
  • 429
  • 430
  • 431
  • 432
  • 433
  • 434
  • 435
  • 436
  • 437
  • 438
  • 439
  • 440
  • 441
  • 442
  • 443
  • 444
  • 445
  • 446
  • 447
  • 448
  • 449
  • 450
  • 451
  • 452
  • 453
  • 454
  • 455
  • 456
  • 457
  • 458
  • 459
  • 460
  • 461
  • 462
  • 463
  • 464
  • 465
  • 466
  • 467
  • 468
  • 469
  • 470
  • 471
  • 472
  • 473
  • 474
  • 475
  • 476
  • 477
  • 478
  • 479
  • 480
  • 481
  • 482
  • 483
  • 484
  • 485
  • 486

跑完后的结果使用tensorfboard进行可视化,结果如下:
准确率曲线:
在这里插入图片描述
召回率曲线:
在这里插入图片描述
损失曲线:
在这里插入图片描述
迭代20W次后的验证集结果:
左右两张图是一致的,左图是模型测试结果,右图是真值(人为标的),进行对比可得准确率。模型准确率较低,主要原因是训练的数据太小,只有三四百张图片,而且标注的目标特征较复杂,学习起来难度较大,导致模型准确率低。从训练的准确率也可以看出,数据量小,迭代次数太多没有意义,反而会过拟合,导致准确率下降。
在这里插入图片描述
在这里插入图片描述

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/AllinToyou/article/detail/687765
推荐阅读
相关标签
  

闽ICP备14008679号