devbox
菜鸟追梦旅行
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

（15）点云数据处理学习——单目深度估计获得RGBD图再重建点云_深度图像和rgb图像生成点云数据

作者：菜鸟追梦旅行 | 2024-02-22 01:11:37

赞

踩

深度图像和rgb图像生成点云数据

1、主要参考

（1）大佬视频

Create Your Own Point Clouds from Depth Maps - Point Cloud Processing in Open3D_哔哩哔哩_bilibili

（2）重要！！！我前面的教程

（7）点云数据处理学习——单摄像头深度估计_chencaw的博客-CSDN博客

（6）点云数据处理学习——RGBD图_chencaw的博客-CSDN博客_点云图由几张rgb图像生成

2、 前提回顾和实现

2.1 思路

陈PS：

  1. 用普通相机获得彩色的RGB图
  2. 然后用MiDaS深度学习模型获得深度图
  3. 然后这两个图已对应不就是RGBD图了嘛
  4. 然后用open3D三维点云重建一下。

2.2摄像头标定和参数

（1）根据我前面的教程，可以进行摄像头标定和内参获取

（7）点云数据处理学习——单摄像头深度估计_chencaw的博客-CSDN博客

 （2）内参矩阵为

mtx:
 [[801.31799138   0.         319.96097314]
 [  0.         804.76125593 206.79594003]
 [  0.           0.           1.        ]]
dist畸变值:
 [[-7.21246445e-02 -6.84714453e-01 -1.25501966e-02  5.75752614e-03
   9.50679972e+00]]

（3）外部调节参数

<?xml version="1.0"?>

<opencv_storage>

<intrinsic type_id="opencv-matrix">

  <rows>3</rows>

  <cols>3</cols>

  <dt>d</dt>

  <data>

    1.1476093750000000e+03 0. 2.8008979619154707e+02 0.

    1.1125493164062500e+03 2.7150911855935556e+02 0. 0. 1.</data></intrinsic>

</opencv_storage>

2.3 根据单目深度模型获得RGB图和深度图

（1）本次实现的代码34mono_to_rgbd.py

  1.  
  2.  
  3. import torch
  4. import matplotlib.pyplot as plt
  5. import cv2
  6. import numpy as np
  7. import time
  8.  
  9.  
  10. rgbd_save_path = "D:/RGBD_CAMERA/python_3d_process/chen_rgbd/"
  11.  
  12. ##直接将D:/RGBD_CAMERA/mis/MiDaS-master/hubconf.py中的transforms拿来使用
  13. def transforms():
  14.     import cv2
  15.     from torchvision.transforms import Compose
  16.     from midas.transforms import Resize, NormalizeImage, PrepareForNet
  17.     from midas import transforms
  18.  
  19.     transforms.default_transform = Compose(
  20.         [
  21.             lambda img: {"image": img / 255.0},
  22.             Resize(
  23.                 384,
  24.                 384,
  25.                 resize_target=None,
  26.                 keep_aspect_ratio=True,
  27.                 ensure_multiple_of=32,
  28.                 resize_method="upper_bound",
  29.                 image_interpolation_method=cv2.INTER_CUBIC,
  30.             ),
  31.             NormalizeImage(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
  32.             PrepareForNet(),
  33.             lambda sample: torch.from_numpy(sample["image"]).unsqueeze(0),
  34.         ]
  35.     )
  36.  
  37.     transforms.small_transform = Compose(
  38.         [
  39.             lambda img: {"image": img / 255.0},
  40.             Resize(
  41.                 256,
  42.                 256,
  43.                 resize_target=None,
  44.                 keep_aspect_ratio=True,
  45.                 ensure_multiple_of=32,
  46.                 resize_method="upper_bound",
  47.                 image_interpolation_method=cv2.INTER_CUBIC,
  48.             ),
  49.             NormalizeImage(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
  50.             PrepareForNet(),
  51.             lambda sample: torch.from_numpy(sample["image"]).unsqueeze(0),
  52.         ]
  53.     )
  54.  
  55.     transforms.dpt_transform = Compose(
  56.         [
  57.             lambda img: {"image": img / 255.0},
  58.             Resize(
  59.                 384,
  60.                 384,
  61.                 resize_target=None,
  62.                 keep_aspect_ratio=True,
  63.                 ensure_multiple_of=32,
  64.                 resize_method="minimal",
  65.                 image_interpolation_method=cv2.INTER_CUBIC,
  66.             ),
  67.             NormalizeImage(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
  68.             PrepareForNet(),
  69.             lambda sample: torch.from_numpy(sample["image"]).unsqueeze(0),
  70.         ]
  71.     )
  72.  
  73.     return transforms
  74.  
  75.  
  76. # （一）方法一、使用torch.hub或者从官网下载
  77. # https://github.com/isl-org/MiDaS#Accuracy
  78. # model_type = "DPT_Large"     # MiDaS v3 - Large     (highest accuracy, slowest inference speed)
  79. # #model_type = "DPT_Hybrid"   # MiDaS v3 - Hybrid    (medium accuracy, medium inference speed)
  80. # #model_type = "MiDaS_small"  # MiDaS v2.1 - Small   (lowest accuracy, highest inference speed)
  81. # midas = torch.hub.load("intel-isl/MiDaS", model_type)
  82.  
  83. # （二）方法二、下载本地后直接加载
  84. # (1)Load a model
  85. model_type = "DPT_Large"
  86. # midas = torch.hub.load('intel-isl/MiDaS', path='D:/BaiduNetdiskDownload/dpt_large-midas-2f21e586.pt', source='local',model =model_type )
  87. # midas = torch.hub.load('D:/RGBD_CAMERA/mis/MiDaS-master', path='D:/BaiduNetdiskDownload/dpt_large-midas-2f21e586.pt', source='local',model =model_type,force_reload = False )
  88. midas = torch.hub.load('D:/RGBD_CAMERA/mis/MiDaS-master', source='local',model =model_type,force_reload = False )
  89.  
  90. #(2)Move model to GPU if available
  91. device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
  92. midas.to(device)
  93. midas.eval()
  94.  
  95. #(3)Load transforms to resize and normalize the image for large or small model
  96. # midas_transforms = torch.hub.load("intel-isl/MiDaS", "transforms")
  97. midas_transforms = transforms()
  98.  
  99. if model_type == "DPT_Large" or model_type == "DPT_Hybrid":
  100.     transform = midas_transforms.dpt_transform
  101. else:
  102.     transform = midas_transforms.small_transform
  103.  
  104. print("chen0")
  105.  
  106.  
  107. #（4）Load image and apply transforms
  108. # filename = 'D:/RGBD_CAMERA/python_3d_process/dog.jpg'
  109. # img = cv2.imread(filename)
  110. # img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
  111.  
  112.  
  113. # Open up the video capture from a webcam
  114. cap = cv2.VideoCapture(1)
  115. print("chencap")
  116. success, img = cap.read()
  117. cv2.imshow("origin_pic",img)
  118. cv2.waitKey(3)
  119. img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
  120.  
  121.  
  122.  
  123. print("chen1")
  124. input_batch = transform(img).to(device)
  125.  
  126. #(5)Predict and resize to original resolution
  127. with torch.no_grad():
  128.     prediction = midas(input_batch)
  129.  
  130.     prediction = torch.nn.functional.interpolate(
  131.         prediction.unsqueeze(1),
  132.         size=img.shape[:2],
  133.         mode="bicubic",
  134.         align_corners=False,
  135.     ).squeeze()
  136.  
  137. output = prediction.cpu().numpy()
  138. print(output.shape)
  139. print("chen2")
  140. #(6)Show result
  141. plt.subplot(1,2,1)
  142. plt.imshow(img)
  143. plt.subplot(1,2,2)
  144. plt.imshow(output)
  145. plt.show()
  146.  
  147. cv2.imwrite(rgbd_save_path+"chen_depth_img.png",output)
  148.  
  149. #转换回来
  150. img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
  151. cv2.imwrite(rgbd_save_path+"chen_color_img.jpg",img)
  152.  
  153. cap.release()
  154. cv2.destroyAllWindows()

（2）RGBD的处理代码

可以参考我前面的教程

（6）点云数据处理学习——RGBD图_chencaw的博客-CSDN博客_点云图由几张rgb图像生成

  1. import open3d as o3d
  2. import numpy as np
  3. from matplotlib import pyplot as plt
  4.  
  5.  
  6. rgbd_save_path = "D:/RGBD_CAMERA/python_3d_process/chen_rgbd/"
  7. color_raw = o3d.io.read_image(rgbd_save_path+"chen_color_img.jpg")
  8. # print(np.asarray(color_raw))
  9. # plt.imshow(np.asarray(color_raw))
  10. # plt.show()
  11.  
  12. depth_raw = o3d.io.read_image(rgbd_save_path+"chen_depth_img.png")
  13. # plt.imshow(np.asarray(depth_raw))
  14. # plt.show()
  15.  
  16. rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(color_raw, depth_raw)
  17. print(rgbd_image)
  18.  
  19. plt.subplot(1, 2, 1)
  20. plt.title('SUN grayscale image')
  21. plt.imshow(rgbd_image.color)
  22. plt.subplot(1, 2, 2)
  23. plt.title('SUN depth image')
  24. plt.imshow(rgbd_image.depth)
  25. plt.show()
  26.  
  27. pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
  28.     rgbd_image,
  29.     o3d.camera.PinholeCameraIntrinsic(
  30.         o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
  31. # Flip it, otherwise the pointcloud will be upside down
  32. # pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
  33. pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
  34. # o3d.visualization.draw_geometries([pcd], zoom=0.5)
  35. o3d.visualization.draw_geometries([pcd])

 （3）效果，咳咳，还是不错的，但是还是放大佬的图吧

 

2.3 把摄像头内参用起来，提高精度

2.3.1 默认内参

（1）注意，以下是默认内参。刚才上面的程序使用了默认内参来实现RGBD图转点云显示

    o3d.camera.PinholeCameraIntrinsic(

        o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault)

默认内参的值，参照官方教程

RGBD images — Open3D 0.16.0 documentation

这里我们使用PinholeCameraIntrinsicParameters。PrimeSenseDefault作为默认的相机参数。它具有图像分辨率640x480，焦距(fx, fy) =(525.0, 525.0)，光学中心(cx, cy) =(319.5, 239.5)。 

2.3.2 使用自己标定的内参

(1)注意：使用了外部调整参数！！！！！

  1. import open3d as o3d
  2. import numpy as np
  3. from matplotlib import pyplot as plt
  4. import cv2
  5.  
  6. rgbd_save_path = "D:/RGBD_CAMERA/python_3d_process/chen_rgbd/"
  7. color_raw = o3d.io.read_image(rgbd_save_path+"chen_color_img.jpg")
  8. # print(np.asarray(color_raw))
  9. # plt.imshow(np.asarray(color_raw))
  10. # plt.show()
  11.  
  12. depth_raw = o3d.io.read_image(rgbd_save_path+"chen_depth_img.png")
  13. # plt.imshow(np.asarray(depth_raw))
  14. # plt.show()
  15.  
  16. rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(color_raw, depth_raw)
  17. print(rgbd_image)
  18.  
  19. plt.subplot(1, 2, 1)
  20. plt.title('SUN grayscale image')
  21. plt.imshow(rgbd_image.color)
  22. plt.subplot(1, 2, 2)
  23. plt.title('SUN depth image')
  24. plt.imshow(rgbd_image.depth)
  25. plt.show()
  26.  
  27.  
  28.  
  29. #方法（一）默认摄像头内参
  30. # pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
  31. #     rgbd_image,
  32. #     o3d.camera.PinholeCameraIntrinsic(
  33. #         o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
  34.  
  35. #方法（二）设定自己的摄像机外部调整参数
  36. cv_file = cv2.FileStorage()
  37. cv_file.open('D:/RGBD_CAMERA/python_3d_process/1mono_to_3d/cameraIntrinsic.xml',cv2.FileStorage_READ)
  38. camera_intrinsic = cv_file.getNode('intrinsic').mat()
  39. print(camera_intrinsic)
  40. camera_intrinsic_o3d =  o3d.camera.PinholeCameraIntrinsic(width=640,height=480,
  41.                                                                     fx=camera_intrinsic[0][0],
  42.                                                                     fy=camera_intrinsic[1][1],
  43.                                                                     cx=camera_intrinsic[0][2],
  44.                                                                     cy=camera_intrinsic[1][2])
  45. print(camera_intrinsic_o3d.intrinsic_matrix)
  46. pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
  47.     rgbd_image,
  48.     camera_intrinsic_o3d)
  49.  
  50. # Flip it, otherwise the pointcloud will be upside down
  51. # pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
  52.  
  53. #注意：左右转换了一下！否则显示是镜像图
  54. pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
  55. # o3d.visualization.draw_geometries([pcd], zoom=0.5)
  56. o3d.visualization.draw_geometries([pcd])

（2）效果还是很不错的，图就不上了，完结撒花

声明：本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号