1.GraspNet-API之Grasp Lable Format_graspnetapi

GraspNet

1.1 Raw Lable Format （原始标签格式）

原始标签由两部分组成，每个物体的所有候选抓取标签and每个场景的collision masks。

（1）Labels on Objects

The raw label on each object is a list of numpy arrays.
每个物体的初始标签均为一个np.arry列表。

import numpy as np
import os

root_dir = '/media/robotics/data/datasets/GraspNet/'

data_dir1 = os.path.join(root_dir, 'grasp_label/000_labels.npz')
label = np.load(data_dir1) # GRASPNET_ROOT/grasp_label/000_labels.npz
print(label.files)                 # ['points', 'offsets', 'collision', 'scores']

print(label['points'].shape)       # (3459, 3)

print(label['offsets'].shape)      # (3459, 300, 12, 4, 3)

print(label['collision'].shape,    # (3459, 300, 12, 4)
      label['collision'].dtype)    # dtype('bool')
      
print(label['scores'].shape,       # (3459, 300, 12, 4)
      label['scores'][0][0][0][0]) # -1.0
      
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Notes：

在原始标签中，grasp的分数越低越好，
但是，score = -1.0表示grasp pose完全不能被接受。

（2）Collision Masks on Each Scene

每个场景的collision mask均为一个np.arry列表。

import numpy as np

data_dir2 = os.path.join(root_dir, 'collision_label/scene_0000/collision_labels.npz')
c_labels = np.load(data_dir2) # GRASPNET_ROOT/collision_label/scene_0000/collision_labels.npz
print(c_labels.files)
#['arr_0', 'arr_4', 'arr_5', 'arr_2', 'arr_3', 'arr_7', 'arr_1', 'arr_8', 'arr_6']

print(c_labels['arr_0'].shape,      # (487, 300, 12, 4)
      c_labels['arr_0'].dtype)      # dtype('bool')
print(c_labels['arr_0'][10][20][3]) # array([ True, True, True, True])


1
2
3
4
5
6
7
8
9
10
11
12

‘arr_i’是每个场景的object_id_list.txt中第i个物体的collision mask，其shape = (num points, 300, 12, 4)。
num_points, 300, 12, 4分别为点数、视角ID、内旋ID、深度ID。

更多细节请参考：graspnetAPI.GraspNet.loadGrasp()

1.2 API Loaded Labels

通过调用graspnetAPI.GraspNet.loadGrasp()，可以方便的get场景中的所以grasp labels、params、scores.

加载grasp labels有四种数据结构：Grasp, GraspGroup, RectGrasp and RectGraspGroup.
每个类的内部数据格式是np.arry数组，这比Python列表运算更高效。
它们的定义见 grasp.py

（1）Labels示例

加载一个GraspGroup实例

from graspnetAPI import GraspNet
import open3d as o3d
import cv2

# loading grasp for a scene.
graspnet_root = '/media/robotics/data/datasets/GraspNet' # ROOT PATH FOR GRASPNET
sceneId = 1
annId = 3

# 1.initialize a GraspNet instance
g = GraspNet(graspnet_root, camera='kinect', split='test_novel')

# load grasps of scene 1 with annotation id = 3, camera = kinect and fric_coef_thresh = 0.2
_6d_grasp = g.loadGrasp(sceneId = sceneId, annId = annId, format = '6d', camera = 'kinect', fric_coef_thresh = 0.2)
print('6d grasp:\n{}'.format(_6d_grasp))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Grasp Group, Number=90332:

Grasp: score:0.9000000357627869, width:0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, translation:[-0.09166837 -0.16910084  0.39480919]
rotation:
[[-0.81045675 -0.57493848  0.11227506]
 [ 0.49874267 -0.77775514 -0.38256073]
 [ 0.30727136 -0.25405255  0.91708326]]
object id:66

Grasp: score:0.9000000357627869, width:0.10030215978622437, height:0.019999999552965164, depth:0.019999999552965164, translation:[-0.09166837 -0.16910084  0.39480919]
rotation:
[[-0.73440629 -0.67870212  0.0033038 ]
 [ 0.64608938 -0.70059127 -0.3028869 ]
 [ 0.20788456 -0.22030747  0.95302087]]
object id:66

Grasp: score:0.9000000357627869, width:0.08487851172685623, height:0.019999999552965164, depth:0.019999999552965164, translation:[-0.10412319 -0.13797761  0.38312319]
rotation:
[[ 0.03316294  0.78667933 -0.61647028]
 [-0.47164679  0.55612749  0.68430358]
 [ 0.88116372  0.26806271  0.38947761]]
object id:66
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

通过index or slice访问元素

# index（索引）
print('_6d_grasp[0](grasp):\n{}'.format(_6d_grasp[0]))
# slice（切片）
print('_6d_grasp[0:2]:\n{}'.format(_6d_grasp[0:2]))
print('_6d_grasp[[0,1]]:\n{}'.format(_6d_grasp[[0,1]]))
1
2
3
4
5

_6d_grasp[0] (grasp):

Grasp: score:0.9000000357627869, width:0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, translation:[-0.09166837 -0.16910084  0.39480919]
rotation:
[[-0.81045675 -0.57493848  0.11227506]
 [ 0.49874267 -0.77775514 -0.38256073]
 [ 0.30727136 -0.25405255  0.91708326]]
object id:66
1
2
3
4
5
6

_6d_grasp[0:2]:

Grasp Group, Number=2:

Grasp: score:0.9000000357627869, width:0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, translation:[-0.09166837 -0.16910084  0.39480919]
rotation:
[[-0.81045675 -0.57493848  0.11227506]
 [ 0.49874267 -0.77775514 -0.38256073]
 [ 0.30727136 -0.25405255  0.91708326]]
object id:66

Grasp: score:0.9000000357627869, width:0.10030215978622437, height:0.019999999552965164, depth:0.019999999552965164, translation:[-0.09166837 -0.16910084  0.39480919]
rotation:
[[-0.73440629 -0.67870212  0.0033038 ]
 [ 0.64608938 -0.70059127 -0.3028869 ]
 [ 0.20788456 -0.22030747  0.95302087]]
object id:66
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

_6d_grasp[[0,1]]:

Grasp Group, Number=2:

Grasp: score:0.9000000357627869, width:0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, translation:[-0.09166837 -0.16910084  0.39480919]
rotation:
[[-0.81045675 -0.57493848  0.11227506]
 [ 0.49874267 -0.77775514 -0.38256073]
 [ 0.30727136 -0.25405255  0.91708326]]
object id:66

Grasp: score:0.9000000357627869, width:0.10030215978622437, height:0.019999999552965164, depth:0.019999999552965164, translation:[-0.09166837 -0.16910084  0.39480919]
rotation:
[[-0.73440629 -0.67870212  0.0033038 ]
 [ 0.64608938 -0.70059127 -0.3028869 ]
 [ 0.20788456 -0.22030747  0.95302087]]
object id:66
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

GraspGroup的每个元素都是一个Grasp实例，
Grasp的属性可以通过提供的方法访问

# grasp is a Grasp instance defined in grasp.py
# access and set properties.
grasp = _6d_grasp[0]

print('grasp.translation={}'.format(grasp.translation))

grasp.translation = np.array([1.0, 2.0, 3.0])
print('After modification, grasp.translation={}'.format(grasp.translation))
print('grasp.rotation_matrix={}'.format(grasp.rotation_matrix))

grasp.rotation_matrix = np.eye(3).reshape((9))
print('After modification, grasp.rotation_matrix={}'.format(grasp.rotation_matrix))
print('grasp.width={}, height:{}, depth:{}, score:{}'.format(grasp.width, grasp.height, grasp.depth, grasp.score))
1
2
3
4
5
6
7
8
9
10
11
12
13

grasp.translation=[-0.09166837 -0.16910084  0.39480919]
After modification, grasp.translation=[1. 2. 3.]

grasp.rotation_matrix=[[-0.81045675 -0.57493848  0.11227506]
 [ 0.49874267 -0.77775514 -0.38256073]
 [ 0.30727136 -0.25405255  0.91708326]]
After modification, grasp.rotation_matrix=[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
 
grasp.width=0.11247877031564713, height:0.019999999552965164, depth:0.029999999329447746, score:0.9000000357627869
1
2
3
4
5
6
7
8
9
10
11

RectGrasp是矩形抓取的类，
格式与Grasp不同，但提供的API类似。

# load rectangle grasps of scene 1 with annotation id = 3, camera = realsense and fric_coef_thresh = 0.2
rect_grasp_group = g.loadGrasp(sceneId = sceneId, annId = annId, format = 'rect', camera = 'realsense', fric_coef_thresh = 0.2)
print('rectangle grasp group:\n{}'.format(rect_grasp_group))

# rect_grasp is an RectGraspGroup instance defined in grasp.py
print('rect_grasp_group:\n{}'.format(rect_grasp_group))

# index
rect_grasp = rect_grasp_group[0]
print('rect_grasp_group[0](rect_grasp):\n{}'.format(rect_grasp))

# slice
print('rect_grasp_group[0:2]:\n{}'.format(rect_grasp_group[0:2]))
print('rect_grasp_group[[0,1]]:\n{}'.format(rect_grasp_group[[0,1]]))

# properties of rect_grasp
print('rect_grasp.center_point:{}, open_point:{}, height:{}, score:{}'.format(rect_grasp.center_point, rect_grasp.open_point, rect_grasp.height, rect_grasp.score))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

rectangle grasp group:
----------
Rectangle Grasp Group, Number=29036:
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.40000003576278687, height:52.5867805480957, open point:(682.6998, -85.73743), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.6000000238418579, height:52.63461685180664, open point:(649.1359, -68.900116), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.30000001192092896, height:52.629859924316406, open point:(651.5441, -67.304474), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.800000011920929, height:52.84688186645508, open point:(625.3405, -55.22821), center point:(560.52704, -127.61005), object id:66
......
Rectangle Grasp: score:0.30000001192092896, height:40.82650375366211, open point:(613.7542, 459.35962), center point:(668.34296, 521.0328), object id:14
Rectangle Grasp: score:0.40000003576278687, height:40.82828140258789, open point:(749.2127, 549.03204), center point:(668.34296, 521.0328), object id:14
Rectangle Grasp: score:0.6000000238418579, height:39.991939544677734, open point:(584.6618, 410.68518), center point:(658.4677, 494.0827), object id:14
Rectangle Grasp: score:0.5, height:39.60657501220703, open point:(596.4554, 400.3207), center point:(670.0193, 483.46442), object id:14
Rectangle Grasp: score:0.40000003576278687, height:39.93021774291992, open point:(602.8676, 408.1247), center point:(673.44824, 487.8963), object id:14
----------


rect_grasp_group:
----------
Rectangle Grasp Group, Number=29036:
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.40000003576278687, height:52.5867805480957, open point:(682.6998, -85.73743), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.6000000238418579, height:52.63461685180664, open point:(649.1359, -68.900116), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.30000001192092896, height:52.629859924316406, open point:(651.5441, -67.304474), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.800000011920929, height:52.84688186645508, open point:(625.3405, -55.22821), center point:(560.52704, -127.61005), object id:66
......
Rectangle Grasp: score:0.30000001192092896, height:40.82650375366211, open point:(613.7542, 459.35962), center point:(668.34296, 521.0328), object id:14
Rectangle Grasp: score:0.40000003576278687, height:40.82828140258789, open point:(749.2127, 549.03204), center point:(668.34296, 521.0328), object id:14
Rectangle Grasp: score:0.6000000238418579, height:39.991939544677734, open point:(584.6618, 410.68518), center point:(658.4677, 494.0827), object id:14
Rectangle Grasp: score:0.5, height:39.60657501220703, open point:(596.4554, 400.3207), center point:(670.0193, 483.46442), object id:14
Rectangle Grasp: score:0.40000003576278687, height:39.93021774291992, open point:(602.8676, 408.1247), center point:(673.44824, 487.8963), object id:14
----------


rect_grasp_group[0](rect_grasp):
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66


rect_grasp_group[0:2]:
----------
Rectangle Grasp Group, Number=2:
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.40000003576278687, height:52.5867805480957, open point:(682.6998, -85.73743), center point:(560.52704, -127.61005), object id:66
----------


rect_grasp_group[[0,1]]:
----------
Rectangle Grasp Group, Number=2:
Rectangle Grasp: score:0.40000003576278687, height:52.59913635253906, open point:(677.3685, -87.56467), center point:(560.52704, -127.61005), object id:66
Rectangle Grasp: score:0.40000003576278687, height:52.5867805480957, open point:(682.6998, -85.73743), center point:(560.52704, -127.61005), object id:66
----------


rect_grasp.center_point:(560.52704, -127.61005), open_point:(677.3685, -87.56467), height:52.59913635253906, score:0.40000003576278687

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

（2）6D Grasp

17 个浮点数用于定义通用的 6d grasp，width、height、 depth、 score and 附加object id也是定义的一部分。

Notes：
抓取的分数越高越好，这与原始标签不同。
实际上，score = 1.1 - raw_score（摩擦系数）

class Grasp():
    def __init__(self, *args):
        '''
        **Input:**

        - args can be a numpy array or tuple of the score, width, height, depth, rotation_matrix, translation, object_id

        - the format of numpy array is [score, width, height, depth, rotation_matrix(9), translation(3), object_id]

        - the length of the numpy array is 17.
        '''
        if len(args) == 0:
            self.grasp_array = np.array([0, 0.02, 0.02, 0.02, 1, 0, 0, 0, 1 ,0 , 0, 0, 1, 0, 0, 0, -1], dtype = np.float64)
        elif len(args) == 1:
            if type(args[0]) == np.ndarray:
                self.grasp_array = copy.deepcopy(args[0])
            else:
                raise TypeError('if only one arg is given, it must be np.ndarray.')
        elif len(args) == 7:
            score, width, height, depth, rotation_matrix, translation, object_id = args
            self.grasp_array = np.concatenate([np.array((score, width, height, depth)),rotation_matrix.reshape(-1), translation, np.array((object_id)).reshape(-1)]).astype(np.float64)
        else:
            raise ValueError('only 1 or 7 arguments are accepted')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

（3）6D Grasp Group

通常，一个场景中有很多grasps，这些抓点组成一个类GraspGroup。
相比Grasp，GraspGroup包含一个二维 numpy 数组，额外的维度是每个grasp的索引。

class GraspGroup():
    def __init__(self, *args):
        '''
        **Input:**

        - args can be (1) nothing (2) numpy array of grasp group array (3) str of the npy file.
        '''
        if len(args) == 0:
            self.grasp_group_array = np.zeros((0, GRASP_ARRAY_LEN), dtype=np.float64)
        elif len(args) == 1:
            if isinstance(args[0], np.ndarray):
                self.grasp_group_array = args[0]
            elif isinstance(args[0], str):
                self.grasp_group_array = np.load(args[0])
            else:
                raise ValueError('args must be nothing, numpy array or string.')
        else:
            raise ValueError('args must be nothing, numpy array or string.')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

实现了对列表的常见操作，例如s indexing, slicing and sorting。
此外，一个重要的功能是用户可以将GraspGroup转储到一个 numpy 文件中，并通过调用GraspGroup.save_npy()和GraspGroup.from_npy()将其加载到另一个程序中。

（4）Rectangle Grasp

7个浮点数用于定义通用的矩形抓取，即the center point, the open point, height, score and the
attached object id.。每个参数的详细定义如上图所示，中心点和开放点的坐标在像素框中。

class RectGrasp():
    def __init__(self, *args):
        '''
        **Input:**

        - args can be a numpy array or tuple of the center_x, center_y, open_x, open_y, height, score, object_id

        - the format of numpy array is [center_x, center_y, open_x, open_y, height, score, object_id]

        - the length of the numpy array is 7.
        '''
        if len(args) == 1:
            if type(args[0]) == np.ndarray:
                self.rect_grasp_array = copy.deepcopy(args[0])
            else:
                raise TypeError('if only one arg is given, it must be np.ndarray.')
        elif len(args) == RECT_GRASP_ARRAY_LEN:
            self.rect_grasp_array = np.array(args).astype(np.float64)
        else:
            raise ValueError('only one or six arguments are accepted')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

（5）Rectangle Grasp Group

RectGraspGroup的格式类似于RectGrasp和GraspGroup

class RectGraspGroup():
    def __init__(self, *args):
        '''
        **Input:**

        - args can be (1) nothing (2) numpy array of rect_grasp_group_array (3) str of the numpy file.
        '''
        if len(args) == 0:
            self.rect_grasp_group_array = np.zeros((0, RECT_GRASP_ARRAY_LEN), dtype=np.float64)
        elif len(args) == 1:
            if isinstance(args[0], np.ndarray):
                self.rect_grasp_group_array = args[0]
            elif isinstance(args[0], str):
                self.rect_grasp_group_array = np.load(args[0])
            else:
                raise ValueError('args must be nothing, numpy array or string.')
        else:
            raise ValueError('args must be nothing, numpy array or string.')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

1.3 Grasp and GraspGroup 转换

可以转换 Grasp 或 GraspGroup 得到 4x4 矩阵。

from graspnetAPI import *
# transform grasp
g = Grasp() # simple Grasp
frame = o3d.geometry.TriangleMesh.create_coordinate_frame(0.1)

# Grasp before transformation
o3d.visualization.draw_geometries([g.to_open3d_geometry(), frame])
g.translation = np.array((0,0,0.01))

# setup a transformation matrix
T = np.eye(4)
T[:3,3] = np.array((0.01, 0.02, 0.03))
T[:3,:3] = np.array([[0,0,1.0],[1,0,0],[0,1,0]])
g.transform(T)

# Grasp after transformation
o3d.visualization.draw_geometries([g.to_open3d_geometry(), frame])

g1 = Grasp()
gg = GraspGroup()
gg.add(g)
gg.add(g1)

# GraspGroup before transformation
o3d.visualization.draw_geometries([*gg.to_open3d_geometry_list(), frame])

gg.transform(T)

# GraspGroup after transformation
o3d.visualization.draw_geometries([*gg.to_open3d_geometry_list(), frame])
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30