Apache DolphinScheduler(海豚调度) - 1.3 系列核心表结构剖析

Apache DolphinScheduler 是一个分布式去中心化，易扩展的可视化 DAG 工作流任务调度系统。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。

近日，伯毅同学给社区贡献了工作流核心表结构的剖析文章，非常细致，喜欢的伙伴请转走

1. 工作流总体存储结构

在 dolphinscheduler 库中创建的所有工作流定义(模板)都保存在 t_ds_process_definition 表中.

该数据库表结构如下表所示:

其中 process_definition_json 字段为核心字段, 定义了 DAG 图中的任务信息.该数据以JSON 的方式进行存储.

公共的数据结构如下表:

数据示例:

{
   
    "globalParams":[
        {
   
            "prop":"golbal_bizdate",
            "direct":"IN",
            "type":"VARCHAR",
            "value":"${system.biz.date}"
        }
    ],
    "tasks":Array[1],
    "tenantId":0,
    "timeout":0
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

2. 各任务类型存储结构详解

2.1 Shell 节点

** Shell 节点数据结构如下:**

Shell 节点数据样例:

{
   
    "type":"SHELL",
    "id":"tasks-80760",
    "name":"Shell Task",
    "params":{
   
        "resourceList":[
            {
   
                "id":3,
                "name":"run.sh",
                "res":"run.sh"
            }
        ],
        "localParams":[
        ],
        "rawScript":"echo "This is a shell script""
    },
    "description":"",
    "runFlag":"NORMAL",
    "conditionResult":{
   
        "successNode":[
            ""
        ],
        "failedNode":[
            ""
        ]
    },
    "dependence":{
   
    },
    "maxRetryTimes":"0",
    "retryInterval":"1",
    "timeout":{
   
        "strategy":"",
        "interval":null,
        "enable":false
    },
    "taskInstancePriority":"MEDIUM",
    "workerGroup":"default",
    "preTasks":[
    ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

2.2 SQL节点

通过 SQL 对指定的数据源进行数据查询、更新操作.

** SQL 节点数据结构如下:**

** SQL 节点数据样例:**

{
   
    "type":"SQL",
    "id":"tasks-95648",
    "name":"SqlTask-Query",
    "params":{
   
        "type":"MYSQL",
        "datasource":1,
        "sql":"select id , namge , age from emp where id =  ${id}",
        "udfs":"",
        "sqlType":"0",
        "title":"xxxx@xxx.com",
        "receivers":"xxxx@xxx.com",
        "receiversCc":"",
        "showType":"TABLE",
        "localParams":[
            {
   
                "prop":"id",
                "direct":"IN",
                "type":"INTEGER",
                "value":"1"
            }
        ],
        "connParams":"",
        "preStatements":[
            "insert into emp ( id,name ) value (1,'Li' )"
        ],
        "postStatements":[
        ]
    },
    "description":"",
    "runFlag":"NORMAL",
    "conditionResult":{
   
        "successNode":[
            ""
        ],
        "failedNode":[
            ""
        ]
    },
    "dependence":{
   
    },
    "maxRetryTimes":"0",
    "retryInterval":"1",
    "timeout":{
   
        "strategy":"",
        "interval":null,
        "enable":false
    },
    "taskInstancePriority":"MEDIUM",
    "workerGroup":"default",
    "preTasks":[
    ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

2.3 Spark 节点

** Spark 节点数据结构如下:**

<