Dolphinscheduler使用Datax同步mongodb到doris_dolphinscheduler 集成datax

作者：不正经 | 2024-02-25 13:59:35

踩

dolphinscheduler 集成datax

1、dolphinscheduler与datax集成环境变量配置


export JAVA_HOME=/opt/soft/java
export  PYTHON_HOME=/usr/bin/python
export  DATAX_HOME=/opt/soft/datax
export  PATH=$PYTHON_HOEM:$DATAX_HOME/bin:$JAVA_HOME/bin:$PATH

截图如下:

注意服务器上安装python2.7，以及datax安装在对应目录下。

2、同步mongodb 到doris


{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "mongodbreader", 
                    "parameter": {
                        "address": ["xxx:27017","xxxx:27017","xxxx:27017"], 
                        "collectionName": "leases", 
                        "column": [
                        {
                        "name":"_id",
                        "type":"string"
                        },
                        {
                        "name":"id",
                        "type":"string"
                        },
                        {
                        "name":"storeid",
                        "type":"string"
                        },
                        {
                        "name":"storename",
                        "type":"string"
                        },
                        {
                        "name":"title",
                        "type":"string"
                        },
                        {
                        "name":"source",
                        "type":"string"
                        },
                        {
                        "name":"code",
                        "type":"string"
                        },
                        {
                        "name":"codeyear",
                        "type":"string"
                        },
                        {
                        "name":"codemonth",
                        "type":"string"
                        },
                        {
                        "name":"codesort",
                        "type":"string"
                        },
                        {
                        "name":"classify",
                        "type":"string"
                        },
                        {
                        "name":"property",
                        "type":"string"
                        },
                        {
                        "name":"propertydir",
                        "type":"string"
                        },
                        {
                        "name":"leaseduration",
                        "type":"string"
                        },
                        {
                        "name":"leasestartdate",
                        "type":"string"
                        },
                        {
                        "name":"leaseenddate",
                        "type":"string"
                        },
                        {
                        "name":"calcacreage",
                        "type":"string"
                        },
                        {
                        "name":"acreage",
                        "type":"string"
                        },
                        {
                        "name":"insidespace",
                        "type":"string"
                        },
                        {
                        "name":"residentialpoolarea",
                        "type":"string"
                        },
                        {
                        "name":"handoverdate",
                        "type":"string"
                        },
                        {
                        "name":"billingdate",
                        "type":"string"
                        },
                        {
                        "name":"deposit",
                        "type":"string"
                        },
                        {
                        "name":"rental",
                        "type":"string"
                        },
                        {
                        "name":"otherdeposit",
                        "type":"string"
                        },
                        {
                        "name":"quitedat",
                        "type":"string"
                        },
                        {
                        "name":"quitamount",
                        "type":"string"
                        }
                        ], 
                        "dbName": "xxxx", 
                        "authDb":"admin",
                        "userName": "xxxx", 
                        "userPassword": "xxxx"
                    }
                }, 
                "writer": {
                    "name": "doriswriter", 
                    "parameter": {
                        "column": ["_id","id","storeid","storename","title","source","code","codeyear","codemonth","codesort","classify","property","propertydir","leaseduration","leasestartdate","leaseenddate","calcacreage","acreage","insidespace","residentialpoolarea","handoverdate","billingdate","deposit","rental","otherdeposit","quitedat","quitamount"], 
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:mysql://xxxx:9030/ODS", 
                                "selectedDatabase":"ODS",
                                "table": ["test"]
                            }
                        ], 
                        "loadProps":{},
                        "loadUrl":["xxxx"],
                        "password": "****",
                        "postSql":[],  
                        "username": "root", 
                    }
                }
            }
        ], 
        "setting": {
            "speed": {
                "channel": "1"
            }
        }
    }
}

3、mogodb中json格式的集合，datax可用如下方式获取属性


                       {
                        "name":"tenant.company_phone",
                        "type":"document.string"
                        },

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/不正经/article/detail/141189