从CSV到数据库(简易)_通过csv建数据库

需求：客户上传CSV文档，要求CSV文档内容查重/插入/更新相关数据。
框架：jdbcTemplate、commons-io、
DB：oracle

相关依赖：
这里本来打算用的2.11.0，无奈正式项目那边用老版本1.3.1，新版本对类型支持和转换好一点。不过无伤大雅。

        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>1.3.1</version>
        </dependency>
1
2
3
4
5

CSV文档格式：

Xxx Code,Yerial,OP
600001,2024082400305, OP20240818_XDFD
600001,2024082400306, OP20240818_XDFD
600001,2024082400307, OP20240818_XDFD
600001,2024082400308, OP20240818_XDFD
600001,2024082400309, OP20240818_XDFD
600001,2024082400310, OP20240818_XDFD
600001,2024082400311, OP20240818_XDFD
600001,2024082400312, OP20240818_XDFD
600001,2024082400313, OP20240818_XDFD
600001,2024082400314, OP20240818_XDFD
600001,2024082400315, OP20240818_XDFD
600001,2024082400316, OP20240818_XDFD
600001,2024082400317, OP20240818_XDFD
600001,2024082400318, OP20240818_XDFD
600001,2024082400319, OP20240818_XDFD
600001,2024082400320, OP20240818_XDFD
600001,2024082400321, OP20240818_XDFD
600001,2024082400322, OP20240818_XDFD
600001,2024082400323, OP20240818_XDFD
600001,2024082400324, OP20240818_XDFD
600001,2024082400325, OP20240818_XDFD
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

接口：

用MultipartFile接收CSV文件

    @PostMapping("/import")
    public ResponseEntity<BaseResponse<?>> importCSV(@RequestBody MultipartFile files) {
        xxxService.importSerial(files);
        return new ResponseEntity<>(new BaseResponse<>(), HttpStatus.OK);
    }
1
2
3
4
5

前端需要在Request-Body中以form-data的形式上传文档，注意参数名要对上。

CSV解析

简单转成Json

    private String convertCsvToJson(MultipartFile multipartFile) throws IOException {
        // read csv as list
        List<String> lines = IOUtils.readLines(multipartFile.getInputStream(), "UTF-8");

        List<List<String>> data = lines.stream()
                .skip(1) // skip label line
                .filter(line -> !line.trim().isEmpty()) // filtering empty line
                .map(line -> Arrays.asList(line.split(",")))
                .collect(Collectors.toList());

        return objectMapper.writeValueAsString(data);
    }
1
2
3
4
5
6
7
8
9
10
11
12

转换后是这样的：
注意下面是 copy 的 print 出来的 json 对象，实际应该是 json 字符串。

[
    ["600001","2024082400305","OP20240818_XDFD"],
    ["600001","2024082400306","OP20240818_XDFD"],
    ["600001","2024082400307","OP20240818_XDFD"],
    ["600001","2024082400308","OP20240818_XDFD"],
    ["600001","2024082400309","OP20240818_XDFD"],
    ["600001","2024082400310","OP20240818_XDFD"],
    ["600001","2024082400311","OP20240818_XDFD"],
    ["600001","2024082400312","OP20240818_XDFD"],
    ["600001","2024082400313","OP20240818_XDFD"],
    ["600001","2024082400314","OP20240818_XDFD"],
    ["600001","2024082400315","OP20240818_XDFD"],
    ["600001","2024082400316","OP20240818_XDFD"],
    ["600001","2024082400317","OP20240818_XDFD"],
    ["600001","2024082400318","OP20240818_XDFD"],
    ["600001","2024082400319","OP20240818_XDFD"],
    ["600001","2024082400320","OP20240818_XDFD"],
    ["600001","2024082400321","OP20240818_XDFD"],
    ["600001","2024082400322","OP20240818_XDFD"],
    ["600001","2024082400323","OP20240818_XDFD"],
    ["600001","2024082400324","OP20240818_XDFD"],
    ["600001","2024082400325","OP20240818_XDFD"]
]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

以Clob参数的形式传到oracle存储过程中处理

这里用Prepare-Call，Simple-Jdbc-Call对 Clob 的支持不好。

    public void import(MultipartFile files) {
        final int[] status = new int[1];
        Object result = jdbcTemplate.execute(new ConnectionCallback<Object>() {
            @Override
            public Object doInConnection(Connection con) throws SQLException, DataAccessException {
                CallableStatement cs = con.prepareCall("{call TEST_PACKAGE.pro_add_csv_data(?, ?)}");
                Clob clob = con.createClob(); // 创建一个Clob对象
                try {
                    String s = convertCsvToJson(files); // csv 转 json字符串
                    clob.setString(1, s); // 把json字符串封装进clob对象中
                    cs.setClob(1, clob);  // 入参
                    cs.registerOutParameter(2, Types.INTEGER); // 出参
                    cs.execute();
                    status[0] = cs.getInt(2); // 取结果
                } catch (IOException e) {
                    throw new RuntimeException("Import failed.");
                }
                return null;
            }
        });
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

存储过程

PROCEDURE pro_add_csv_data(v_data_list IN CLOB,v_status OUT NUMBER) 
AS 
    v_code VARCHAR2(10);  
    v_yerial VARCHAR2(20);
    v_op VARCHAR2(20);
    v_cur SYS_REFCURSOR;
BEGIN    
	-- 解析成json表
    OPEN v_cur FOR    
        SELECT jt.*      
        FROM (    
            SELECT j.*    
            FROM JSON_TABLE(    
                v_data_list , '$[*]' COLUMNS (
                    code VARCHAR2(10) PATH '$[0]',        
                    yerial VARCHAR2(20) PATH '$[1]',        
                    op VARCHAR2(20) PATH '$[2]'        
                )              
            ) j    
        ) jt;  
    -- 遍历操作
    LOOP FETCH v_cur INTO v_code, v_yerial, v_op;  
    EXIT WHEN v_cur%NOTFOUND;
            INSERT INTO table_one
            VALUES(v_code, v_yerial, v_op);
    END LOOP;
    
    COMMIT;
    CLOSE v_cur;
    v_status := 0;
EXCEPTION  
	WHEN OTHERS THEN
        ROLLBACK;
        v_status:=1;
END pro_add_csv_data;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

完美