devbox
很楠不爱3
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Amazon S3 使用Java sdk实现文件分段上传 后台+前端Vue_aws sdk for java 分片上传

作者：很楠不爱3 | 2024-03-22 06:06:14

赞

踩

aws sdk for java 分片上传

后台

官方文档

一，s3 sdk 依赖

<dependency>
	<groupId>com.amazonaws</groupId>
	<artifactId>aws-java-sdk</artifactId>
	<version>1.11.126</version>
</dependency>

  • 1
  • 2
  • 3
  • 4
  • 5

二，上传工具类

package test.s3;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import java.io.InputStream;
import java.net.URL;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;

public class AmazonS3Util {

    /**
     * S3服务器地址
     */
    private final static String S3_URL = "";

    /**
     * 账号key
     */
    private final static String ACCESS_KEY = "";

    /**
     * 密码key
     */
    private final static String SECRET_KEY = "";

    /**
     * 存放合并信息集合 如果服务是集群或多活部署，需改成redis存储
     */
    private final static Map<String, Map<String, Object>> uploadIdMap = new ConcurrentHashMap<>();

    private static Logger logger = LoggerFactory.getLogger(AmazonS3Util.class);

    /**
     * 获取连接
     * @return
     * @throws Exception
     */
    public static AmazonS3 connectS3Client() throws Exception {
        URL url = new URL(S3_URL);
        ClientConfiguration config = new ClientConfiguration().withProtocol(Protocol.HTTP);
        AmazonS3 client = AmazonS3ClientBuilder.standard()
                .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(url.toString(),Regions.CN_NORTH_1.toString()))
                .withClientConfiguration(config)
                .withPathStyleAccessEnabled(true)
                .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY)))
                .build();
        return client;
    }

    /**
     * 上传文件
     * @param upload
     * @return
     * @throws Exception
     */
    public static void uploadFile(Upload upload) throws Exception {
        AmazonS3 s3client = connectS3Client();
        InputStream inputStream = null;
        try {
            String fileKey = getFileKey(upload.getMultipartFile().getOriginalFilename());
            inputStream = upload.getMultipartFile().getInputStream();
            PutObjectRequest request = new PutObjectRequest(upload.getBucketName(), fileKey, inputStream,null);
            s3client.putObject(request);
        } catch (Exception e) {
            throw e;
        } finally {
            if (inputStream != null) {
                try {
                    inputStream.close();
                } catch (Exception e) {
                    throw e;
                }
            }
        }
    }

    /**
     * 分段上传文件
     * @param upload
     * @return
     * @throws Exception
     */
    public static String mutilpartUpload(Upload upload) throws Exception {
        logger.info("开始上传");
        AmazonS3 s3client = connectS3Client();
        InputStream inputStream = null;
        try {
            String fileKey = null;
            String uploadId = null;
            List<PartETag> partETagList = new ArrayList<>();
            Map<String, Object> result = uploadIdMap.get(upload.getIdentifier());
            if (result == null) {
                fileKey = getFileKey(upload.getMultipartFile().getOriginalFilename());
                InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(upload.getBucketName(), fileKey);
                InitiateMultipartUploadResult initResponse = s3client.initiateMultipartUpload(initRequest);
                uploadId = initResponse.getUploadId();
                result = new HashMap<>();
                result.put("uploadId", uploadId);
                result.put("partETagList", partETagList);
                result.put("fileKey", fileKey);
            } else {
                uploadId = result.get("uploadId").toString();
                partETagList = (List<PartETag>) result.get("partETagList");
                fileKey = result.get("fileKey").toString();
            }

            // 分段上传
            inputStream = upload.getMultipartFile().getInputStream();
            UploadPartRequest uploadRequest = new UploadPartRequest()
                    .withBucketName(upload.getBucketName())
                    .withKey(fileKey)
                    .withUploadId(uploadId)
                    .withPartNumber(upload.getChunkNumber())
                    .withInputStream(inputStream)
                    .withPartSize(upload.getMultipartFile().getSize());
            UploadPartResult uploadResult = s3client.uploadPart(uploadRequest);

            partETagList.add(uploadResult.getPartETag());

            logger.info("上传完成: upload.getTotalChunks()：{}, upload.getChunkNumber()：{}", upload.getTotalChunks(), upload.getChunkNumber());
            if (upload.getTotalChunks() == upload.getChunkNumber()) {
                logger.info("开始合并");
                CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(upload.getBucketName(), fileKey, uploadId, partETagList);
                s3client.completeMultipartUpload(compRequest);
                // 删除map数据
                uploadIdMap.remove(upload.getIdentifier());
            } else {
                // 更新map数据
                uploadIdMap.put(upload.getIdentifier(), result);
            }
            return fileKey;
        } catch (Exception e) {
            throw e;
        } finally {
            if (inputStream != null) {
                try {
                    inputStream.close();
                } catch (Exception e) {
                    throw e;
                }
            }
        }
    }

    /**
     * 下载
     * @param bucketName
     * @param fileKey
     * @return
     * @throws Exception
     */
    public static InputStream downloadFile(String bucketName, String fileKey) throws Exception{
        AmazonS3 s3client = connectS3Client();
        GetObjectRequest request = new GetObjectRequest(bucketName, fileKey);
        S3Object response = s3client.getObject(request);
        InputStream input = response.getObjectContent();
        return input;
    }

    /**
     * 删除文件
     * @param bucketName
     * @param fileKey
     * @throws Exception
     */
    public static void deleteFile(String bucketName, String fileKey) throws Exception {
        AmazonS3 s3client = connectS3Client();
        DeleteObjectRequest request = new DeleteObjectRequest(bucketName, fileKey);
        s3client.deleteObject(request);
    }

    /**
     * Bucket列表
     * @return
     * @throws Exception
     */
    public static List<Bucket> listFile() throws  Exception{
        AmazonS3 s3client = connectS3Client();
        ListBucketsRequest request = new ListBucketsRequest();
        List<Bucket> list = s3client.listBuckets(request);
        return list;
    }

    /**
     * 是否存在Bucket
     * @param bucketName
     * @return
     * @throws Exception
     */
    public static boolean isExistBucket(String bucketName) throws Exception{
        AmazonS3 s3client = connectS3Client();
        try {
            HeadBucketRequest request = new HeadBucketRequest(bucketName);
            s3client.headBucket(request);
        } catch (Exception e) {
            return false;
        }
        return true;
    }

    /**
     * 创建Bucket
     * @param bucketName
     * @return
     * @throws Exception
     */
    public static void createBucket(String bucketName) throws Exception{
        boolean isBucketExists = isExistBucket(bucketName);
        AmazonS3 s3client = null;
        if (!isBucketExists) {
            s3client = connectS3Client();
            CreateBucketRequest request = new CreateBucketRequest(bucketName);
            s3client.createBucket(request);
        }
    }

    /**
     * 删除Bucket
     * @param bucketName
     * @return
     * @throws Exception
     */
    public static void deleteBucket(String bucketName) throws Exception {
        AmazonS3 s3client = connectS3Client();
        DeleteBucketRequest request = new DeleteBucketRequest(bucketName);
        s3client.deleteBucket(request);
    }

    /**
     * fileKey是否存在
     * @param bucketName
     * @param fileKey
     * @return
     * @throws Exception
     */
    public static boolean isExistFileKey(String bucketName, String fileKey) throws  Exception{
        AmazonS3 s3client = connectS3Client();
        GetObjectRequest request = new GetObjectRequest(bucketName,fileKey);
        S3Object response = s3client.getObject(request);
        boolean result =  response != null &&  fileKey.equals(response.getKey());
        return result;
    }

    /**
     * 获取文件key
     * @param fileName
     * @return
     * @throws Exception
     */
    private static String getFileKey(String fileName) throws Exception{
        String uuid = UUID.randomUUID().toString().replaceAll("-","");
        String dateDir = new SimpleDateFormat("yyyy-MM-dd").format(new Date());
        String fileKey = dateDir + "/" + uuid + "/" + fileName;
        return fileKey;
    }

}


  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140
  • 141
  • 142
  • 143
  • 144
  • 145
  • 146
  • 147
  • 148
  • 149
  • 150
  • 151
  • 152
  • 153
  • 154
  • 155
  • 156
  • 157
  • 158
  • 159
  • 160
  • 161
  • 162
  • 163
  • 164
  • 165
  • 166
  • 167
  • 168
  • 169
  • 170
  • 171
  • 172
  • 173
  • 174
  • 175
  • 176
  • 177
  • 178
  • 179
  • 180
  • 181
  • 182
  • 183
  • 184
  • 185
  • 186
  • 187
  • 188
  • 189
  • 190
  • 191
  • 192
  • 193
  • 194
  • 195
  • 196
  • 197
  • 198
  • 199
  • 200
  • 201
  • 202
  • 203
  • 204
  • 205
  • 206
  • 207
  • 208
  • 209
  • 210
  • 211
  • 212
  • 213
  • 214
  • 215
  • 216
  • 217
  • 218
  • 219
  • 220
  • 221
  • 222
  • 223
  • 224
  • 225
  • 226
  • 227
  • 228
  • 229
  • 230
  • 231
  • 232
  • 233
  • 234
  • 235
  • 236
  • 237
  • 238
  • 239
  • 240
  • 241
  • 242
  • 243
  • 244
  • 245
  • 246
  • 247
  • 248
  • 249
  • 250
  • 251
  • 252
  • 253
  • 254
  • 255
  • 256
  • 257
  • 258
  • 259
  • 260
  • 261
  • 262
  • 263
  • 264
  • 265
  • 266
  • 267
  • 268
  • 269
  • 270

封装参数实体类

package test.s3;

import org.springframework.web.multipart.MultipartFile;

public class Upload {

	/**
	 * 桶名称
	 */
	private String bucketName;

	/**
	 * 总块数
	 */
	private int totalChunks;

	/**
	 * 当前块数，从1开始
	 */
	private int chunkNumber;

	/**
	 * 上传id，每个文件唯一
	 */
	private String identifier;

	/**
	 * 上传文件信息， 需手动赋值
	 */
    private MultipartFile multipartFile;

	public int getTotalChunks() {
		return totalChunks;
	}

	public String getBucketName() {
		return bucketName;
	}

	public void setBucketName(String bucketName) {
		this.bucketName = bucketName;
	}

	public void setTotalChunks(int totalChunks) {
		this.totalChunks = totalChunks;
	}

	public int getChunkNumber() {
		return chunkNumber;
	}

	public void setChunkNumber(int chunkNumber) {
		this.chunkNumber = chunkNumber;
	}

	public String getIdentifier() { return identifier; }

	public void setIdentifier(String identifier) { this.identifier = identifier; }

	public MultipartFile getMultipartFile() {
		return multipartFile;
	}

	public void setMultipartFile(MultipartFile multipartFile) { this.multipartFile = multipartFile; }
}


  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66

控制类

package test.s3;

import io.swagger.annotations.ApiOperation;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletResponse;
import java.io.InputStream;

@RestController
@RequestMapping("/s3")
public class FileS3Controller {

	@ApiOperation("上传")
	@RequestMapping(method = RequestMethod.POST, value = "/upload")
	public String upload(Upload upload, MultipartFile file) throws Exception {
		upload.setMultipartFile(file);
		return AmazonS3Util.mutilpartUpload(upload);
	}

	@ApiOperation("下载")
	@RequestMapping(method = RequestMethod.GET, value = "/download")
	public void download(String bucketName, String fileKey, String fileName, HttpServletResponse response) throws Exception {
		InputStream inputStream = AmazonS3Util.downloadFile(bucketName, fileKey);

		response.setContentType( "application/x-msdownload");
		fileName = new String(fileName.getBytes("gbk"), "iso8859-1");
		response.setHeader("Content-Disposition", "attachment;filename=" + fileName);
		ServletOutputStream out = response.getOutputStream();
		byte[] bffer = new byte[1024];
		int r = 0;
		while ((r = inputStream.read(bffer, 0, 1024)) != -1) {
			out.write(bffer, 0, r);
		}
		try {
			inputStream.close();
		} catch (Exception e) {
			e.printStackTrace();
		}
		try {
			out.close();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	@ApiOperation("删除")
	@RequestMapping(value = "/delete", method = RequestMethod.GET)
	public String delete(String bucketName, String fileKey) throws Exception {
		AmazonS3Util.deleteFile(bucketName, fileKey);
		return  "删除成功";
	}
}


  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56

前端

官方文档

一，组件

npm install vue-simple-uploader --save

  • 1

二，main.js

import Vue from 'vue'
import App from './App.vue'
import uploader from 'vue-simple-uploader'

Vue.config.productionTip = false
Vue.use(uploader)

new Vue({
  render: h => h(App),
}).$mount('#app')


  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11

三，App.vue

<template>
  <uploader :options="options" class="uploader-example">
    <uploader-unsupport></uploader-unsupport>
    <uploader-drop>
      <p>Drop files here to upload or</p>
      <uploader-btn>select files</uploader-btn>
      <uploader-btn :attrs="attrs">select images</uploader-btn>
      <uploader-btn :directory="true">select folder</uploader-btn>
    </uploader-drop>
    <uploader-list></uploader-list>
  </uploader>
</template>

<script>
  export default {
    data () {
      return {
        options: {
          target: '//localhost:8082/s3/upload',
          query: { // 格外参数
              bucketName: 'test'
          },
          chunkSize: 10*1024*1024, // 分块大小
          forceChunkSize: true, // 是否强制所有的块都是小于等于 chunkSize 的值。默认是 false
          simultaneousUploads: 1, // 并发上传数，默认 3
          maxChunkRetries: 3, // 最大自动失败重试上传次数，值可以是任意正整数，如果是 undefined 则代表无限次，默认 0
          testChunks: false
        },
        attrs: {
          accept: 'image/*'
        }
      }
    }
  }
</script>

<style>
  .uploader-example {
    width: 880px;
    padding: 15px;
    margin: 40px auto 0;
    font-size: 12px;
    box-shadow: 0 0 10px rgba(0, 0, 0, .4);
  }
  .uploader-example .uploader-btn {
    margin-right: 4px;
  }
  .uploader-example .uploader-list {
    max-height: 440px;
    overflow: auto;
    overflow-x: hidden;
    overflow-y: auto;
  }
</style>

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54

效果

在这里插入图片描述

注意事项

1、以上代码不支持并发上传。并发上传后续再研究解决
2、如果服务是多活部署（集群部署），AmazonS3Util 工具类里的uploadIdMap要换成redis存储

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/很楠不爱3/article/detail/285781
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号