赞
踩
SpringBoot实现大文件上传/下载(分片、断点续传)
分片:按照自定义缓冲区大小,将大文件分成多个小文件片段。
断点续传:根据分片数量,给每个小文件通过循环起对应名称,当文件下载中断在续传时,判断小文件名称若存在则不存了,此时还需要判断文件若不是最后一个分片则大小为缓冲区固定大小,若没达到则证明小文件没传完需要重新传输。
合并:下载时通过线程池创建任务进行下载或上传、当判断最后一个分片传完时,调用合并方法,根据之前定义的文件名称顺序进行合并,肯能出现最后一个分片传完,之前分片未传完的情况,需要使用while循环进行判断,多文件未传输完,则等待一会继续判断。
大文件秒传:实际上是根据文件名称区一个唯一的md5值存储,传文件时进行判断,若存在则不传。
创建Springboot项目
2.1 选择Spring Initializr(springboot项目)
2.2 配置属性,完成点击next
2.3 选择依赖,也可以不选择,稍后在pom文件里添加
2.4 项目启动类
2.5 前端页面展示
3. 添加项目配置
3.1 Pom文件添加依赖
<dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>commons-fileupload</groupId> <artifactId>commons-fileupload</artifactId> <version>1.3.1</version> </dependency> <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> <version>2.4</version> </dependency> <!-- 做断点下载使用 --> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpcore</artifactId> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.22</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> <version>3.12.0</version> </dependency> </dependencies>
3.2 修改application.yml配置
# 配置服务端口
server:
port: 8018
spring:
servlet:
multipart:
# Spring Boot中有默认的文件上传组件,在使用ServletFileUpload时需要关闭Spring Boot的默认配置
enabled: false
# 设置单个文件大小
max-file-size: 1GB
# 设置单次请求文件的总大小
max-request-size: 10GB
注意:yml文件要配置 max-file-size 和 max-request-size 配置文件大小,否则文件过大会报错
org.apache.tomcat.util.http.fileupload.impl.FileSizeLimitExceededException: The field file exceeds its maximum permitted size of 1048576 bytes.
at org.apache.tomcat.util.http.fileupload.impl.FileItemStreamImpl$1.raiseError(FileItemStreamImpl.java:117) ~[tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.tomcat.util.http.fileupload.util.LimitedInputStream.checkLimit(LimitedInputStream.java:76) ~[tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.tomcat.util.http.fileupload.util.LimitedInputStream.read(LimitedInputStream.java:135) ~[tomcat-embed-core-9.0.60.jar:9.0.60]
3.3 前端页面:file.html
webUpload组件支持分片上传:利用多进程并发上传,将大文件拆分成一个一个的小文件,每一个小文件属于大文件的一个分片
html文件分片上传需要引入4个文件(代码太长,需要的自取:SpringBoot-File)
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>大文件上传下载</title> <link rel="stylesheet" type="text/css" href="webuploader.css"> <script src="jquery.js"></script> <script src="webuploader.js"></script> <style> #upload-container { width: 100px; height: 50px; background: #94d3e7; padding-bottom: 10px; } </style> </head> <body> <div id="upload-container"><span>文件拖拽上传</span></div> <button id="picker" style="margin-top: 20px">分片上传</button> <div id="upload-list"></div> <hr/> <a href="/file/download" >普通下载</a> <hr/> <a href="/file/downloads" target="_blank">分片下载</a> </body> <script> $('#upload-container').click(function (event) { $("#picker").find('input').click(); }); // 初始化上传组件 const uploader = WebUploader.create({ auto: true, swf: 'Uploader.swf', // swf文件路径 server: '/file/upload', // 上传接口 dnd: '#upload-container', pick: '#picker', // 内部根据当前运行创建 multiple: true, // 选择多个 chunked: true, // 开启分片 threads: 8, // 并发数,默认 3 chunkRetry: 8, // 如果遇到网络错误,重新上传次数 method: 'POST', fileSizeLimit: 1024 * 1024 * 1024 * 10, // 文件总大小为10G fileSingleSizeLimit: 1024 * 1024 * 1024 * 1, // 单个文件大小最大为1G fileVal: 'upload' }); // 入队之前触发事件 uploader.on("beforeFileQueued", function (file) { // 获取文件后缀 console.log(file.name); }); // 当有文件被添加进队列的时候 uploader.on('fileQueued', function (file) { $('#upload-list').append( '<div id="' + file.id + '" class="item">' + '<h4 class="info">' + file.name + '</h4>' + '<p class="state">等待上传...</p>' + '</div>' ); }); // 文件上传过程中创建进度条实时显示。 uploader.on('uploadProgress', function (file, percentage) { var $li = $('#' + file.id), $percent = $li.find('.progress .progress-bar'); // 避免重复创建 if (!$percent.length) { $percent = $('<div class="progress progress-striped active">' + '<div class="progress-bar" role="progressbar" style="width: 0%">' + '</div>' + '</div>').appendTo($li).find('.progress-bar'); } $li.find('p.state').text('上传中'); $percent.css('width', percentage * 100 + '%'); }); uploader.on( 'uploadSuccess', function( file ) { $( '#'+file.id ).find('p.state').text('已上传'); }); uploader.on( 'uploadError', function( file ) { $( '#'+file.id ).find('p.state').text('上传出错'); }); uploader.on( 'uploadComplete', function( file ) { $( '#'+file.id ).find('.progress').fadeOut(); }); </script> </html>
3.4 编写文件测试接口FileController
package com.liyh.controller; import com.liyh.service.FileService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RestController; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.io.IOException; /** * 文件上传测试接口 * * @author liyh */ @RestController @RequestMapping("/file") public class FileController { @Autowired private FileService fileService; /** * 单个文件上传,支持断点续传 */ @PostMapping("/upload") public void upload(HttpServletRequest request, HttpServletResponse response) { try { fileService.upload(request, response); } catch (Exception e) { e.printStackTrace(); } } /** * 普通文件下载 */ @GetMapping("/download") public void download(HttpServletRequest request, HttpServletResponse response) throws IOException { fileService.download(request, response); } /** * 分片文件下载 */ @GetMapping("/downloads") public String downloads() throws IOException { fileService.downloads(); return "下载成功"; } }
3.5 添加需要的对象
FileInfo
UploadFileInfo
DownloadFileInfo
3.6 详细业务逻辑处理FileService
package com.liyh.service; import com.liyh.entity.DownloadFileInfo; import com.liyh.entity.FileInfo; import com.liyh.entity.UploadFileInfo; import org.apache.commons.fileupload.FileItem; import org.apache.commons.fileupload.disk.DiskFileItemFactory; import org.apache.commons.fileupload.servlet.ServletFileUpload; import org.apache.commons.io.FileUtils; import org.apache.commons.lang3.StringUtils; import org.apache.http.HttpEntity; import org.apache.http.HttpResponse; import org.apache.http.client.HttpClient; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.client.HttpClients; import org.springframework.stereotype.Service; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.io.*; import java.net.URLDecoder; import java.net.URLEncoder; import java.util.List; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; @Service public class FileService { /** * 编码 */ private static final String UTF_8 = "UTF-8"; /** * 文件上传路径(当前项目路径下,也可配置固定路径) */ private String uploadPath = System.getProperty("user.dir") + "/springboot-file/upload/"; /** * 下载指定文件 */ private String downloadFile = "D:\\Download\\git.exe"; /** * 文件下载地址(当前项目路径下,也可配置固定路径) */ private String downloadPath = System.getProperty("user.dir") + "/springboot-file/download/"; /** * 分片下载每一片大小为50M */ private static final Long PER_SLICE = 1024 * 1024 * 50L; /** * 定义分片下载线程池 */ private ExecutorService executorService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()); /** * final string */ private static final String RANGE = "Range"; /** * 上传文件 */ public void upload(HttpServletRequest request, HttpServletResponse response) throws Exception { // 获取ServletFileUpload ServletFileUpload servletFileUpload = getServletFileUpload(); List<FileItem> items = servletFileUpload.parseRequest(request); // 获取文件信息 UploadFileInfo uploadFileInfo = getFileInfo(items); // 写入临时文件 writeTempFile(items, uploadFileInfo); // 判断是否合并 mergeFile(uploadFileInfo); // 返回结果 response.setCharacterEncoding(UTF_8); response.getWriter().write("上传成功"); } /** * 获取ServletFileUpload */ private ServletFileUpload getServletFileUpload() { // 设置缓冲区大小,先读到内存里在从内存写 DiskFileItemFactory factory = new DiskFileItemFactory(); factory.setSizeThreshold(1024); File file = new File(uploadPath); // 如果文件夹不存在则创建 if (!file.exists() && !file.isDirectory()) { file.mkdirs(); } factory.setRepository(file); // 解析 ServletFileUpload upload = new ServletFileUpload(factory); // 设置单个大小与最大大小 upload.setFileSizeMax(1 * 1024 * 1024 * 1024L); upload.setSizeMax(10 * 1024 * 1024 * 1024L); return upload; } /** * 获取文件信息 * * @param items * @return * @throws UnsupportedEncodingException */ private UploadFileInfo getFileInfo(List<FileItem> items) throws UnsupportedEncodingException { UploadFileInfo uploadFileInfo = new UploadFileInfo(); for (FileItem item : items) { if (item.isFormField()) { // 获取分片数据 if ("chunk".equals(item.getFieldName())) { uploadFileInfo.setCurrentChunk(Integer.parseInt(item.getString(UTF_8))); } if ("chunks".equals(item.getFieldName())) { uploadFileInfo.setChunks(Integer.parseInt(item.getString(UTF_8))); } if ("name".equals(item.getFieldName())) { uploadFileInfo.setFileName(item.getString(UTF_8)); } } } return uploadFileInfo; } /** * 写入临时文件 * * @param items * @param uploadFileInfo * @throws Exception */ private void writeTempFile(List<FileItem> items, UploadFileInfo uploadFileInfo) throws Exception { // 获取文件基本信息后 for (FileItem item : items) { if (!item.isFormField()) { // 有分片需要临时目录 String tempFileName = uploadFileInfo.getFileName(); if (StringUtils.isNotBlank(tempFileName)) { if (uploadFileInfo.getCurrentChunk() != null) { tempFileName = uploadFileInfo.getCurrentChunk() + "_" + uploadFileInfo.getFileName(); } // 判断文件是否存在 File tempFile = new File(uploadPath, tempFileName); // 断点续传,判断文件是否存在,若存在则不传 if (!tempFile.exists()) { item.write(tempFile); } } } } } /** * 判断是否合并 * * @param uploadFileInfo * @throws IOException * @throws InterruptedException */ private void mergeFile(UploadFileInfo uploadFileInfo) throws IOException, InterruptedException { Integer currentChunk = uploadFileInfo.getCurrentChunk(); Integer chunks = uploadFileInfo.getChunks(); String fileName = uploadFileInfo.getFileName(); // 如果当前分片等于总分片那么合并文件 if (currentChunk != null && chunks != null && currentChunk.equals(chunks - 1)) { File tempFile = new File(uploadPath, fileName); try (BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(tempFile))) { // 根据之前命名规则找到所有分片 for (int i = 0; i < chunks; i++) { File file = new File(uploadPath, i + "_" + fileName); // 并发情况,需要判断所有,因为可能最后一个分片传完,之前有的还没传完 while (!file.exists()) { // 不存在休眠100毫秒后在重新判断 Thread.sleep(100); } // 分片存在,读入数组中 byte[] bytes = FileUtils.readFileToByteArray(file); os.write(bytes); os.flush(); file.delete(); } os.flush(); } } } /** * 文件下载 * * @param request * @param response * @throws IOException */ public void download(HttpServletRequest request, HttpServletResponse response) throws IOException { // 获取文件 File file = new File(downloadFile); // 获取下载文件信息 DownloadFileInfo downloadFileInfo = getDownloadFileInfo(file.length(), request, response); // 设置响应头 setResponse(response, file.getName(), downloadFileInfo); // 下载文件 try (InputStream is = new BufferedInputStream(new FileInputStream(file)); OutputStream os = new BufferedOutputStream(response.getOutputStream())) { // 跳过已经读取文件 is.skip(downloadFileInfo.getPos()); byte[] buffer = new byte[1024]; long sum = 0; // 读取 while (sum < downloadFileInfo.getRangeLength()) { int length = is.read(buffer, 0, (downloadFileInfo.getRangeLength() - sum) <= buffer.length ? (int) (downloadFileInfo.getRangeLength() - sum) : buffer.length); sum = sum + length; os.write(buffer, 0, length); } } } /** * 有两个map,我要去判断里面相同键的值一致不一致,除了双重for循环,有没有别的好办法 */ private DownloadFileInfo getDownloadFileInfo(long fSize, HttpServletRequest request, HttpServletResponse response) { long pos = 0; long last = fSize - 1; // 判断前端是否需要分片下载 if (request.getHeader(RANGE) != null) { response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT); String numRange = request.getHeader(RANGE).replace("bytes=", ""); String[] strRange = numRange.split("-"); if (strRange.length == 2) { pos = Long.parseLong(strRange[0].trim()); last = Long.parseLong(strRange[1].trim()); // 若结束字节超出文件大小,取文件大小 if (last > fSize - 1) { last = fSize - 1; } } else { // 若只给一个长度,开始位置一直到结束 pos = Long.parseLong(numRange.replace("-", "").trim()); } } long rangeLength = last - pos + 1; String contentRange = "bytes " + pos + "-" + last + "/" + fSize; return new DownloadFileInfo(fSize, pos, last, rangeLength, contentRange); } /** * 分片下载 * * @throws IOException */ public void downloads() throws IOException { File file = new File(downloadPath); // 如果文件夹不存在则创建 if (!file.exists() && !file.isDirectory()) { file.mkdirs(); } // 探测下载,获取文件相关信息 FileInfo fileInfoDto = sliceDownload(1, 10, -1, null); // 如果不为空,执行分片下载 if (fileInfoDto != null) { // 计算有多少分片 long pages = fileInfoDto.getFileSize() / PER_SLICE; // 适配最后一个分片 for (long i = 0; i <= pages; i++) { long start = i * PER_SLICE; long end = (i + 1) * PER_SLICE - 1; executorService.execute(new SliceDownloadRunnable(start, end, i, fileInfoDto.getFileName())); } } } /** * 分片下载 * * @param start 分片起始位置 * @param end 分片结束位置 * @param page 第几个分片, page=-1时是探测下载 */ private FileInfo sliceDownload(long start, long end, long page, String fName) throws IOException { // 断点下载 File file = new File(downloadPath, page + "-" + fName); // 如果当前文件已经存在,并且不是探测任务,并且文件的长度等于分片的大小,那么不用下载当前文件 if (file.exists() && page != -1 && file.length() == PER_SLICE) { return null; } // 创建HttpClient HttpClient client = HttpClients.createDefault(); HttpGet httpGet = new HttpGet("http://localhost:8018/file/download"); httpGet.setHeader(RANGE, "bytes=" + start + "-" + end); HttpResponse httpResponse = client.execute(httpGet); String fSize = httpResponse.getFirstHeader("fSize").getValue(); fName = URLDecoder.decode(httpResponse.getFirstHeader("fName").getValue(), UTF_8); HttpEntity entity = httpResponse.getEntity(); // 下载 try (InputStream is = entity.getContent(); FileOutputStream fos = new FileOutputStream(file)) { byte[] buffer = new byte[1024]; int ch; while ((ch = is.read(buffer)) != -1) { fos.write(buffer, 0, ch); } fos.flush(); } // 判断是否是最后一个分片,如果是那么合并 if (end - Long.parseLong(fSize) > 0) { mergeFile(fName, page); } return new FileInfo(Long.parseLong(fSize), fName); } private void mergeFile(String fName, long page) throws IOException { File file = new File(downloadPath, fName); try (BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(file))) { for (int i = 0; i <= page; i++) { File tempFile = new File(downloadPath, i + "-" + fName); // 文件不存在或文件没写完 while (!tempFile.exists() || (i != page && tempFile.length() < PER_SLICE)) { Thread.sleep(100); } byte[] bytes = FileUtils.readFileToByteArray(tempFile); os.write(bytes); os.flush(); tempFile.delete(); } // 删除文件 File f = new File(downloadPath, "-1" + "-null"); if (f.exists()) { f.delete(); } } catch (InterruptedException e) { e.printStackTrace(); } } private class SliceDownloadRunnable implements Runnable { private final long start; private final long end; private final long page; private final String fName; private SliceDownloadRunnable(long start, long end, long page, String fName) { this.start = start; this.end = end; this.page = page; this.fName = fName; } @Override public void run() { try { sliceDownload(start, end, page, fName); } catch (IOException e) { e.printStackTrace(); } } } /** * 设置响应头 */ private void setResponse(HttpServletResponse response, String fileName, DownloadFileInfo downloadFileInfo) throws UnsupportedEncodingException { response.setCharacterEncoding(UTF_8); response.setContentType("application/x-download"); response.addHeader("Content-Disposition", "attachment;filename=" + URLEncoder.encode(fileName, UTF_8)); // 支持分片下载 response.setHeader("Accept-Range", "bytes"); response.setHeader("fSize", String.valueOf(downloadFileInfo.getFSize())); response.setHeader("fName", URLEncoder.encode(fileName, UTF_8)); // range响应头 response.setHeader("Content-Range", downloadFileInfo.getContentRange()); response.setHeader("Content-Length", String.valueOf(downloadFileInfo.getRangeLength())); } }
http://localhost:8018/file.html
4.2 测试文件分片上传(拖拽或点击上传都可以),上传成功到配置的上传目录查看文件
4.3 测试文件普通下载
4.4 测试文件分片下载(分片下载会先生成多个文件,下载完成会合并)
5. 完整项目地址(SpringBoot-File)
5.1 项目结构
参考文章:http://blog.ncmem.com/wordpress/2023/11/04/springboot%e5%ae%9e%e7%8e%b0%e5%a4%a7%e6%96%87%e4%bb%b6%e4%b8%8a%e4%bc%a0-%e4%b8%8b%e8%bd%bd%e5%88%86%e7%89%87%e3%80%81%e6%96%ad%e7%82%b9%e7%bb%ad%e4%bc%a0/
欢迎入群一起讨论
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。