devbox
2023面试高手
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

【分布式文件存储】MinIO部署及实现文件上传下载_minio文件下载

作者：2023面试高手 | 2024-05-17 11:23:52

赞

踩

minio文件下载

目录

概述

MinIO集群部署

准备docker-compose.yml

测试启动

MinIO用户管理

Buckets管理

 创建Buckets

MinIO客户端

引入依赖 

 文件上传下载Demo

调用API碰到的问题

概述

MinIO | 高性能, Kubernetes 原生对象存储

MinIO是全球领先的对象存储先锋，目前在全世界有数百万的用户。

  • 高性能 ，在标准硬件上，读/写速度上高达183GB/秒和171GB/秒，拥有更高的吞吐量和更低的延迟

  • 可扩展性 ，为对象存储带来了简单的缩放模型，通过添加更多集群可以扩展空间

  • 简单 ，极简主义是MinIO的指导性设计原则，即可在几分钟内安装和配置

  • 与Amazon S3兼容 ，亚马逊云的 S3 API（接口协议）是在全球范围内达到共识的对象存储的协议，是全世界内大家都认可的标准

  • 数据安全 ，使用纠删码来保护数据免受硬件故障和无声数据损坏

MinIO集群部署

基于docker-compose部署入门案例

准备docker-compose.yml

  1. version: '3.7'
  2.  
  3. # 所有容器通用的设置和配置
  4. x-minio-common: &minio-common
  5.   image: minio/minio
  6.   command: server --console-address ":9001" http://minio{1...4}/data
  7.   expose:
  8.     - "9000"
  9.   # environment:
  10.     # MINIO_ROOT_USER: minioadmin
  11.     # MINIO_ROOT_PASSWORD: minioadmin
  12.   healthcheck:
  13.     test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
  14.     interval: 30s
  15.     timeout: 20s
  16.     retries: 3
  17.  
  18. # 启动4个docker容器运行minio服务器实例
  19. # 使用nginx反向代理9000端口，负载均衡, 你可以通过9001、9002、9003、9004端口访问它们的web console
  20. services:
  21.   minio1:
  22.     <<: *minio-common
  23.     hostname: minio1
  24.     ports:
  25.       - "9001:9001"
  26.     volumes:
  27.       - ./data/data1:/data
  28.  
  29.   minio2:
  30.     <<: *minio-common
  31.     hostname: minio2
  32.     ports:
  33.       - "9002:9001"
  34.     volumes:
  35.       - ./data/data2:/data
  36.  
  37.   minio3:
  38.     <<: *minio-common
  39.     hostname: minio3
  40.     ports:
  41.       - "9003:9001"
  42.     volumes:
  43.       - ./data/data3:/data
  44.  
  45.   minio4:
  46.     <<: *minio-common
  47.     hostname: minio4
  48.     ports:
  49.       - "9004:9001"
  50.     volumes:
  51.       - ./data/data4:/data
  52.  
  53.   nginx:
  54.     image: nginx:1.19.2-alpine
  55.     hostname: nginx
  56.     volumes:
  57.       - ./config/nginx.conf:/etc/nginx/nginx.conf:ro
  58.     ports:
  59.       - "9000:9000"
  60.     depends_on:
  61.       - minio1
  62.       - minio2
  63.       - minio3
  64.       - minio4

 在当前目录下，创建config，创建nginx.conf

  1. user  nginx;
  2. worker_processes  auto;
  3.  
  4. error_log  /var/log/nginx/error.log warn;
  5. pid        /var/run/nginx.pid;
  6.  
  7. events {
  8.     worker_connections  4096;
  9. }
  10.  
  11. http {
  12.     include       /etc/nginx/mime.types;
  13.     default_type  application/octet-stream;
  14.  
  15.     log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
  16.                       '$status $body_bytes_sent "$http_referer" '
  17.                       '"$http_user_agent" "$http_x_forwarded_for"';
  18.  
  19.     access_log  /var/log/nginx/access.log  main;
  20.     sendfile        on;
  21.     keepalive_timeout  65;
  22.  
  23.     # include /etc/nginx/conf.d/*.conf;
  24.  
  25.     upstream minio {
  26.         server minio1:9000;
  27.         server minio2:9000;
  28.         server minio3:9000;
  29.         server minio4:9000;
  30.     }
  31.  
  32.     server {
  33.         listen       9000;
  34.         listen  [::]:9000;
  35.         server_name  localhost;
  36.  
  37.         # To allow special characters in headers
  38.         ignore_invalid_headers off;
  39.         # Allow any size file to be uploaded.
  40.         # Set to a value such as 1000m; to restrict file size to a specific value
  41.         client_max_body_size 0;
  42.         # To disable buffering
  43.         proxy_buffering off;
  44.  
  45.         location / {
  46.             proxy_set_header Host $http_host;
  47.             proxy_set_header X-Real-IP $remote_addr;
  48.             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  49.             proxy_set_header X-Forwarded-Proto $scheme;
  50.  
  51.             proxy_connect_timeout 300;
  52.             # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
  53.             proxy_http_version 1.1;
  54.             proxy_set_header Connection "";
  55.             chunked_transfer_encoding off;
  56.  
  57.             proxy_pass http://minio;
  58.         }
  59.     }
  60.  
  61. }

测试启动

docker-compose up -d 

访问任意节点

默认账户密码：minioadmin minioadmin

MinIO用户管理

需要创建一个用户，客户端用户可上传下载权限。

注意：长度最好是32位，否则后续调用API会报错

Buckets管理

MinIO uses buckets to organize objects. A bucket is similar to a folder or directory in a filesystem, where each bucket can hold an arbitrary number of objects.

MinIO使用bucket来组织对象。bucket类似于文件系统中的文件夹或目录，其中每个bucket可以容纳任意数量的对象。

 创建Buckets

注意桶名的命名规范，否则后续调用api创建桶会报错

 可以在控制台管理文件，上传下载预览删除等

Simple Storage Service（Amazon S3）存储桶命名要求 - AWS CloudTrail

MinIO客户端

官方接口API文档

Java Client API Reference — MinIO Object Storage for Linux

MinIO支持多种客户端，本文以java为例

引入依赖 

  1. <dependency>
  2.     <groupId>io.minio</groupId>
  3.     <artifactId>minio</artifactId>
  4.     <version>8.5.3</version>
  5. </dependency>

 文件上传下载Demo

  1. package pers.kw.minio.demo;
  2.  
  3. import com.alibaba.fastjson.JSON;
  4. import io.minio.*;
  5. import io.minio.errors.*;
  6. import org.apache.commons.io.FileUtils;
  7. import org.apache.commons.io.IOUtils;
  8.  
  9. import java.io.File;
  10. import java.io.IOException;
  11. import java.io.InputStream;
  12. import java.security.InvalidKeyException;
  13. import java.security.NoSuchAlgorithmException;
  14. import java.util.List;
  15.  
  16. public class MinIODemo {
  17.  
  18.     private static final String accessKey = "PPeCeQbHtFsEcGNxnlmSpbYasxELHoGy";
  19.     private static final String secretKey = "GEMQDMUhQSHaIJCqmxperpESNYCJXvjJ";
  20.     private static final String endpoint = "http://192.168.8.88:9000";
  21.  
  22.  
  23.     public static void main(String[] args)
  24.             throws ServerException, InsufficientDataException, ErrorResponseException,
  25.             IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
  26.         String bucket = "test01";
  27.         String objectName = "small125750JkEMk1625374670.jpg";
  28.         downloadFile(bucket, objectName);
  29.  
  30.     }
  31.  
  32.     public static void downloadFile(String bucket, String objectName)
  33.             throws ServerException, InsufficientDataException, ErrorResponseException, IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidResponseException, XmlParserException, InternalException {
  34.         MinioClient minioClient =
  35.                 MinioClient.builder()
  36.                         .endpoint(endpoint)
  37.                         .credentials(accessKey, secretKey)
  38.                         .build();
  39.         //todo 需要关闭流
  40.         try (InputStream inputStream = minioClient.getObject(
  41.                 GetObjectArgs.builder()
  42.                         .bucket(bucket)
  43.                         .object(objectName)
  44.                         .build())) {
  45.             // Read data from stream
  46.             byte[] bytes = IOUtils.toByteArray(inputStream);
  47.             String fileName = "22.jpg";
  48.             System.out.println("下载文件：" + bytes.length);
  49.             FileUtils.writeByteArrayToFile(new File("C:\\Users\\pc\\Desktop\\" + fileName), bytes);
  50.             System.out.println("文件下载成功");
  51.             IOUtils.closeQuietly(inputStream);
  52.         }
  53.     }
  54.  
  55.     public static void uploadFile(String bucket) throws IOException, NoSuchAlgorithmException, InvalidKeyException {
  56.         try {
  57.             // Create a minioClient with the MinIO server playground, its access key and secret key.
  58.             MinioClient minioClient =
  59.                     MinioClient.builder()
  60.                             .endpoint(endpoint)
  61.                             .credentials(accessKey, secretKey)
  62.                             .build();
  63.  
  64.             // Make 'asiatrip' bucket if not exist.
  65.             boolean found =
  66.                     minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
  67.             if (!found) {
  68.                 // Make a new bucket called bucket.
  69.                 minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());
  70.             } else {
  71.                 System.out.println("Bucket " + bucket + " already exists.");
  72.             }
  73.  
  74.             // Upload '/home/user/Photos/asiaphotos.zip' as object name 'asiaphotos-2015.zip' to bucket
  75.             // 'asiatrip'.
  76.             String fileName = "C:\\Users\\pc\\Desktop\\gateway-server.rar";
  77.             String objectName = "gateway-server_2023058.rar";
  78.             minioClient.uploadObject(
  79.                     UploadObjectArgs.builder()
  80.                             .bucket(bucket)
  81.                             .object(objectName)
  82.                             .filename(fileName)
  83.                             .build());
  84.             System.out.println(
  85.                     fileName + " is successfully uploaded as"
  86.                             + objectName + " to bucket " + bucket + ".");
  87.         } catch (MinioException e) {
  88.             System.out.println("Error occurred: " + e);
  89.             System.out.println("HTTP trace: " + e.httpTrace());
  90.         }
  91.     }
  92.  
  93. }

调用API碰到的问题

问题一：

java.lang.NoSuchMethodError: okhttp3.RequestBody.create ( BLokhttp3/Medialype:)Lokhttp3/RequestBody;

 解决: 引入如下依赖

  1. <dependency>
  2.     <groupId>com.squareup.okhttp3</groupId>
  3.     <artifactId>okhttp</artifactId>
  4.     <version>4.9.0</version>
  5. </dependency>

问题二：

bucket name does not follow Amazon S3 standards.

命名不符合规则换个名字，最好在代码中增加正则校验

问题三：

RequestTimeTooSkewed

客户端时间与服务器时间相差过大导致

 设置服务器时间

 ntpdate time.windows.com

问题4：

InvalidAccessKeyId

秘钥信息太短

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/2023面试高手/article/detail/583406
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号