HDFS的Java API操作_hdfs的java apl 操作

1、环境准备

1）下载依赖文件

根据安装Hadoop版本，下载对应的github(winutils) 、csdn(winutils) (点击蓝色字体跳转)

winutils： 适用于Hadoop版本的Windows二进制文件，这些是直接从用于创建官方ASF版本的相同git提交构建的; 它们被检出并构建在Windows VM上，该VM专用于在Windows上测试Hadoop / YARN应用程序。它不是日常使用的系统，因此与驱动器/电子邮件安全攻击隔离。

[root@node01 file]# hadoop version
Hadoop 3.1.3
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r ba631c436b806728f8ec2f54ab1e289526c90579
Compiled by ztang on 2019-09-12T02:47Z
Compiled with protoc 2.5.0
From source with checksum ec785077c385118ac91aadde5ec9799
This command was run using /export/server/hadoop-3.1.3/share/hadoop/common/hadoop-common-3.1.3.jar
1
2
3
4
5
6
7

2）配置 HADOOP_HOME 环境变量

3）配置 Path 环境变量

注意：如果环境变量不起作用，可以重启电脑试试。

2、HDFS API案例

<dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>3.1.3</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.30</version>
        </dependency>
    </dependencies>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

在项目的 src/main/resources 目录下，新建一个文件，命名为“log4j.properties”，在文件中填入

log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n
log4j.appender.logfile=org.apache.log4j.FileAppender
log4j.appender.logfile.File=target/spring.log
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n
1
2
3
4
5
6
7
8

将 hdfs-site.xml 拷贝到项目的 resources 资源目录下

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
	<!--指定hdfs保存数据的副本数量-->
	<property>
		<name>dfs.replication</name>
		<value>2</value>
	</property>
</configuration>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

参数优先级
参数优先级排序：（1）客户端代码中设置的值 >（2）ClassPath 下的用户自定义配置文
件 >（3）然后是服务器的自定义配置（xxx-site.xml）>（4）服务器的默认配置（xxx-default.xml）

1） HDFS 文件上传

@Test
public void copyFromLocalFile() throws Exception {
    // 获取文件系统
    Configuration configuration = new Configuration();
    configuration.set("dfs.replication", "2");
    FileSystem fs = FileSystem.get(new URI("hdfs://node01:8020"), configuration, "root");
    // 上传文件
    fs.copyFromLocalFile(new Path("src/file/test01.txt"), new Path("/test"));
    // 关闭资源
    fs.close();
}
1
2
3
4
5
6
7
8
9
10
11

2）HDFS 文件下载

@Test
public void copyToLocalFile() throws Exception {
    // 获取文件系统
    Configuration configuration = new Configuration();
    FileSystem fs = FileSystem.get(new URI("hdfs://node01:8020"), configuration, "root");
    // 下载 boolean
    // delSrc:指是否将原文件删除
    // Path src:指要下载的文件路径
    // Path dst:指将文件下载到的路径
    // boolean useRawLocalFileSystem:是否开启文件校验
    fs.copyToLocalFile(false, new Path("/test/wordCount.txt"), new Path("src/file/"), true);
    fs.close();
}
1
2
3
4
5
6
7
8
9
10
11
12
13

3）HDFS 文件更名和移动

@Test
public void rename() throws Exception {
    // 获取文件系统
    Configuration configuration = new Configuration();
    FileSystem fs = FileSystem.get(new URI("hdfs://node01:8020"), configuration, "root");
    // 修改文件名称
    fs.rename(new Path("/test/test01.txt"), new Path("/test/test02.txt"));
    fs.close();
}
1
2
3
4
5
6
7
8
9

4）HDFS 删除文件和目录

@Test
public void delete() throws Exception {
    // 获取文件系统
    Configuration configuration = new Configuration();
    FileSystem fs = FileSystem.get(new URI("hdfs://node01:8020"), configuration, "root");
    // 删除
    fs.delete(new Path("/test/test02.txt"), true);
    fs.close();
}
1
2
3
4
5
6
7
8
9

5）HDFS 文件详情查看

@Test
public void listFiles() throws Exception {
    // 获取文件系统
    Configuration configuration = new Configuration();
    FileSystem fs = FileSystem.get(new URI("hdfs://node01:8020"), configuration, "root");
    // 获取文件详情
    RemoteIterator<LocatedFileStatus> listFiles = fs.listFiles(new Path("/"), true);
    while (listFiles.hasNext()) {
        LocatedFileStatus fileStatus = listFiles.next();
        System.out.println("========" + fileStatus.getPath() + "=========");
        System.out.println(fileStatus.getPermission());
        System.out.println(fileStatus.getOwner());
        System.out.println(fileStatus.getGroup());
        System.out.println(fileStatus.getLen());
        System.out.println(fileStatus.getModificationTime());
        System.out.println(fileStatus.getReplication());
        System.out.println(fileStatus.getBlockSize());
        System.out.println(fileStatus.getPath().getName());
        BlockLocation[] blockLocations = fileStatus.getBlockLocations();
        System.out.println(Arrays.toString(blockLocations));
    }
    fs.close();
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

6）HDFS 文件和文件夹判断

@Test
public void listStatus() throws Exception{
    // 获取文件系统
    Configuration configuration = new Configuration();
    FileSystem fs = FileSystem.get(new URI("hdfs://node01:8020"), configuration, "root");
    FileStatus[] fileStatuses = fs.listStatus(new Path("/"));
    for (FileStatus fileStatus : fileStatuses) {
        if (fileStatus.isFile()){
            System.out.println("f:"+fileStatus.getPath().getName());
        }else {
            System.out.println("d:"+fileStatus.getPath().getName());
        }
    }
    fs.close();
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15