大数据学习之HDFS基石——Hadoop应用单实例部署_hdfs单机部署

1 背景

由于我本人服务器有限，因此尝试进行单机安装，主要为了学习使用。

2 前提

hadoop安装依赖JDK。
我这里尝试用下JDK18看看。
下载地址：https://jdk.java.net/18/

这里咱们选择Linux/x64的版本。

2.1 配置JDK环境变量

# vi /etc/profile
在文件最后补充：
export JAVA_HOME=/usr/local/java/jdk-18.0.2.1
export PATH=$PATH:$JAVA_HOME/bin
配置生效
# source /etc/profile
1
2
3
4
5
6

4 下载

http://archive.apache.org/dist/hadoop/core/hadoop-3.3.4/
在这个链接，你可以看到各个版本，我这里选择的是3.3.4
在服务器上下载命令：

# wget http://archive.apache.org/dist/hadoop/core/hadoop-3.3.4/hadoop-3.3.4.tar.gz
1

5 解压

我这边上传到centos上，进行解压

# tar -zxvf hadoop-3.3.4.tar.gz 
1

6 配置环境变量

同样，在/etc/profile文件中配置hadoop配置文件：

# vi /etc/profile
 在文件末尾补充：
export HADOOP_HOME=/usr/local/hadoop/hadoop-3.3.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

 配置生效
# source /etc/profile
1
2
3
4
5
6
7

检查配置是否生效

[root@instance-1mfp2mc1 jdk-18.0.2.1]# hadoop version
Hadoop 3.3.4
Source code repository https://github.com/apache/hadoop.git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb
Compiled by stevel on 2022-07-29T12:32Z
Compiled with protoc 3.7.1
From source with checksum fb9dd8918a7b8a5b430d61af858f6ec
This command was run using /usr/local/hadoop/hadoop-3.3.4/share/hadoop/common/hadoop-common-3.3.4.jar
1
2
3
4
5
6
7

7 配置Hadoop文件

所有配置文件都在etc目录下：hadoop-3.3.4/etc/hadoop，所以，我们只要关心这个目录下相关文件的配置。
各个配置文件说明如下：

7.1 修改hadoop-env.sh

在文件末尾加上如下配置：

export JAVA_HOME=/usr/local/java/jdk-18.0.2.1
export HDFS_NAMENODE_USER=hdfs
1
2

7.2 修改 core-site.xml

将配置文件修改为：

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9820</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/hadoopdata</value>
    </property>
</configuration>
1
2
3
4
5
6
7
8
9
10

7.3 修改hdfs-site.xml

将配置文件修改为如下，表示只有一个副本，因为我这里是单机：

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>
1
2
3
4
5
6

7.4 修改mapred-site.xml

将配置文件修改为：

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>
1
2
3
4
5
6

7.5 修改yarn-site.xml

将配置文件修改为：

<configuration>

<!-- Site specific YARN configuration properties -->


    <property>
　　    <name>yarn.nodemanager.aux-services</name>
　　    <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>
1
2
3
4
5
6
7
8
9
10
11
12
13
14

8 启动

8.1 对安装目录进行授权

# chmod -R 733 hadoop-3.3.4
1

8.1 创建hdfs用户

[root@instance-0pep91c1 bin]# useradd hdfs
[root@instance-0pep91c1 bin]# passwd hdfs
Changing password for user hdfs.
New password: 
BAD PASSWORD: The password is shorter than 8 characters
Retype new password: 
passwd: all authentication tokens updated successfully.
[root@instance-0pep91c1 bin]# 
[root@instance-0pep91c1 local]# chown -R hdfs:hdfs hadoop/
1
2
3
4
5
6
7
8
9

8.2 格式化hdfs文件系统

# cd /usr/local/hadoop/hadoop-3.3.4/
# bin/hdfs namenode -format
1
2

8.3 启动hadoop

[hdfs@instance-0pep91c1 sbin]$ start-dfs.sh 
Starting namenodes on [localhost]
localhost: hdfs@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting datanodes
localhost: hdfs@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting secondary namenodes [instance-0pep91c1]
instance-0pep91c1: Warning: Permanently added 'instance-0pep91c1' (ED25519) to the list of known hosts.
instance-0pep91c1: hdfs@instance-0pep91c1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[hdfs@instance-0pep91c1 sbin]$ 
1
2
3
4
5
6
7
8
9

不过，我这里失败了。

9 配置ssh并再次启动

操作步骤我就直接贴出来了：

[hdfs@instance-0pep91c1 .ssh]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /home/hdfs/.ssh/id_rsa
Your public key has been saved in /home/hdfs/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:fx5d+EsH+y68Lcbgw5Jyq8YZ75tolvXl4NtrYWdTebc hdfs@instance-0pep91c1
The key's randomart image is:
+---[RSA 3072]----+
|                 |
|                 |
|                .|
|               o+|
|        S     o *|
|        ... o.+Eo|
|       . *.*oOo=+|
|        O.=+*oXoo|
|       +o==+o=+B+|
+----[SHA256]-----+
[hdfs@instance-0pep91c1 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hdfs@instance-0pep91c1 .ssh]$ ssh localhost
Last login: Sun Nov 12 15:37:32 2023
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

再次启动下hadoop:

[hdfs@instance-0pep91c1 sbin]$ start-dfs.sh 
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [instance-0pep91c1]
[hdfs@instance-0pep91c1 sbin]$ 
1
2
3
4
5

打开namenode，http://120.48.16.193:9870/，namenode的作用是存储元数据。

打开datanode，http://120.48.16.193:9864/，datanode作用是存储真正的数据。