赞
踩
Hadoop是一个开源的分布式计算平台,用于处理大规模数据集的存储和处理。它由Apache软件基金会开发,旨在提供可靠、高效和可扩展的分布式计算环境。以下是对Hadoop的完整介绍:
分布式存储:
Hadoop的核心组件之一是Hadoop分布式文件系统(HDFS)。HDFS设计用于存储大规模数据集,并在多个计算节点之间进行分布式存储。它通过将数据划分为多个块,并在集群中的多个节点上复制这些块来确保数据的高可靠性和容错性。
分布式计算:
Hadoop的另一个核心组件是MapReduce。MapReduce是一种编程模型和处理框架,用于在Hadoop集群上进行分布式数据处理。MapReduce将任务分解为一系列的Map和Reduce阶段,并将数据处理任务分发到集群中的多个计算节点上,从而实现并行处理和计算。
高可靠性和容错性:
Hadoop具有高度可靠性和容错性的特点。通过在集群中的多个节点上复制数据块,HDFS可以保证数据的持久性和可靠性。此外,Hadoop集群中的节点是相互独立的,如果某个节点发生故障,任务可以在其他节点上继续执行,从而提高了系统的容错性。
可扩展性:
Hadoop具有良好的可扩展性,可以轻松地扩展以处理不断增长的数据量。通过添加更多的计算节点和存储节点,Hadoop集群可以在需要时动态地扩展,以满足不断增长的数据处理需求。
生态系统:
除了核心的HDFS和MapReduce之外,Hadoop生态系统还包括许多其他的组件和工具,用于数据存储、数据处理、数据查询和分析等各种场景。例如,Hadoop生态系统中的工具包括Apache Hive、Apache Pig、Apache Spark、Apache HBase等,这些工具提供了丰富的功能和API,以满足不同的数据处理需求。
要在Ubuntu上配置Hadoop,你可以按照以下步骤进行:
下载Hadoop:
首先,从Hadoop官方网站下载Hadoop安装包。你提供的链接是Hadoop 3.1.3版本的安装包。
地址:https://archive.apache.org/dist/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz
解压安装包:
在终端中导航到你下载Hadoop安装包的目录,并解压缩它。你可以使用以下命令:
sudo apt-get install ssh
tar -zxvf hadoop-3.1.3.tar.gz
配置环境变量:
打开终端,并编辑你的~/.bashrc文件:
vim ~/.bashrc
在文件的末尾添加以下行,指定Hadoop的安装路径:
export HADOOP_HOME=/path/to/your/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HOME/bin
假设下面是我解压后的hadoop的路径:/data/myapp2024/myBigDataApplicationDevelopment
那么就有:
export HADOOP_HOME=/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HOME/bin
保存并关闭文件。然后,运行以下命令使更改生效:
source ~/.bashrc
配置Hadoop:
进入Hadoop的配置目录:
cd /path/to/your/hadoop-3.1.3/etc/hadoop
修改hadoop-env.sh文件,设置Java路径。找到以下行并修改为你的Java安装路径:
export JAVA_HOME=/path/to/your/java
修改core-site.xml文件,添加以下内容来配置Hadoop的核心设置:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
可以使用vs code打开xml文件:
修改hdfs-site.xml文件,添加以下内容来配置Hadoop的HDFS设置:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
格式化Hadoop文件系统:
运行以下命令格式化Hadoop文件系统:
hdfs namenode -format
启动Hadoop:
运行以下命令启动Hadoop集群:
start-all.sh
现在,你已经在Ubuntu上成功配置和启动了Hadoop。你可以通过访问http://localhost:9870/来查看Hadoop的Web界面。
如果以上操作还是不行,如下:
sudo ./start-all.sh
ERROR: JAVA_HOME is not set and could not be found.
错误表明您的系统中没有设置JAVA_HOME环境变量,或者该变量未正确设置。JAVA_HOME环境变量是指向Java安装目录的路径,它告诉系统在哪里找到Java的安装目录。
最终解决方法:
sudo tar -zxvf ./jdk-8u212-linux-x64.tar.gz -C /usr/lib/jvm
apt install net-tool
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_212
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
source ~/.bashrc
此外,可能会出现,在集群环境下,即使各结点都正确地配置了JAVA_HOME,也会报如下错误。解决方法 是在路径hadoop-3.1.3/etc/hadoop/hadoop-env.sh添加如下配置,也就是在hadoop-env.sh中,再显示地重新声明一遍JAVA_HOME:
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_212
如果再次启动出现下面的问题:
User
./start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [98031e181845]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
您遇到的问题是因为缺少了必要的 Hadoop 用户环境变量。在启动 Hadoop 时,系统尝试以 root 用户身份运行各个组件,但由于缺少相应的用户环境变量,因此出现了错误。
要解决这个问题,您需要设置以下 Hadoop 用户环境变量:
HDFS_NAMENODE_USER:用于指定 NameNode 运行时所使用的用户。
HDFS_DATANODE_USER:用于指定 DataNode 运行时所使用的用户。
HDFS_SECONDARYNAMENODE_USER:用于指定 Secondary NameNode 运行时所使用的用户。
YARN_RESOURCEMANAGER_USER:用于指定 ResourceManager 运行时所使用的用户。
YARN_NODEMANAGER_USER:用于指定 NodeManager 运行时所使用的用户。
以上解决方法如下:
sudo apt update
sudo apt install ufw
export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"
再次操作:
sbin# hdfs namenode -format
namenode is running as process 71854. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh -P 2699
Starting namenodes on [localhost]
localhost: namenode is running as process 71854. Stop it first.
Starting datanodes
localhost: datanode is running as process 72115. Stop it first.
Starting secondary namenodes [98031e181845]
98031e181845: secondarynamenode is running as process 72519. Stop it first.
Starting resourcemanager
resourcemanager is running as process 72893. Stop it first.
Starting nodemanagers
localhost: nodemanager is running as process 73233. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin#
以上操作说明:启动Hadoop时,系统会检查是否有相关的进程正在运行。如果系统发现某些Hadoop组件已经在运行,它会拒绝启动新的实例,因为这可能会导致冲突或数据丢失。
在您的情况下,当您尝试使用 start-all.sh 启动Hadoop时,系统发现其中一些组件已经在运行中,例如namenode、datanode、secondary namenode、resourcemanager和nodemanagers。由于这些组件已经在运行中,因此Hadoop无法再启动相同的组件实例。
为了解决这个问题,您需要先停止正在运行的Hadoop组件,然后再尝试重新启动它们。您可以使用 stop-all.sh 命令停止所有Hadoop组件的运行。一旦停止了所有组件,您就可以再次尝试使用 start-all.sh 命令来启动Hadoop。这样就可以确保在启动新的Hadoop实例之前,先停止现有的运行实例,避免了可能的冲突或数据损坏。
集群打开结果:会自动跳出打开浏览器的提示
http://localhost:9870/explorer.html#/
操作命令:
re/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-api-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-common-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-client-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-router-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-common-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-registry-3.1.3.jar:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.3.jar
STARTUP_MSG: build = https://gitbox.apache.org/repos/asf/hadoop.git -r ba631c436b806728f8ec2f54ab1e289526c90579; compiled by 'ztang' on 2019-09-12T02:47Z
STARTUP_MSG: java = 11.0.19
************************************************************/
2024-04-09 07:25:35,413 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2024-04-09 07:25:35,469 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-13c722d2-a66a-498b-a887-5e7f11b96838
2024-04-09 07:25:35,757 INFO namenode.FSEditLog: Edit logging is async:true
2024-04-09 07:25:35,765 INFO namenode.FSNamesystem: KeyProvider: null
2024-04-09 07:25:35,766 INFO namenode.FSNamesystem: fsLock is fair: true
2024-04-09 07:25:35,766 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2024-04-09 07:25:35,776 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)
2024-04-09 07:25:35,776 INFO namenode.FSNamesystem: supergroup = supergroup
2024-04-09 07:25:35,776 INFO namenode.FSNamesystem: isPermissionEnabled = true
2024-04-09 07:25:35,776 INFO namenode.FSNamesystem: HA Enabled: false
2024-04-09 07:25:35,798 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2024-04-09 07:25:35,804 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2024-04-09 07:25:35,804 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2024-04-09 07:25:35,807 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2024-04-09 07:25:35,807 INFO blockmanagement.BlockManager: The block deletion will start around 2024 Apr 09 07:25:35
2024-04-09 07:25:35,808 INFO util.GSet: Computing capacity for map BlocksMap
2024-04-09 07:25:35,808 INFO util.GSet: VM type = 64-bit
2024-04-09 07:25:35,809 INFO util.GSet: 2.0% max memory 3.9 GB = 79.5 MB
2024-04-09 07:25:35,809 INFO util.GSet: capacity = 2^23 = 8388608 entries
2024-04-09 07:25:35,822 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2024-04-09 07:25:35,825 INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
2024-04-09 07:25:35,825 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2024-04-09 07:25:35,825 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2024-04-09 07:25:35,825 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2024-04-09 07:25:35,825 INFO blockmanagement.BlockManager: defaultReplication = 1
2024-04-09 07:25:35,825 INFO blockmanagement.BlockManager: maxReplication = 512
2024-04-09 07:25:35,825 INFO blockmanagement.BlockManager: minReplication = 1
2024-04-09 07:25:35,826 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
2024-04-09 07:25:35,826 INFO blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms
2024-04-09 07:25:35,826 INFO blockmanagement.BlockManager: encryptDataTransfer = false
2024-04-09 07:25:35,826 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
2024-04-09 07:25:35,836 INFO namenode.FSDirectory: GLOBAL serial map: bits=24 maxEntries=16777215
2024-04-09 07:25:35,845 INFO util.GSet: Computing capacity for map INodeMap
2024-04-09 07:25:35,845 INFO util.GSet: VM type = 64-bit
2024-04-09 07:25:35,845 INFO util.GSet: 1.0% max memory 3.9 GB = 39.7 MB
2024-04-09 07:25:35,845 INFO util.GSet: capacity = 2^22 = 4194304 entries
2024-04-09 07:25:35,850 INFO namenode.FSDirectory: ACLs enabled? false
2024-04-09 07:25:35,850 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2024-04-09 07:25:35,850 INFO namenode.FSDirectory: XAttrs enabled? true
2024-04-09 07:25:35,850 INFO namenode.NameNode: Caching file names occurring more than 10 times
2024-04-09 07:25:35,853 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2024-04-09 07:25:35,854 INFO snapshot.SnapshotManager: SkipList is disabled
2024-04-09 07:25:35,856 INFO util.GSet: Computing capacity for map cachedBlocks
2024-04-09 07:25:35,856 INFO util.GSet: VM type = 64-bit
2024-04-09 07:25:35,857 INFO util.GSet: 0.25% max memory 3.9 GB = 9.9 MB
2024-04-09 07:25:35,857 INFO util.GSet: capacity = 2^20 = 1048576 entries
2024-04-09 07:25:35,862 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2024-04-09 07:25:35,862 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2024-04-09 07:25:35,862 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2024-04-09 07:25:35,864 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2024-04-09 07:25:35,864 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2024-04-09 07:25:35,865 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2024-04-09 07:25:35,865 INFO util.GSet: VM type = 64-bit
2024-04-09 07:25:35,865 INFO util.GSet: 0.029999999329447746% max memory 3.9 GB = 1.2 MB
2024-04-09 07:25:35,865 INFO util.GSet: capacity = 2^17 = 131072 entries
Re-format filesystem in Storage Directory root= /tmp/hadoop-root/dfs/name; location= null ? (Y or N) ^C2024-04-09 07:26:22,548 ERROR namenode.NameNode: RECEIVED SIGNAL 2: SIGINT
2024-04-09 07:26:22,551 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at 98031e181845/172.17.0.2
************************************************************/
^Z
[1]+ Stopped hdfs namenode -format
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# start-all.sh
org.apache.spark.deploy.master.Master running as process 60899. Stop it first.
root@localhost's password:
localhost: org.apache.spark.deploy.worker.Worker running as process 50601. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# start-all.sh
org.apache.spark.deploy.master.Master running as process 60899. Stop it first.
root@localhost's password:
root@localhost's password: localhost: Permission denied, please try again.
localhost: Permission denied, please try again.
root@localhost's password:
localhost: root@localhost: Permission denied (publickey,password).
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# start-all.sh
org.apache.spark.deploy.master.Master running as process 60899. Stop it first.
root@localhost's password:
root@localhost's password: localhost: Permission denied, please try again.
localhost: Permission denied, please try again.
root@localhost's password:
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop#
[Alocalhost: root@localhost: Permission denied (publickey,password).
^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# sudo ^Cart-all.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# ls
capacity-scheduler.xml hadoop-metrics2.properties httpfs-signature.secret log4j.properties ssl-client.xml.example yarnservice-log4j.properties
configuration.xsl hadoop-policy.xml httpfs-site.xml mapred-env.cmd ssl-server.xml.example yarn-site.xml
container-executor.cfg hadoop-user-functions.sh.example kms-acls.xml mapred-env.sh user_ec_policies.xml.template
core-site.xml hdfs-site.xml kms-env.sh mapred-queues.xml.template workers
hadoop-env.cmd httpfs-env.sh kms-log4j.properties mapred-site.xml yarn-env.cmd
hadoop-env.sh httpfs-log4j.properties kms-site.xml shellprofile.d yarn-env.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# sudo ./start-all.sh
sudo: ./start-all.sh: command not found
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# sudo start-all.sh
sudo: start-all.sh: command not found
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# ls
capacity-scheduler.xml hadoop-metrics2.properties httpfs-signature.secret log4j.properties ssl-client.xml.example yarnservice-log4j.properties
configuration.xsl hadoop-policy.xml httpfs-site.xml mapred-env.cmd ssl-server.xml.example yarn-site.xml
container-executor.cfg hadoop-user-functions.sh.example kms-acls.xml mapred-env.sh user_ec_policies.xml.template
core-site.xml hdfs-site.xml kms-env.sh mapred-queues.xml.template workers
hadoop-env.cmd httpfs-env.sh kms-log4j.properties mapred-site.xml yarn-env.cmd
hadoop-env.sh httpfs-log4j.properties kms-site.xml shellprofile.d yarn-env.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# sudo ./start-all.sh^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# cd ../hadoop/
capacity-scheduler.xml hadoop-policy.xml kms-acls.xml mapred-queues.xml.template yarn-env.cmd
configuration.xsl hadoop-user-functions.sh.example kms-env.sh mapred-site.xml yarn-env.sh
container-executor.cfg hdfs-site.xml kms-log4j.properties shellprofile.d/ yarnservice-log4j.properties
core-site.xml httpfs-env.sh kms-site.xml ssl-client.xml.example yarn-site.xml
hadoop-env.cmd httpfs-log4j.properties log4j.properties ssl-server.xml.example
hadoop-env.sh httpfs-signature.secret mapred-env.cmd user_ec_policies.xml.template
hadoop-metrics2.properties httpfs-site.xml mapred-env.sh workers
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# cd ../hadoop/s
shellprofile.d/ ssl-client.xml.example ssl-server.xml.example
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/etc/hadoop# cd /data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ls
distribute-exclude.sh hadoop-daemons.sh mr-jobhistory-daemon.sh start-all.sh start-dfs.sh start-yarn.sh stop-balancer.sh stop-secure-dns.sh workers.sh
FederationStateStore httpfs.sh refresh-namenodes.sh start-balancer.sh start-secure-dns.sh stop-all.cmd stop-dfs.cmd stop-yarn.cmd yarn-daemon.sh
hadoop-daemon.sh kms.sh start-all.cmd start-dfs.cmd start-yarn.cmd stop-all.sh stop-dfs.sh stop-yarn.sh yarn-daemons.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# sudo ./start-all.sh
ERROR: JAVA_HOME is not set and could not be found.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# sudo ./s^Crt-all.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java to provide /usr/bin/java (java) in manual mode
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
* 2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
Press <enter> to keep the current choice[*], or type selection number: 2
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# sudo ./start-all.sh
ERROR: JAVA_HOME is not set and could not be found.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
* 2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
Press <enter> to keep the current choice[*], or type selection number: 2
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# sudo ./start-all.sh
ERROR: JAVA_HOME is not set and could not be found.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# gedit ~/.bashrc
(gedit:155851): dbind-WARNING **: 15:36:28.781: Couldn't connect to accessibility bus: Failed to connect to socket /run/user/1000/at-spi/bus_1: No such file or directory
(gedit:155851): GLib-GIO-CRITICAL **: 15:36:28.799: g_dbus_proxy_new_sync: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(gedit:155851): dconf-WARNING **: 15:36:28.804: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:155851): dconf-WARNING **: 15:36:28.807: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:155851): WARNING **: 15:36:28.898: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
(gedit:155851): dconf-WARNING **: 15:36:28.898: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:155851): dconf-WARNING **: 15:36:28.898: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:155851): dconf-WARNING **: 15:36:28.898: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.905: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:28.906: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:36.432: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:36.514: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:155851): WARNING **: 15:36:36.514: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:155851): WARNING **: 15:36:36.689: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:36.763: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:155851): WARNING **: 15:36:36.811: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:155851): WARNING **: 15:36:36.956: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:37.006: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:155851): WARNING **: 15:36:37.006: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:155851): WARNING **: 15:36:37.214: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:37.227: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:155851): WARNING **: 15:36:37.227: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:155851): WARNING **: 15:36:37.261: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:37.321: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:155851): WARNING **: 15:36:37.321: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:155851): WARNING **: 15:36:37.372: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:155851): WARNING **: 15:36:37.425: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:155851): WARNING **: 15:36:37.425: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:155851): WARNING **: 15:36:37.749: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:155851): WARNING **: 15:36:37.749: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# source ~/.bashrc
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# gedit ~/.bashrc
(gedit:156207): dbind-WARNING **: 15:36:53.547: Couldn't connect to accessibility bus: Failed to connect to socket /run/user/1000/at-spi/bus_1: No such file or directory
(gedit:156207): GLib-GIO-CRITICAL **: 15:36:53.565: g_dbus_proxy_new_sync: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(gedit:156207): dconf-WARNING **: 15:36:53.571: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:156207): dconf-WARNING **: 15:36:53.574: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:156207): WARNING **: 15:36:53.664: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
(gedit:156207): dconf-WARNING **: 15:36:53.665: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:156207): dconf-WARNING **: 15:36:53.665: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:156207): dconf-WARNING **: 15:36:53.665: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.672: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:53.673: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:59.664: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:59.712: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:156207): WARNING **: 15:36:59.712: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:156207): WARNING **: 15:36:59.836: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:156207): WARNING **: 15:36:59.894: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:156207): WARNING **: 15:36:59.894: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:156207): WARNING **: 15:37:01.178: Set document metadata failed: Setting attribute metadata::gedit-position not supported
(gedit:156207): dconf-WARNING **: 15:37:01.183: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# sudo ./start-all.sh
ERROR: JAVA_HOME is not set and could not be found.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh
org.apache.spark.deploy.master.Master running as process 60899. Stop it first.
root@localhost's password:
localhost: org.apache.spark.deploy.worker.Worker running as process 50601. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ls
distribute-exclude.sh hadoop-daemons.sh mr-jobhistory-daemon.sh start-all.sh start-dfs.sh start-yarn.sh stop-balancer.sh stop-secure-dns.sh workers.sh
FederationStateStore httpfs.sh refresh-namenodes.sh start-balancer.sh start-secure-dns.sh stop-all.cmd stop-dfs.cmd stop-yarn.cmd yarn-daemon.sh
hadoop-daemon.sh kms.sh start-all.cmd start-dfs.cmd start-yarn.cmd stop-all.sh stop-dfs.sh stop-yarn.sh yarn-daemons.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# stop-all.sh
root@localhost's password:
localhost: stopping org.apache.spark.deploy.worker.Worker
stopping org.apache.spark.deploy.master.Master
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# kill -9 60899 // 停止Spark Master进程
bash: kill: (60899) - No such process
bash: kill: //: arguments must be process or job IDs
bash: kill: 停止Spark: arguments must be process or job IDs
bash: kill: Master进程: arguments must be process or job IDs
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# kill -9 50601 // 停止Spark Worker进程
bash: kill: (50601) - No such process
bash: kill: //: arguments must be process or job IDs
bash: kill: 停止Spark: arguments must be process or job IDs
bash: kill: Worker进程: arguments must be process or job IDs
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-3.5.1/logs/spark--org.apache.spark.deploy.master.Master-1-98031e181845.out
failed to launch: nice -n 0 /opt/spark-3.5.1/bin/spark-class org.apache.spark.deploy.master.Master --host 98031e181845 --port 7077 --webui-port 8080
Spark Command: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /opt/spark-3.5.1/conf/:/opt/spark-3.5.1/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host 98031e181845 --port 7077 --webui-port 8080
========================================
full log in /opt/spark-3.5.1/logs/spark--org.apache.spark.deploy.master.Master-1-98031e181845.out
root@localhost's password:
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.5.1/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-98031e181845.out
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ls
distribute-exclude.sh hadoop-daemons.sh mr-jobhistory-daemon.sh start-all.sh start-dfs.sh start-yarn.sh stop-balancer.sh stop-secure-dns.sh workers.sh
FederationStateStore httpfs.sh refresh-namenodes.sh start-balancer.sh start-secure-dns.sh stop-all.cmd stop-dfs.cmd stop-yarn.cmd yarn-daemon.sh
hadoop-daemon.sh kms.sh start-all.cmd start-dfs.cmd start-yarn.cmd stop-all.sh stop-dfs.sh stop-yarn.sh yarn-daemons.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [98031e181845]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh
org.apache.spark.deploy.master.Master running as process 158414. Stop it first.
root@localhost's password:
localhost: org.apache.spark.deploy.worker.Worker running as process 158732. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# stop-all.sh
root@localhost's password:
localhost: stopping org.apache.spark.deploy.worker.Worker
stopping org.apache.spark.deploy.master.Master
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# stop-all.sh
root@localhost's password:
localhost: no org.apache.spark.deploy.worker.Worker to stop
no org.apache.spark.deploy.master.Master to stop
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [98031e181845]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# stop-all.sh
root@localhost's password:
localhost: no org.apache.spark.deploy.worker.Worker to stop
no org.apache.spark.deploy.master.Master to stop
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-3.5.1/logs/spark--org.apache.spark.deploy.master.Master-1-98031e181845.out
root@localhost's password:
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.5.1/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-98031e181845.out
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# stop-all.sh
root@localhost's password:
localhost: Connection closed by ::1 port 22
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ^C
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# stop-all.sh
root@localhost's password:
localhost: stopping org.apache.spark.deploy.worker.Worker
stopping org.apache.spark.deploy.master.Master
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-yarn.sh
bash: start-yarn.sh: command not found
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ls
distribute-exclude.sh hadoop-daemons.sh mr-jobhistory-daemon.sh start-all.sh start-dfs.sh start-yarn.sh stop-balancer.sh stop-secure-dns.sh workers.sh
FederationStateStore httpfs.sh refresh-namenodes.sh start-balancer.sh start-secure-dns.sh stop-all.cmd stop-dfs.cmd stop-yarn.cmd yarn-daemon.sh
hadoop-daemon.sh kms.sh start-all.cmd start-dfs.cmd start-yarn.cmd stop-all.sh stop-dfs.sh stop-yarn.sh yarn-daemons.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-yarn.sh
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [98031e181845]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin#
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin#
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin#
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin#
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# pwd
/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# gedit ~/.bashrc
(gedit:166641): dbind-WARNING **: 15:49:50.908: Couldn't connect to accessibility bus: Failed to connect to socket /run/user/1000/at-spi/bus_1: No such file or directory
(gedit:166641): GLib-GIO-CRITICAL **: 15:49:50.926: g_dbus_proxy_new_sync: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(gedit:166641): dconf-WARNING **: 15:49:50.932: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:166641): dconf-WARNING **: 15:49:50.935: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:166641): WARNING **: 15:49:51.028: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
(gedit:166641): dconf-WARNING **: 15:49:51.028: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:166641): dconf-WARNING **: 15:49:51.028: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:166641): dconf-WARNING **: 15:49:51.028: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:166641): WARNING **: 15:49:51.035: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.035: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.035: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.036: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.037: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.037: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.037: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.037: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.037: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:166641): WARNING **: 15:49:51.037: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
(gedit:166641): dconf-WARNING **: 15:50:50.370: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# pwd
/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# export HADOOP_HOME=/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [98031e181845]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-3.5.1/logs/spark--org.apache.spark.deploy.master.Master-1-98031e181845.out
root@localhost's password:
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.5.1/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-98031e181845.out
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# gedit ~/.bashrc
(gedit:168769): dbind-WARNING **: 15:51:58.255: Couldn't connect to accessibility bus: Failed to connect to socket /run/user/1000/at-spi/bus_1: No such file or directory
(gedit:168769): GLib-GIO-CRITICAL **: 15:51:58.273: g_dbus_proxy_new_sync: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(gedit:168769): dconf-WARNING **: 15:51:58.278: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:168769): dconf-WARNING **: 15:51:58.281: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:168769): WARNING **: 15:51:58.406: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
(gedit:168769): dconf-WARNING **: 15:51:58.407: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:168769): dconf-WARNING **: 15:51:58.407: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:168769): dconf-WARNING **: 15:51:58.407: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.417: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.418: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:51:58.419: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:10.520: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:10.645: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:10.645: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:10.698: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:10.750: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:10.750: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:13.814: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:13.915: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:13.915: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:25.170: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:25.259: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:25.259: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:25.366: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:25.419: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:25.419: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:25.609: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:25.618: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:25.618: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:25.778: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:25.833: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:25.833: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:25.960: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:25.987: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:25.987: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:26.204: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:26.353: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:26.353: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:30.380: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:30.476: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:30.476: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:30.613: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:30.669: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:30.670: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:33.462: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:33.531: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:33.531: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:33.621: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:33.680: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:33.680: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:34.576: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:34.701: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:34.701: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:168769): WARNING **: 15:52:34.757: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:168769): WARNING **: 15:52:34.817: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:168769): WARNING **: 15:52:34.817: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
(gedit:168769): dconf-WARNING **: 15:52:35.935: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# source ~/.bashrc
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh
org.apache.spark.deploy.master.Master running as process 168316. Stop it first.
root@localhost's password:
localhost: org.apache.spark.deploy.worker.Worker running as process 168537. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# hdfs namenode -format
namenode is running as process 148907. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh
org.apache.spark.deploy.master.Master running as process 168316. Stop it first.
root@localhost's password:
localhost: org.apache.spark.deploy.worker.Worker running as process 168537. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# source ~/.bashrc
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# gedit ~/.bashrc
(gedit:170624): dbind-WARNING **: 15:54:35.151: Couldn't connect to accessibility bus: Failed to connect to socket /run/user/1000/at-spi/bus_1: No such file or directory
(gedit:170624): GLib-GIO-CRITICAL **: 15:54:35.468: g_dbus_proxy_new_sync: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(gedit:170624): dconf-WARNING **: 15:54:35.902: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:170624): dconf-WARNING **: 15:54:35.906: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:170624): WARNING **: 15:54:36.443: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
(gedit:170624): dconf-WARNING **: 15:54:36.444: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:170624): dconf-WARNING **: 15:54:36.444: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(gedit:170624): dconf-WARNING **: 15:54:36.444: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
** (gedit:170624): WARNING **: 15:54:36.495: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.495: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.495: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.495: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.495: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.496: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:36.497: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:50.380: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:50.476: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:170624): WARNING **: 15:54:50.476: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:170624): WARNING **: 15:54:50.963: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:51.008: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:170624): WARNING **: 15:54:51.008: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:170624): WARNING **: 15:54:51.413: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme Yaru-magenta
** (gedit:170624): WARNING **: 15:54:51.489: Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported
** (gedit:170624): WARNING **: 15:54:51.489: Set document metadata failed: Setting attribute metadata::gedit-encoding not supported
** (gedit:170624): WARNING **: 15:54:53.029: Set document metadata failed: Setting attribute metadata::gedit-position not supported
(gedit:170624): dconf-WARNING **: 15:54:53.034: failed to commit changes to dconf: Failed to execute child process “dbus-launch” (No such file or directory)
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# source ~/.bashrc
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh
org.apache.spark.deploy.master.Master running as process 168316. Stop it first.
root@localhost's password:
localhost: org.apache.spark.deploy.worker.Worker running as process 168537. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# start-all.sh
org.apache.spark.deploy.master.Master running as process 168316. Stop it first.
root@localhost's password:
localhost: org.apache.spark.deploy.worker.Worker running as process 168537. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ls
distribute-exclude.sh hadoop-daemons.sh mr-jobhistory-daemon.sh start-all.sh start-dfs.sh start-yarn.sh stop-balancer.sh stop-secure-dns.sh workers.sh
FederationStateStore httpfs.sh refresh-namenodes.sh start-balancer.sh start-secure-dns.sh stop-all.cmd stop-dfs.cmd stop-yarn.cmd yarn-daemon.sh
hadoop-daemon.sh kms.sh start-all.cmd start-dfs.cmd start-yarn.cmd stop-all.sh stop-dfs.sh stop-yarn.sh yarn-daemons.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-yarn.sh
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [98031e181845]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin#
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3# java –version
Error: Could not find or load main class –version
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3# java --version
Unrecognized option: --version
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3# java -version
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3# ../sbin/start-all.sh
bash: ../sbin/start-all.sh: No such file or directory
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3# cd s
sbin/ share/
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3# cd sbin/
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ls
distribute-exclude.sh hadoop-daemons.sh mr-jobhistory-daemon.sh start-all.sh start-dfs.sh start-yarn.sh stop-balancer.sh stop-secure-dns.sh workers.sh
FederationStateStore httpfs.sh refresh-namenodes.sh start-balancer.sh start-secure-dns.sh stop-all.cmd stop-dfs.cmd stop-yarn.cmd yarn-daemon.sh
hadoop-daemon.sh kms.sh start-all.cmd start-dfs.cmd start-yarn.cmd stop-all.sh stop-dfs.sh stop-yarn.sh yarn-daemons.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./sbin/start-all.sh
bash: ./sbin/start-all.sh: No such file or directory
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ls
distribute-exclude.sh hadoop-daemons.sh mr-jobhistory-daemon.sh start-all.sh start-dfs.sh start-yarn.sh stop-balancer.sh stop-secure-dns.sh workers.sh
FederationStateStore httpfs.sh refresh-namenodes.sh start-balancer.sh start-secure-dns.sh stop-all.cmd stop-dfs.cmd stop-yarn.cmd yarn-daemon.sh
hadoop-daemon.sh kms.sh start-all.cmd start-dfs.cmd start-yarn.cmd stop-all.sh stop-dfs.sh stop-yarn.sh yarn-daemons.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [98031e181845]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# export HDFS_NAMENODE_USER="root"
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# export HDFS_DATANODE_USER="root"
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# export HDFS_SECONDARYNAMENODE_USER="root"
port YARN_NODEMANAGER_USER="root"(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# export YARN_RESOURCEMANAGER_USER="root"
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# export YARN_NODEMANAGER_USER="root"
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
localhost: namenode is running as process 10230. Stop it first.
Starting datanodes
localhost: datanode is running as process 10628. Stop it first.
Starting secondary namenodes [98031e181845]
Starting resourcemanager
resourcemanager is running as process 29637. Stop it first.
Starting nodemanagers
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
localhost: namenode is running as process 10230. Stop it first.
Starting datanodes
localhost: datanode is running as process 10628. Stop it first.
Starting secondary namenodes [98031e181845]
98031e181845: secondarynamenode is running as process 52321. Stop it first.
Starting resourcemanager
resourcemanager is running as process 29637. Stop it first.
Starting nodemanagers
localhost: nodemanager is running as process 52830. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# jps
52321 SecondaryNameNode
10628 DataNode
29637 ResourceManager
10230 NameNode
56060 Jps
52830 NodeManager
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
localhost: namenode is running as process 10230. Stop it first.
Starting datanodes
localhost: datanode is running as process 10628. Stop it first.
Starting secondary namenodes [98031e181845]
98031e181845: secondarynamenode is running as process 52321. Stop it first.
Starting resourcemanager
resourcemanager is running as process 29637. Stop it first.
Starting nodemanagers
localhost: nodemanager is running as process 52830. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ls
distribute-exclude.sh hadoop-daemons.sh mr-jobhistory-daemon.sh start-all.sh start-dfs.sh start-yarn.sh stop-balancer.sh stop-secure-dns.sh workers.sh
FederationStateStore httpfs.sh refresh-namenodes.sh start-balancer.sh start-secure-dns.sh stop-all.cmd stop-dfs.cmd stop-yarn.cmd yarn-daemon.sh
hadoop-daemon.sh kms.sh start-all.cmd start-dfs.cmd start-yarn.cmd stop-all.sh stop-dfs.sh stop-yarn.sh yarn-daemons.sh
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ../bin/hdfs namenode -format
namenode is running as process 10230. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
localhost: namenode is running as process 10230. Stop it first.
Starting datanodes
localhost: datanode is running as process 10628. Stop it first.
Starting secondary namenodes [98031e181845]
98031e181845: secondarynamenode is running as process 52321. Stop it first.
Starting resourcemanager
resourcemanager is running as process 29637. Stop it first.
Starting nodemanagers
localhost: nodemanager is running as process 52830. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
localhost: namenode is running as process 10230. Stop it first.
Starting datanodes
localhost: datanode is running as process 10628. Stop it first.
Starting secondary namenodes [98031e181845]
98031e181845: secondarynamenode is running as process 52321. Stop it first.
Starting resourcemanager
resourcemanager is running as process 29637. Stop it first.
Starting nodemanagers
localhost: nodemanager is running as process 52830. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# stop-all.sh
Stopping namenodes on [localhost]
Stopping datanodes
Stopping secondary namenodes [98031e181845]
Stopping nodemanagers
Stopping resourcemanager
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [98031e181845]
Starting resourcemanager
Starting nodemanagers
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./sbin/start-all.sh -P <您的SSH端口号>
bash: syntax error near unexpected token `newline'
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./sbin/start-all.sh -P 2699
bash: ./sbin/start-all.sh: No such file or directory
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh -P 2699
Starting namenodes on [localhost]
localhost: namenode is running as process 63282. Stop it first.
Starting datanodes
localhost: datanode is running as process 63523. Stop it first.
Starting secondary namenodes [98031e181845]
98031e181845: secondarynamenode is running as process 63865. Stop it first.
Starting resourcemanager
resourcemanager is running as process 64239. Stop it first.
Starting nodemanagers
localhost: nodemanager is running as process 64458. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# stop-all.sh
Stopping namenodes on [localhost]
Stopping datanodes
Stopping secondary namenodes [98031e181845]
Stopping nodemanagers
Stopping resourcemanager
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh -P 2699
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [98031e181845]
Starting resourcemanager
Starting nodemanagers
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# stop-all.sh
Stopping namenodes on [localhost]
Stopping datanodes
Stopping secondary namenodes [98031e181845]
Stopping nodemanagers
Stopping resourcemanager
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh -P 2699
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [98031e181845]
Starting resourcemanager
Starting nodemanagers
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# hdfs namenode -format
namenode is running as process 71854. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin# ./start-all.sh -P 2699
Starting namenodes on [localhost]
localhost: namenode is running as process 71854. Stop it first.
Starting datanodes
localhost: datanode is running as process 72115. Stop it first.
Starting secondary namenodes [98031e181845]
98031e181845: secondarynamenode is running as process 72519. Stop it first.
Starting resourcemanager
resourcemanager is running as process 72893. Stop it first.
Starting nodemanagers
localhost: nodemanager is running as process 73233. Stop it first.
(tfv1) root@98031e181845:/data/myapp2024/myBigDataApplicationDevelopment/hadoop-3.1.3/sbin#
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。