当前位置:   article > 正文

基于Hadoop平台的电信客服数据的处理与分析③项目开发:搭建基于Hadoop的全分布式集群---任务7:格式化并启动Hadoop集群

基于Hadoop平台的电信客服数据的处理与分析③项目开发:搭建基于Hadoop的全分布式集群---任务7:格式化并启动Hadoop集群

任务描述

任务内容为格式化并启动Hadoop集群,并修复可能出现的Bug。

任务指导

Hadoop集群启动前需要在NameNode上格式化元数据,成功格式化后才能启动Hadoop的HDFS和YARN。

格式化启动Hadoop集群的步骤如下:

1. 在NameNode(master1)格式化Hadoop的元数据(只需要第一次启动集群时执行一次)

2. 启动HDFS集群

3. 启动YARN集群

任务实现

1. 格式化集群的NameNode(在master1执行)

[root@master1 ~]# hdfs namenode -format

回显如下:

  1. [root@master1 ~]# hdfs namenode -format
  2. 23/10/18 08:57:10 INFO namenode.NameNode: STARTUP_MSG:
  3. /************************************************************
  4. STARTUP_MSG: Starting NameNode
  5. STARTUP_MSG: host = master1/192.168.3.129
  6. STARTUP_MSG: args = [-format]
  7. STARTUP_MSG: version = 2.10.1
  8. STARTUP_MSG: classpath = ...略
  9. STARTUP_MSG: build = https://github.com/apache/hadoop -r 1827467c9a56f133025f28557bfc2c562d78e816; compiled by 'centos' on 2020-09-14T13:17Z
  10. STARTUP_MSG: java = 1.8.0_181
  11. ************************************************************/
  12. 23/10/18 08:57:10 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
  13. 23/10/18 08:57:10 INFO namenode.NameNode: createNameNode [-format]
  14. Formatting using clusterid: CID-c67c639d-7eec-459d-9b60-3d5e696ccce8
  15. 23/10/18 08:57:10 INFO namenode.FSEditLog: Edit logging is async:true
  16. 23/10/18 08:57:10 INFO namenode.FSNamesystem: KeyProvider: null
  17. 23/10/18 08:57:10 INFO namenode.FSNamesystem: fsLock is fair: true
  18. 23/10/18 08:57:10 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
  19. 23/10/18 08:57:10 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)
  20. 23/10/18 08:57:10 INFO namenode.FSNamesystem: supergroup = supergroup
  21. 23/10/18 08:57:10 INFO namenode.FSNamesystem: isPermissionEnabled = false
  22. 23/10/18 08:57:10 INFO namenode.FSNamesystem: HA Enabled: false
  23. 23/10/18 08:57:10 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
  24. 23/10/18 08:57:10 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
  25. 23/10/18 08:57:10 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
  26. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
  27. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: The block deletion will start around 2023 Oct 18 08:57:10
  28. 23/10/18 08:57:10 INFO util.GSet: Computing capacity for map BlocksMap
  29. 23/10/18 08:57:10 INFO util.GSet: VM type = 64-bit
  30. 23/10/18 08:57:10 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
  31. 23/10/18 08:57:10 INFO util.GSet: capacity = 2^21 = 2097152 entries
  32. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
  33. 23/10/18 08:57:10 WARN conf.Configuration: No unit for dfs.heartbeat.interval(3) assuming SECONDS
  34. 23/10/18 08:57:10 WARN conf.Configuration: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
  35. 23/10/18 08:57:10 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
  36. 23/10/18 08:57:10 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
  37. 23/10/18 08:57:10 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
  38. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: defaultReplication = 2
  39. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: maxReplication = 512
  40. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: minReplication = 1
  41. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
  42. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
  43. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: encryptDataTransfer = false
  44. 23/10/18 08:57:10 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
  45. 23/10/18 08:57:10 INFO namenode.FSNamesystem: Append Enabled: true
  46. 23/10/18 08:57:10 INFO namenode.FSDirectory: GLOBAL serial map: bits=24 maxEntries=16777215
  47. 23/10/18 08:57:10 INFO util.GSet: Computing capacity for map INodeMap
  48. 23/10/18 08:57:10 INFO util.GSet: VM type = 64-bit
  49. 23/10/18 08:57:10 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
  50. 23/10/18 08:57:10 INFO util.GSet: capacity = 2^20 = 1048576 entries
  51. 23/10/18 08:57:10 INFO namenode.FSDirectory: ACLs enabled? false
  52. 23/10/18 08:57:10 INFO namenode.FSDirectory: XAttrs enabled? true
  53. 23/10/18 08:57:10 INFO namenode.NameNode: Caching file names occurring more than 10 times
  54. 23/10/18 08:57:10 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: falseskipCaptureAccessTimeOnlyChange: false
  55. 23/10/18 08:57:10 INFO util.GSet: Computing capacity for map cachedBlocks
  56. 23/10/18 08:57:10 INFO util.GSet: VM type = 64-bit
  57. 23/10/18 08:57:10 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
  58. 23/10/18 08:57:10 INFO util.GSet: capacity = 2^18 = 262144 entries
  59. 23/10/18 08:57:10 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
  60. 23/10/18 08:57:10 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
  61. 23/10/18 08:57:10 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
  62. 23/10/18 08:57:10 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
  63. 23/10/18 08:57:10 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
  64. 23/10/18 08:57:10 INFO util.GSet: Computing capacity for map NameNodeRetryCache
  65. 23/10/18 08:57:10 INFO util.GSet: VM type = 64-bit
  66. 23/10/18 08:57:10 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
  67. 23/10/18 08:57:10 INFO util.GSet: capacity = 2^15 = 32768 entries
  68. 23/10/18 08:57:10 INFO namenode.FSImage: Allocated new BlockPoolId: BP-894844368-192.168.3.129-1697619430610
  69. 23/10/18 08:57:10 INFO common.Storage: Storage directory /opt/app/hadoop_path/hdfs/name has been successfully formatted.
  70. 23/10/18 08:57:10 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/app/hadoop_path/hdfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
  71. 23/10/18 08:57:10 INFO namenode.FSImageFormatProtobuf: Image file /opt/app/hadoop_path/hdfs/name/current/fsimage.ckpt_0000000000000000000 of size 322 bytes saved in 0 seconds .
  72. 23/10/18 08:57:10 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
  73. 23/10/18 08:57:10 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid = 0 when meet shutdown.
  74. 23/10/18 08:57:10 INFO namenode.NameNode: SHUTDOWN_MSG:
  75. /************************************************************
  76. SHUTDOWN_MSG: Shutting down NameNode at master1/192.168.3.129
  77. ************************************************************/

回显的日志中未出现报错信息代表NameNode格式化成功。

此时在NameNode的元数据目录中会生成【current】文件夹,其中保存了HDFS的元数据文件,如下:

  1. [root@master1 name]# cd /opt/app/hadoop_path/hdfs/name
  2. [root@master1 name]# ll
  3. total 0
  4. drwxr-xr-x 2 root root 112 Oct 18 08:57 current
  5. [root@master1 name]# ll current/
  6. total 16
  7. -rw-r--r-- 1 root root 322 Oct 18 08:57 fsimage_0000000000000000000
  8. -rw-r--r-- 1 root root 62 Oct 18 08:57 fsimage_0000000000000000000.md5
  9. -rw-r--r-- 1 root root 2 Oct 18 08:57 seen_txid
  10. -rw-r--r-- 1 root root 215 Oct 18 08:57 VERSION

2.  启动HDFS集群,在master1上执行

[root@master1 ~]# start-dfs.sh

3. 启动YARN集群,在master1上执行

[root@master1 ~]# start-yarn.sh

  查看集群中的守护进程

全部启动完后分别在各个服务器上执行jps是可以看到下面这些进程的,输入【jps】命令,可以查看启动的守护进程,分别为:

  1. master1:NameNode、ResourceManager
  2. 从slave1:DataNode、NodeManager
  3. 从slave2:DataNode、NodeManager、SecondaryNameNode
本文内容由网友自发贡献,转载请注明出处:https://www.wpsshop.cn/w/黑客灵魂/article/detail/785723
推荐阅读
相关标签
  

闽ICP备14008679号