热门标签
热门文章
- 1（十三）springboot实战——springboot前后端分离方式项目集成spring securtity安全框架_springboot前后端分离项目
- 2使用LangChain和Neo4j快速创建RAG应用_langchain neo4j
- 3计算机毕业设计Node.js+uniapp校园设备报修小程序(源码+程序+lw+远程调试)_电脑维修小程序源码
- 4机器学习的五大流派
- 5如何在 VsCode 中从 Github 运行 Flutter 项目到您的系统并在您的设备中安装 Flutter App？_vs code怎么运行flutter工程
- 6Webkit简介以及工作流程
- 7利用python爬虫可视化分析当当网的图书数据_python爬取当当网数据并可视化分析_当当网爬虫数据可视化
- 8图像相似度比较算法总结_图片相似度算法
- 9linux下常用命令以及安装redis和启动_redis5.1安装
- 10当单片机遇上状态机（二） 为什么QP难以入门？_qp状态机为什么用的少
当前位置: article > 正文
大数据平台运维之MapReduce_mapreaduce运维
作者：人工智能uu | 2024-07-20 04:50:16
赞
踩
mapreaduce运维
大数据系列之运维（自主搭建的大数据平台）
（2）MapReduce运维
- 存在一个案例JAR 包 hadoop-mapreduce-examples.jar。运行 JAR 包中的 PI 程序来进行计算圆周率π的近似值，要求运行 5 次 Map 任务，每个 Map 任务的投掷次数为 5。
[root@master ~]# hadoop jar /usr/hadoop/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 5 5 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Wrote input for Map #4 Starting Job 20/03/29 22:04:42 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.100.160:18040 20/03/29 22:04:43 INFO input.FileInputFormat: Total input paths to process : 5 20/03/29 22:04:44 INFO mapreduce.JobSubmitter: number of splits:5 20/03/29 22:04:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1585482514288_0001 20/03/29 22:04:45 INFO impl.YarnClientImpl: Submitted application application_1585482514288_0001 20/03/29 22:04:45 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1585482514288_0001/ 20/03/29 22:04:45 INFO mapreduce.Job: Running job: job_1585482514288_0001 20/03/29 22:04:58 INFO mapreduce.Job: Job job_1585482514288_0001 running in uber mode : false 20/03/29 22:04:58 INFO mapreduce.Job: map 0% reduce 0% 20/03/29 22:05:43 INFO mapreduce.Job: map 20% reduce 0% 20/03/29 22:05:44 INFO mapreduce.Job: map 40% reduce 0% 20/03/29 22:05:45 INFO mapreduce.Job: map 100% reduce 0% 20/03/29 22:05:53 INFO mapreduce.Job: map 100% reduce 100% 20/03/29 22:05:54 INFO mapreduce.Job: Job job_1585482514288_0001 completed successfully 20/03/29 22:05:55 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=116 FILE: Number of bytes written=714927 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1300 HDFS: Number of bytes written=215 HDFS: Number of read operations=23 HDFS: Number of large read operations=0 HDFS: Number of write operations=3 Job Counters Launched map tasks=5 Launched reduce tasks=1 Data-local map tasks=5 Total time spent by all maps in occupied slots (ms)=215089 Total time spent by all reduces in occupied slots (ms)=7814 Total time spent by all map tasks (ms)=215089 Total time spent by all reduce tasks (ms)=7814 Total vcore-milliseconds taken by all map tasks=215089 Total vcore-milliseconds taken by all reduce tasks=7814 Total megabyte-milliseconds taken by all map tasks=220251136 Total megabyte-milliseconds taken by all reduce tasks=8001536 Map-Reduce Framework Map input records=5 Map output records=10 Map output bytes=90 Map output materialized bytes=140 Input split bytes=710 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=140 Reduce input records=10 Reduce output records=0 Spilled Records=20 Shuffled Maps =5 Failed Shuffles=0 Merged Map outputs=5 GC time elapsed (ms)=12761 CPU time spent (ms)=31390 Physical memory (bytes) snapshot=1536512000 Virtual memory (bytes) snapshot=12615630848 Total committed heap usage (bytes)=1083703296 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=590 File Output Format Counters Bytes Written=97 Job Finished in 72.642 seconds Estimated value of Pi is 3.68000000000000000000
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 存在一个案例JAR 包 hadoop-mapreduce-examples.jar。运行 JAR 包中的 wordcount 程序来对/1daoyun/file/BigDataSkills.txt 文件进行单词计数，将运算结果输出到/1daoyun/output 目录中，使用相关命令查询单词计数结果。
[root@master ~]# hadoop jar /usr/hadoop/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /1daoyun/file/BigDataSkills.txt /1daoyun/output 20/03/29 22:12:01 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.100.160:18040 20/03/29 22:12:02 INFO input.FileInputFormat: Total input paths to process : 1 20/03/29 22:12:03 INFO mapreduce.JobSubmitter: number of splits:1 20/03/29 22:12:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1585482514288_0002 20/03/29 22:12:04 INFO impl.YarnClientImpl: Submitted application application_1585482514288_0002 20/03/29 22:12:04 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1585482514288_0002/ 20/03/29 22:12:04 INFO mapreduce.Job: Running job: job_1585482514288_0002 20/03/29 22:12:13 INFO mapreduce.Job: Job job_1585482514288_0002 running in uber mode : false 20/03/29 22:12:13 INFO mapreduce.Job: map 0% reduce 0% 20/03/29 22:12:21 INFO mapreduce.Job: map 100% reduce 0% 20/03/29 22:12:28 INFO mapreduce.Job: map 100% reduce 100% 20/03/29 22:12:28 INFO mapreduce.Job: Job job_1585482514288_0002 completed successfully 20/03/29 22:12:29 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=1712 FILE: Number of bytes written=241031 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1289 HDFS: Number of bytes written=1203 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5804 Total time spent by all reduces in occupied slots (ms)=4842 Total time spent by all map tasks (ms)=5804 Total time spent by all reduce tasks (ms)=4842 Total vcore-milliseconds taken by all map tasks=5804 Total vcore-milliseconds taken by all reduce tasks=4842 Total megabyte-milliseconds taken by all map tasks=5943296 Total megabyte-milliseconds taken by all reduce tasks=4958208 Map-Reduce Framework Map input records=4 Map output records=174 Map output bytes=1868 Map output materialized bytes=1712 Input split bytes=114 Combine input records=174 Combine output records=126 Reduce input groups=126 Reduce shuffle bytes=1712 Reduce input records=126 Reduce output records=126 Spilled Records=252 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=268 CPU time spent (ms)=2260 Physical memory (bytes) snapshot=433119232 Virtual memory (bytes) snapshot=4201779200 Total committed heap usage (bytes)=298844160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=1175 File Output Format Counters Bytes Written=1203 [root@master ~]# hadoop fs -ls /1daoyun/output #看一下这个文件里输出的是什么 [root@master ~]# hadoop fs -cat /1daoyun/output/part-r-00000 (Hadoop 1 2) 1 Consequently, 1 Hadoop 6 However, 1 Its 1 Java 1 Organizations 1 Since 1 T 1 The 1 This 1 a 5 allows 1 among 1 analytics, 1 and 9 applications 1 approach 1 as 2 based 1 become 1 been 1 big 3 business 1 can 1 case 1 catastrophic 1 center. 1 commodity 1 components 1 computing 1 continue 1 continuously 1 data 6 data, 1 data. 1 depend 1 deploy 1 developed 1 distributed 2 emerged 1 enormous 1 even 1 facilitates 1 failure 1 failure. 1 file 2 for 1 foundation 1 framework 1 from 1 handle 1 hardware 1 has 1 helps 1 if 1 improved 1 in 2 including 1 initial 1 inoperative. 1 internet 1 is 1 it 1 iteration 1 its 1 local 1 loss, 1 lowers 1 makes 1 management 1 most 1 node 1 nodes 2 nodes, 1 number 1 of 10 offers 1 on 3 operating 1 organizations 1 packages 1 planning, 1 possible 1 process 1 processing 2 programming 1 projects 1 quickly 1 rapid 1 rates 1 release, 1 resource 1 resources. 1 risk 1 run 1 sales 1 scheduling. 1 scientific 1 second 1 sensor 1 sensors. 1 sets. 1 short-term 1 significant 1 software 1 substantial 1 such 1 supporting 1 system 4 systems 1 tasks, 1 terabytes 1 that 1 the 2 their 1 things 1 thousands 2 to 3 transfer 1 unexpected 1 updated. 1 use 1 volumes 1 with 1
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
- 131
- 132
- 133
- 134
- 135
- 136
- 137
- 138
- 139
- 140
- 141
- 142
- 143
- 144
- 145
- 146
- 147
- 148
- 149
- 150
- 151
- 152
- 153
- 154
- 155
- 156
- 157
- 158
- 159
- 160
- 161
- 162
- 163
- 164
- 165
- 166
- 167
- 168
- 169
- 170
- 171
- 172
- 173
- 174
- 175
- 176
- 177
- 178
- 179
- 180
- 181
- 182
- 183
- 184
- 185
- 186
- 187
- 188
- 189
- 190
- 191
- 192
- 193
- 194
- 195
- 196
- 197
- 存在一个案例JAR 包 hadoop-mapreduce-examples.jar。运行 JAR 包中的 sudoku 程序来计算下表中数独运算题的结果。
[root@master ~]# cat /root/tiku/MapReduce/puzzle1.dta #此文件是在本地系统file，而不是HDFS上
8 ? ? ? ? ? ? ? ?
? ? 3 6 ? ? ? ? ?
? 7 ? ? 9 ? 2 ? ?
? 5 ? ? ? 7 ? ? ?
? ? ? ? 4 5 7 ? ?
? ? ? 1 ? ? ? 3 ?
? ? 1 ? ? ? ? 6 8
? ? 8 5 ? ? ? 1 ?
? 9 ? ? ? ? 4 ? ?
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
[root@master ~]# hadoop jar /usr/hadoop/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar sudoku /root/tiku/MapReduce/puzzle1.dta
Solving /root/tiku/MapReduce/puzzle1.dta
8 1 2 7 5 3 6 4 9
9 4 3 6 8 2 1 7 5
6 7 5 4 9 1 2 8 3
1 5 4 2 3 7 8 9 6
3 6 9 8 4 5 7 2 1
2 8 7 1 6 9 5 3 4
5 2 1 9 7 4 3 6 8
4 3 8 5 2 6 9 1 7
7 9 6 3 1 8 4 5 2
Found 1 solutions
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 存在一个案例JAR 包 hadoop-mapreduce-examples.jar。运行 JAR 包中的 grep 程序来统计文件系统中/1daoyun/file/BigDataSkills.txt 文件中“Hadoop”出现的次数，统计完成后，查询统计结果信息。
[root@master ~]# hadoop jar /usr/hadoop/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep /1daoyun/file/BigDataSkills.txt /output Hadoop 20/03/29 22:31:39 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.100.160:18040 20/03/29 22:31:41 INFO input.FileInputFormat: Total input paths to process : 1 20/03/29 22:31:41 INFO mapreduce.JobSubmitter: number of splits:1 20/03/29 22:31:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1585482514288_0003 20/03/29 22:31:41 INFO impl.YarnClientImpl: Submitted application application_1585482514288_0003 20/03/29 22:31:41 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1585482514288_0003/ 20/03/29 22:31:41 INFO mapreduce.Job: Running job: job_1585482514288_0003 20/03/29 22:31:51 INFO mapreduce.Job: Job job_1585482514288_0003 running in uber mode : false 20/03/29 22:31:51 INFO mapreduce.Job: map 0% reduce 0% 20/03/29 22:31:58 INFO mapreduce.Job: map 100% reduce 0% 20/03/29 22:32:05 INFO mapreduce.Job: map 100% reduce 100% 20/03/29 22:32:05 INFO mapreduce.Job: Job job_1585482514288_0003 completed successfully 20/03/29 22:32:05 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=23 FILE: Number of bytes written=238359 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1289 HDFS: Number of bytes written=109 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=4370 Total time spent by all reduces in occupied slots (ms)=4518 Total time spent by all map tasks (ms)=4370 Total time spent by all reduce tasks (ms)=4518 Total vcore-milliseconds taken by all map tasks=4370 Total vcore-milliseconds taken by all reduce tasks=4518 Total megabyte-milliseconds taken by all map tasks=4474880 Total megabyte-milliseconds taken by all reduce tasks=4626432 Map-Reduce Framework Map input records=4 Map output records=7 Map output bytes=105 Map output materialized bytes=23 Input split bytes=114 Combine input records=7 Combine output records=1 Reduce input groups=1 Reduce shuffle bytes=23 Reduce input records=1 Reduce output records=1 Spilled Records=2 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=240 CPU time spent (ms)=2190 Physical memory (bytes) snapshot=434061312 Virtual memory (bytes) snapshot=4208771072 Total committed heap usage (bytes)=293076992 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=1175 File Output Format Counters Bytes Written=109 20/03/29 22:32:05 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.100.160:18040 20/03/29 22:32:06 INFO input.FileInputFormat: Total input paths to process : 1 20/03/29 22:32:06 INFO mapreduce.JobSubmitter: number of splits:1 20/03/29 22:32:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1585482514288_0004 20/03/29 22:32:06 INFO impl.YarnClientImpl: Submitted application application_1585482514288_0004 20/03/29 22:32:06 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1585482514288_0004/ 20/03/29 22:32:06 INFO mapreduce.Job: Running job: job_1585482514288_0004 20/03/29 22:32:19 INFO mapreduce.Job: Job job_1585482514288_0004 running in uber mode : false 20/03/29 22:32:19 INFO mapreduce.Job: map 0% reduce 0% 20/03/29 22:32:25 INFO mapreduce.Job: map 100% reduce 0% 20/03/29 22:32:33 INFO mapreduce.Job: map 100% reduce 100% 20/03/29 22:32:33 INFO mapreduce.Job: Job job_1585482514288_0004 completed successfully 20/03/29 22:32:33 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=23 FILE: Number of bytes written=237287 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=235 HDFS: Number of bytes written=9 HDFS: Number of read operations=7 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=4135 Total time spent by all reduces in occupied slots (ms)=4724 Total time spent by all map tasks (ms)=4135 Total time spent by all reduce tasks (ms)=4724 Total vcore-milliseconds taken by all map tasks=4135 Total vcore-milliseconds taken by all reduce tasks=4724 Total megabyte-milliseconds taken by all map tasks=4234240 Total megabyte-milliseconds taken by all reduce tasks=4837376 Map-Reduce Framework Map input records=1 Map output records=1 Map output bytes=15 Map output materialized bytes=23 Input split bytes=126 Combine input records=0 Combine output records=0 Reduce input groups=1 Reduce shuffle bytes=23 Reduce input records=1 Reduce output records=1 Spilled Records=2 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=190 CPU time spent (ms)=2010 Physical memory (bytes) snapshot=449654784 Virtual memory (bytes) snapshot=4206751744 Total committed heap usage (bytes)=292552704 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=109 File Output Format Counters Bytes Written=9
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
- 131
- 132
- 133
- 134
- 135
- 136
- 137
[root@master ~]# hadoop fs -ls /output #检查该文件下有什么输出。/output 不用先创建
[root@master ~]# hadoop fs -cat /output/part-r-00000
7 Hadoop
- 1
- 2
- 3
在此感谢先电云提供的题库。
感谢Apache开源技术服务支持
感谢抛物线、mn525520、菜鸟一枚2019三位博主的相关博客。
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/人工智能uu/article/detail/855224
推荐阅读
相关标签
