Hadoop框架下运行MapReduce程序_将mapreduce程序打包,在hadoop环境下运行程序并观察结果;

本文介绍了在Linux中Hadoop环境下，利用mapReduce框架写wordCount应用程序的主要方法，并且提供程序的解释说明。

1. 首先在工程中创建一个package：my.examples.hadoop.mr，在这个包下新建一个class：WCMapper；再新建一个class：WCReducer；最后新建一个class：WCRunner。

2. WCMapper
   主要说明：map 和 reduce 的数据输入输出都是以 key-value对的形式封装的.

package my.examples.hadoop.mr;

import java.io.IOException;

import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WCMapper extends Mapper<LongWritable, Text, Text, LongWritable>{   
    @Override
    protected void map(LongWritable key, Text value, Context context)
            throws IOException, InterruptedException {
/*
在Mapper<LongWritable, Text, Text, LongWritable>4个泛型中，前两个是指定mapper输入数据的类型，KEYIN是输入的key的类型，VALUEIN是输入的value的类型
*/
//map 和 reduce 的数据输入输出都是以 key-value对的形式封装的
/*
默认情况下，框架传递给我们的mapper的输入数据中，key是要处理的文本中一行的起始偏移量，这一行的内容作为value
*/      
//下面这一行代码是把读到的一行数据的内容转换成string类型
        String line = value.toString();
//下面这一行代码是读到的一行数据的转化好的string类型的文本按特定分隔符切分，功能是分割成一个个单词，保存到数组中        
        String[] words = StringUtils.split(line, " ");
//下面for循环遍历这个单词数组输出为kv形式  k：单词   v ： 1      
        for(String word : words){

            context.write(new Text(word), new LongWritable(1));

        }       
    }   
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

3.WCReducer

package my.examples.hadoop.mr;

import java.io.IOException;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WCReducer extends Reducer<Text, LongWritable, Text, LongWritable>{



    //框架在map处理完成之后，将所有kv对缓存起来，进行分组，然后传递一个组<key,valus{}>，调用一次reduce方法
    //例如<hello,{1,1,1,1,1,1.....}>
    @Override
    protected void reduce(Text key, Iterable<LongWritable> values,Context context)
            throws IOException, InterruptedException {

        long count = 0;
        //遍历value的list，进行累加求和
        for(LongWritable value:values){

            count += value.get();
        }

        //输出这一个单词的统计结果

        context.write(key, new LongWritable(count));

    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

4.WCRunner

package my.examples.hadoop.mr;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

/**
 * 用来描述一个特定的作业
 * 比如，该作业使用哪个类作为逻辑处理中的map，哪个作为reduce
 * 还可以指定该作业要处理的数据所在的路径
 * 还可以指定改作业输出的结果放到哪个路径
 * ....
 * @author duanhaitao@itcast.cn
 *
 */
public class WCRunner {

    public static void main(String[] args) throws Exception {

        Configuration conf = new Configuration();

        Job wcjob = Job.getInstance(conf);

        //设置整个job所用的那些类在哪个jar包
        wcjob.setJarByClass(WCRunner.class);


        //本job使用的mapper和reducer的类
        wcjob.setMapperClass(WCMapper.class);
        wcjob.setReducerClass(WCReducer.class);


        //指定输出数据kv类型（如果M和R的相同，不同的话，下面再写mapper的输出类型）
        wcjob.setOutputKeyClass(Text.class);
        wcjob.setOutputValueClass(LongWritable.class);

        //指定mapper的输出数据kv类型（如果和上面参数一样的话可以省略不写）
        wcjob.setMapOutputKeyClass(Text.class);
        wcjob.setMapOutputValueClass(LongWritable.class);


        //指定要处理的输入数据存放路径
        FileInputFormat.setInputPaths(wcjob, new Path("hdfs://192.168.126.100:9000/wc/srcdata/"));

        //指定处理结果的输出数据存放路径
        FileOutputFormat.setOutputPath(wcjob, new Path("hdfs://192.168.126.100:9000/wc/output3/"));

        //将job提交给集群运行 
        wcjob.waitForCompletion(true);  
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

5.打成Jar包后，创建数据的源文件，存放在输入文件夹下，在hadooop环境下执行，
命令是：
hadoop jar wc.jar my.examples.hadoop.mr.WCRunner
源文件中输入：
hello world
hello tom
hello jim
baby is my nvshen
运行过程
16/04/13 21:00:39 INFO client.RMProxy: Connecting to ResourceManager at weekend110/192.168.126.100:8032
16/04/13 21:00:40 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/04/13 21:00:40 INFO input.FileInputFormat: Total input paths to process : 1
16/04/13 21:00:40 INFO mapreduce.JobSubmitter: number of splits:1
16/04/13 21:00:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460606168153_0001
16/04/13 21:00:41 INFO impl.YarnClientImpl: Submitted application application_1460606168153_0001
16/04/13 21:00:41 INFO mapreduce.Job: The url to track the job: http://weekend110:8088/proxy/application_1460606168153_0001/
16/04/13 21:00:41 INFO mapreduce.Job: Running job: job_1460606168153_0001
16/04/13 21:00:50 INFO mapreduce.Job: Job job_1460606168153_0001 running in uber mode : false
16/04/13 21:00:50 INFO mapreduce.Job: map 0% reduce 0%
16/04/13 21:00:55 INFO mapreduce.Job: map 100% reduce 0%
16/04/13 21:01:00 INFO mapreduce.Job: map 100% reduce 100%
16/04/13 21:01:01 INFO mapreduce.Job: Job job_1460606168153_0001 completed successfully
16/04/13 21:01:01 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=156
FILE: Number of bytes written=186093
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=160
HDFS: Number of bytes written=54
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3037
Total time spent by all reduces in occupied slots (ms)=2342
Total time spent by all map tasks (ms)=3037
Total time spent by all reduce tasks (ms)=2342
Total vcore-seconds taken by all map tasks=3037
Total vcore-seconds taken by all reduce tasks=2342
Total megabyte-seconds taken by all map tasks=3109888
Total megabyte-seconds taken by all reduce tasks=2398208
Map-Reduce Framework
Map input records=5
Map output records=10
Map output bytes=130
Map output materialized bytes=156
Input split bytes=108
Combine input records=0
Combine output records=0
Reduce input groups=8
Reduce shuffle bytes=156
Reduce input records=10
Reduce output records=8
Spilled Records=20
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=139
CPU time spent (ms)=1310
Physical memory (bytes) snapshot=218542080
Virtual memory (bytes) snapshot=725782528
Total committed heap usage (bytes)=137433088
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=52
File Output Format Counters
Bytes Written=54
查看输出文件
命令：hadoop fs -cat /wc/output/part-r-00000
baby 1
hello 3
is 1
jim 1
my 1
nvshen 1
tom 1
world 1