当前位置:   article > 正文

mapreduce-多个文件的输入,实现每个单词的文件次数倒序排序_mapreduce多文件按顺序处理

mapreduce多文件按顺序处理

1、第一次mapreduce

  1. package cn.itcast.mr.combineSort2;
  2. import java.io.IOException;
  3. import org.apache.hadoop.conf.Configuration;
  4. import org.apache.hadoop.fs.FileSystem;
  5. import org.apache.hadoop.fs.Path;
  6. import org.apache.hadoop.io.IntWritable;
  7. import org.apache.hadoop.io.LongWritable;
  8. import org.apache.hadoop.io.Text;
  9. import org.apache.hadoop.mapreduce.Job;
  10. import org.apache.hadoop.mapreduce.Mapper;
  11. import org.apache.hadoop.mapreduce.Reducer;
  12. import org.apache.hadoop.mapreduce.Mapper.Context;
  13. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  14. import org.apache.hadoop.mapreduce.lib.input.FileSplit;
  15. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  16. public class combineSortMRD {
  17. static class combineSortMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
  18. @Override
  19. protected void map(LongWritable key, Text value,Context context)
  20. throws IOException, InterruptedException {
  21. String line = value.toString();
  22. String[] words = line.split(" ");
  23. //获取切片信息
  24. FileSplit inputSplit = (FileSplit) context.getInputSplit();
  25. String name = inputSplit.getPath().getName();
  26. for(String word:words){
  27. context.write(new Text(word+"-->"+name),new IntWritable(1));
  28. }
  29. }
  30. }
  31. static class combineSortReducer extends Reducer<Text, IntWritable, Text, IntWritable>{
  32. @Override
  33. protected void
声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/Li_阴宅/article/detail/781058
推荐阅读
相关标签
  

闽ICP备14008679号