devbox
人工智能uu
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

MapReduce--平均分，最高，低分以及及格率的计算

作者：人工智能uu | 2024-07-07 23:51:50

赞

踩

使用mapreduce实现求平均值,最高分,最低分

                    MapReduce--平均分，最高，低分以及及格率的计算

计算班级的平均分，以及个人的最高最低分，以及每个班级的及格率。

来先看一下我的数据。

  1. 时间			班级		姓名		科目			成绩
  2. 20180501 		1708a1 		li 		bishi 			80
  3. 20180501 		1708a1 		li 		jishi 			55
  4. 20180501 		1708a1 		li 		project 		90
  5. 20180501 		1708a1 		li2		bishi 			80
  6. 20180501 		1708a1 		li2		jishi 			20
  7. 20180501 		1708a1 		li2		project 		90
  8. 20180501 		1708a1 		li3		bishi 			50
  9. 20180501 		1708a1 		li3		jishi 			70
  10. 20180501 		1708a1 		li3		project 		60
  11. 20180501 		1708a1 		zhangsan 	bishi 			88
  12. 20180501 		1708a1 		zhangsan 	jishi 			55
  13. 20180501 		1708a1 		zhangsan 	project 		98
  14. 20180501 		1708a1 		lishi 		bishi 			18
  15. 20180501 		1708a1 		lishi 		jishi 			15
  16. 20180501 		1708a1 		lishi 		project 		15
  17. 20180501 		1708a1 		wangwu		bishi 			88
  18. 20180501 		1708a1 		wangwu		jishi 			76
  19. 20180501 		1708a1 		wangwu		project 		70
  20. 20180501 		1708a2 		li1 		bishi 			80
  21. 20180501 		1708a2 		li1 		jishi 			71
  22. 20180501 		1708a2 		li1 		project 		96
  23. 20180501 		1708a2 		li2 		bishi 			80
  24. 20180501 		1708a2 		li2 		jishi 			26
  25. 20180501 		1708a2 		li2 		project 		90
  26. 20180501 		1708a2 		li3 		bishi 			80
  27. 20180501 		1708a2 		li3 		jishi 			55
  28. 20180501 		1708a2 		li3 		project 		90
  29. 20180501 		1708a2 		zhangliang 	bishi 			81
  30. 20180501 		1708a2 		zhangliang 	jishi 			55
  31. 20180501 		1708a2 		zhangliang 	project 		98
  32. 20180501 		1708a2 		liuli 		bishi	 		70
  33. 20180501 		1708a2 		liuli 		jishi 			95
  34. 20180501 		1708a2 		liuli 		project 		75
  35. 20180501 		1708a2 		wangwu 		bishi 			80
  36. 20180501 		1708a2 		wangwu 		jishi 			76
  37. 20180501 		1708a2 		wangwu 		project 		70
  38. 20180501 		1708a2 		zhangxi 	bishi 			18
  39. 20180501 		1708a2 		zhangxi 	jishi 			16
  40. 20180501 		1708a2 		zhangxi 	project 		10

数据之间是空格。。。。

代码来了 -- 平均分，最高分，最低分

  1. package com.huhu.day01;
  2.  
  3. import java.io.IOException;
  4.  
  5. import org.apache.hadoop.conf.Configuration;
  6. import org.apache.hadoop.fs.Path;
  7. import org.apache.hadoop.io.LongWritable;
  8. import org.apache.hadoop.io.Text;
  9. import org.apache.hadoop.mapreduce.Job;
  10. import org.apache.hadoop.mapreduce.Mapper;
  11. import org.apache.hadoop.mapreduce.Reducer;
  12. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  13. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  14.  
  15. /**
  16.  * 切割文本: 平均分，最高低分
  17.  * 
  18.  * @author huhu_k
  19.  *
  20.  */
  21. public class HomeWork2 {
  22.  
  23. 	// map
  24. 	public static class MyMapper extends Mapper<LongWritable, Text, Text, Text> {
  25. 		Text keys = new Text();
  26. 		Text values = new Text();
  27.  
  28. 		@Override
  29. 		protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  30. 			// 数据切割方式(文本中的内容)
  31. 			// 按行分
  32. 			String[] line = value.toString().split(" ");
  33. 			keys.set(line[0] + ":" + line[2]);
  34. 			values.set(line[3] + ":" + line[4]);
  35. 			context.write(keys, values);
  36. 		}
  37. 	}
  38.  
  39. 	// reduce
  40. 	public static class MyReducer extends Reducer<Text, Text, Text, Text> {
  41.  
  42. 		@Override
  43. 		protected void reduce(Text key, Iterable<Text> value, Context context)
  44. 				throws IOException, InterruptedException {
  45. 			int max = Integer.MIN_VALUE;
  46. 			int min = Integer.MAX_VALUE;
  47. 			// 和
  48. 			int sum = 0;
  49. 			// 人数
  50. 			int count = 0;
  51. 			// 分数
  52. 			int score = 0;
  53. 			String classs = "";
  54. 			for (Text t : value) {
  55. 				classs = t.toString().split(":")[0];
  56. 				score = Integer.parseInt(t.toString().split(":")[1]);
  57. 				if (max < score)
  58. 					max = score;
  59. 				if (min > score)
  60. 					min = score;
  61. 				switch (classs) {
  62. 				case "bishi":
  63. 					score += score * 0.4;
  64. 					break;
  65. 				case "jishi":
  66. 					score += score * 0.3;
  67. 					break;
  68. 				case "project":
  69. 					score += score * 0.3;
  70. 					break;
  71. 				}
  72. 				sum += score;
  73. 				count++;
  74. 			}
  75. 			int avg = (int) sum / count;
  76. 			String[] student = key.toString().split(":");
  77. 			Text ky = new Text(student[0] + "\t" + student[1]);
  78. 			context.write(ky, new Text("平均分   " + avg));
  79. 			context.write(ky, new Text("最高值为   " + max));
  80. 			context.write(ky, new Text("最低值  " + min));
  81. 		}
  82.  
  83. 	}
  84.  
  85. 	public static void main(String[] args) throws Exception {
  86.  
  87. 		// 配置容器
  88. 		Configuration conf = new Configuration();
  89. 		// 创建一个job
  90. 		@SuppressWarnings("deprecation")
  91. 		Job job = new Job(conf, "MyMapReduce Two");
  92. 		// 配置job
  93. 		job.setJarByClass(HomeWork2.class);
  94. 		job.setMapperClass(MyMapper.class);
  95. 		job.setMapOutputKeyClass(Text.class);
  96. 		job.setMapOutputValueClass(Text.class);
  97.  
  98. 		job.setReducerClass(MyReducer.class);
  99. 		job.setOutputKeyClass(Text.class);
  100. 		job.setOutputValueClass(Text.class);
  101.  
  102. 		// 输入输出
  103. 		FileInputFormat.addInputPath(job, new Path(args[0]));
  104. 		FileOutputFormat.setOutputPath(job, new Path(args[1]));
  105.  
  106. 		// 执行程序
  107. 		boolean waitForCompletion = job.waitForCompletion(true);
  108. 		System.exit(waitForCompletion ? 0 : 1);
  109.  
  110. 	}
  111.  
  112. }

运行结果：




2.及格率

  1. package com.huhu.day01;
  2.  
  3. import java.io.IOException;
  4. import java.text.DecimalFormat;
  5. import java.util.HashMap;
  6. import java.util.Map;
  7.  
  8. import org.apache.hadoop.conf.Configuration;
  9. import org.apache.hadoop.fs.Path;
  10. import org.apache.hadoop.io.LongWritable;
  11. import org.apache.hadoop.io.Text;
  12. import org.apache.hadoop.mapreduce.Job;
  13. import org.apache.hadoop.mapreduce.Mapper;
  14. import org.apache.hadoop.mapreduce.Reducer;
  15. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  16. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  17.  
  18. /**
  19.  * 切割文本:及格率
  20.  * 
  21.  * @author huhu_k
  22.  *
  23.  */
  24. public class HomeWork3 {
  25.  
  26. 	// map
  27. 	public static class MyMapper extends Mapper<LongWritable, Text, Text, Text> {
  28. 		Text keys = new Text();
  29. 		Text values = new Text();
  30.  
  31. 		@Override
  32. 		protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  33. 			// 数据切割方式(文本中的内容)
  34. 			// 按行分
  35. 			String[] line = value.toString().split(" ");
  36. 			keys.set(line[0] + ":" + line[1]);
  37. 			context.write(keys, value);
  38. 		}
  39. 	}
  40.  
  41. 	// reduce
  42. 	public static class MyReducer extends Reducer<Text, Text, Text, Text> {
  43. 		Map<String, Double> map = new HashMap<>();
  44. 		Map<String, String> maps = new HashMap<>();
  45.  
  46. 		@Override
  47. 		protected void reduce(Text key, Iterable<Text> value, Context context)
  48. 				throws IOException, InterruptedException {
  49. 			for (Text t : value) {
  50. 				String[] values = t.toString().split(" ");
  51. 				String student = values[2] + ":" + values[0] + ":" + values[1];
  52. 				String subject = values[3];
  53. 				double score = Integer.valueOf(values[4]);
  54. 				if ("bishi".equals(subject)) {
  55. 					score *= 0.4;
  56. 				} else {
  57. 					score *= 0.3;
  58. 				}
  59. 				// 如果map中有学生，累加学生的没门课程的分数
  60. 				if (map.containsKey(student)) {
  61. 					double scores = map.get(student);
  62. 					scores += score;
  63. 					map.put(student, scores);
  64. 				} else {
  65. 					// 第一次进入时不包含，则直接添加
  66. 					map.put(student, score);
  67. 				}
  68. 			}
  69.  
  70. 			for (Map.Entry<String, Double> m : map.entrySet()) {
  71. 				String classname = m.getKey().split(":")[2];
  72. 				Double score = m.getValue();
  73. 				if (maps.containsKey(classname) && score >= 60) {
  74. 					String k = Integer.parseInt(maps.get(classname).split(":")[0]) + 1 + "";
  75. 					String v = Integer.parseInt(maps.get(classname).split(":")[1]) + 1 + "";
  76. 					maps.put(classname, k + ":" + v);
  77. 				} else if (maps.containsKey(classname) && score < 60) {
  78. 					String k = Integer.parseInt(maps.get(classname).split(":")[0]) + 1 + "";
  79. 					String v = Integer.parseInt(maps.get(classname).split(":")[1]) + "";
  80. 					maps.put(classname, k + ":" + v);
  81. 				} else if (!maps.containsKey(classname) && score < 60) {
  82. 					maps.put(classname, "1:0");
  83. 				} else if (!maps.containsKey(classname) && score >= 60) {
  84. 					maps.put(classname, "1:1");
  85. 				}
  86. 			}
  87.  
  88. 		}
  89.  
  90. 		@Override
  91. 		protected void cleanup(Reducer<Text, Text, Text, Text>.Context context)
  92. 				throws IOException, InterruptedException {
  93. 			for (Map.Entry<String, String> m : maps.entrySet()) {
  94. 				DecimalFormat d = new DecimalFormat("0.00%");
  95. 				double pass = Double.valueOf(m.getValue().split(":")[1]) / Double.valueOf(m.getValue().split(":")[0]);
  96. 				context.write(new Text(m.getKey()), new Text("及格率为:" + d.format(pass)));
  97. 			}
  98. 		}
  99. 	}
  100.  
  101. 	public static void main(String[] args) throws Exception {
  102.  
  103. 		// 配置容器
  104. 		Configuration conf = new Configuration();
  105. 		// 创建一个job
  106. 		@SuppressWarnings("deprecation")
  107. 		Job job = new Job(conf, "MyMapReduce Count");
  108. 		// 配置job
  109. 		job.setJarByClass(HomeWork3.class);
  110. 		job.setMapperClass(MyMapper.class);
  111. 		job.setMapOutputKeyClass(Text.class);
  112. 		job.setMapOutputValueClass(Text.class);
  113.  
  114. 		job.setReducerClass(MyReducer.class);
  115. 		job.setOutputKeyClass(Text.class);
  116. 		job.setOutputValueClass(Text.class);
  117.  
  118. 		// 输入输出
  119. 		FileInputFormat.addInputPath(job, new Path(args[0]));
  120. 		FileOutputFormat.setOutputPath(job, new Path(args[1]));
  121.  
  122. 		// 执行程序
  123. 		boolean waitForCompletion = job.waitForCompletion(true);
  124. 		System.exit(waitForCompletion ? 0 : 1);
  125.  
  126. 	}
  127.  
  128. }


   MapReduce一个分布式并行离线计算框架。我们只需要知道map()，reduce(),input,output，剩下的由框架完成



基于yarn的工作流程




转载于:https://www.cnblogs.com/meiLinYa/p/9231026.html

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/人工智能uu/article/detail/797083
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号