devbox
很楠不爱3
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

用户画像_setscale(3, 4);

作者：很楠不爱3 | 2024-02-12 12:14:09

赞

踩

setscale(3, 4);
  1. 一、用户画像项目分析
  2. -------------------------------------------------------
  3.     1.概念
  4.         用户画像也叫用户信息标签化、客户信息。 根据用户的信息和行为动作，用一些标签把用户描绘出来，描绘的标签就是用户画像。
  5.  
  6.     2.项目简介
  7.         通过手机安装的app，判断机主的年龄区间和性别
  8.         通过对app设置性别和年龄段的相应权重结合用户每天使用app的时长和次数
  9.         //核心算法
  10.         /** 性别融合 */
  11.         public void protraitSex(double newdata1, double newdata2, long factor) {
  12.             double sum = (this.data1 + this.data2 + (newdata1 + newdata2) * factor);
  13.             if(sum != 0){
  14.                 this.data1 = (this.data1 + newdata1 * times) / sum;
  15.                 this.data2 = (this.data2 + newdata2 * times) / sum;
  16.             }
  17.         }
  18.  
  19.     3.几个概念
  20.         a.appID:用于标记APP，APP的唯一编号，判断终端安装的APP
  21.  
  22.         b.年龄段说明
  23.         名称 ==> 说明
  24.         年龄段1 ==> 24岁以下
  25.         年龄段2 ==> 25-30岁
  26.         年龄段3 ==> 31-35岁
  27.         年龄段4 ==> 36-40岁
  28.         年龄段5 ==> 40岁以上
  29.  
  30.         c.标签库
  31.            标签库的设计，主要是针对应用市场上主流的App，对App进行分类，App分类信息包括：App名称，男女权重信息，各个年龄段的权重信息。
  32.            AppID|App名称|男性权重|女性权重|年龄段1|年龄段2|年龄段3|年龄段4|年龄段5
  33.            10001|QQ|0.001|0.001|0|0.2|0.3|0.2|0.3
  34.            10002|飞信|0.001|0.001|0|0.2|0.3|0.2|0.3
  35.            10003|MSN|0.001|0.001|0|0.2|0.3|0.2|0.3
  36.            10004|阿里旺旺|0.001|0.001|0|0.2|0.3|0.2|0.3
  37.            10005|微信|0.001|0.001|0|0.2|0.3|0.2|0.3
  38.            10006|陌陌|0.001|0.001|0|0.2|0.3|0.2|0.3
  39.            10007|米聊|0.001|0.001|0|0.2|0.3|0.2|0.3
  40.            10008|啪啪|0.001|0.001|0|0.2|0.3|0.2|0.3
  41.            10009|飞聊|0.001|0.001|0|0.2|0.3|0.2|0.3
  42.            10010|来往|0.001|0.001|0|0.2|0.3|0.2|0.3
  43.            10011|连我|0.001|0.001|0|0.2|0.3|0.2|0.3
  44.            10012|有你|0.001|0.001|0|0.2|0.3|0.2|0.3
  45.  
  46.     4.画像数据
  47.         10011|连我|0.001|0.001|0|0.2|0.3|0.2|0.3
  48.         10012|有你|0.001|0.001|0|0.2|0.3|0.2|0.3
  49.         10013|Kakao Talk|0.001|0.001|0|0.2|0.3|0.2|0.3
  50.         10014|Whatsapp|0.001|0.001|0|0.2|0.3|0.2|0.3
  51.         10015|比邻|0.001|0.001|0|0.2|0.3|0.2|0.3
  52.         20016|新浪读书|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  53.         20017|潇湘书院|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  54.         20018|红袖添香|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  55.         20019|纵横中文网|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  56.         20020|掌上书院|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  57.         20021|和阅读|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  58.         20022|掌阅iReader|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  59.         20023|QQ阅读|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  60.         20024|百阅|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  61.         20025|塔读小说|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  62.         20026|Flipboard|0.001|0.001|0.1|0.3|0.3|0.2|0.1
  63.         ...
  64.         ...
  65.  
  66.     5.汇总周期
  67.         每天汇总一次，建议当天凌晨统计前一天数据
  68.  
  69.     6.注意事项
  70.         a.必备条件：性别比例、年龄比例必须在识别出AppID后进行，如果AppID编号为空，则不做处理。
  71.         b.性别和年龄的判断，都依赖于标签库的设计，也就是用户手机安装的App，根据手机App计算用户的性别和年龄权重信息，最后判断用户的年龄和性别。
  72.  
  73.  
  74. 二、开始项目
  75. -----------------------------------------------------
  76.     1.编写用户画像类
  77.         
  1. package userdraw;
  2.         import java.math.BigDecimal;
  3.  
  4.         /**
  5.          * 用户画像
  6.          **/
  7.         public class UserDraw {
  8.  
  9.             // 属性
  10.             private String statTimeDay;
  11.             private String MDN;       //手机号
  12.             private double male;   //男性
  13.             private double female; // 女性
  14.             private double age1;
  15.             private double age2;
  16.             private double age3;
  17.             private double age4;
  18.             private double age5;
  19.  
  20.  
  21.             // 重写toString
  22.             public String toString() {
  23.                 StringBuffer sb = new StringBuffer();
  24.                 sb.append(statTimeDay).append("|");
  25.                 sb.append(MDN).append("|");
  26.                 sb.append(new BigDecimal(male).setScale(3, 4).doubleValue()).append("|");
  27.                 sb.append(new BigDecimal(female).setScale(3, 4).doubleValue()).append("|");
  28.                 sb.append(new BigDecimal(age1).setScale(3, 4).doubleValue()).append("|");
  29.                 sb.append(new BigDecimal(age2).setScale(3, 4).doubleValue()).append("|");
  30.                 sb.append(new BigDecimal(age3).setScale(3, 4).doubleValue()).append("|");
  31.                 sb.append(new BigDecimal(age4).setScale(3, 4).doubleValue()).append("|");
  32.                 sb.append(new BigDecimal(age5).setScale(3, 4).doubleValue()).append("|");
  33.  
  34.                 return sb.toString();
  35.             }
  36.  
  37.             // 融合方法
  38.             /** 性别融合 */
  39.             public void protraitSex(double male2, double female2, long times) {
  40.                 double sum = (this.male + this.female + (male2 + female2) * times);
  41.                 if(sum != 0){
  42.                     this.male = (this.male + male2 * times) / sum;
  43.                     this.female = (this.female + female2 * times) / sum;
  44.                 }
  45.             }
  46.  
  47.             /** 年龄段融合 */
  48.             public void protraitAge(double pAge1, double pAge2, double pAge3, double pAge4, double pAge5, long times) {
  49.                 double sum = (age1 + age2 + age3 + age4 + age5 ) // 之前的APP的
  50.                         + (pAge1 + pAge2 + pAge3 + pAge4 + pAge5 ) * times;// 当前的APP的
  51.                 if(sum != 0){
  52.                     this.age1 = (pAge1 * times + age1) / sum;
  53.                     this.age2 = (pAge2 * times + age2) / sum;
  54.                     this.age3 = (pAge3 * times + age3) / sum;
  55.                     this.age4 = (pAge4 * times + age4) / sum;
  56.                     this.age5 = (pAge5 * times + age5) / sum;
  57.                 }
  58.             }
  59.  
  60.             /** 初始化男女概率 */
  61.             public void initSex(float male, float female) {
  62.                 float sum = male + female;
  63.                 if(sum != 0){
  64.                     this.male = male / sum;
  65.                     this.female = female / sum;
  66.                 }
  67.             }
  68.  
  69.             /** 初始化年龄段概率 */
  70.             public void initAge(float pAge1, float pAge2, float pAge3, float pAge4, float pAge5) {
  71.                 float sum = pAge1 + pAge2 + pAge3 + pAge4 + pAge5;
  72.                 if(sum != 0){
  73.                     this.age1 = pAge1 / sum;
  74.                     this.age2 = pAge2 / sum;
  75.                     this.age3 = pAge3 / sum;
  76.                     this.age4 = pAge4 / sum;
  77.                     this.age5 = pAge5 / sum;
  78.                 }
  79.             }
  80.  
  81.  
  82.             // setter and getter method
  83.             public String getStatTimeDay() {
  84.                 return statTimeDay;
  85.             }
  86.  
  87.             public void setStatTimeDay(String statTimeDay) {
  88.                 this.statTimeDay = statTimeDay;
  89.             }
  90.  
  91.  
  92.             public String getMDN() {
  93.                 return MDN;
  94.             }
  95.  
  96.             public void setMDN(String mDN) {
  97.                 MDN = mDN;
  98.             }
  99.  
  100.  
  101.             public double getMale() {
  102.                 return male;
  103.             }
  104.  
  105.             public double getFemale() {
  106.                 return female;
  107.             }
  108.  
  109.             public double getAge1() {
  110.                 return age1;
  111.             }
  112.  
  113.             public double getAge2() {
  114.                 return age2;
  115.             }
  116.  
  117.             public double getAge3() {
  118.                 return age3;
  119.             }
  120.  
  121.             public double getAge4() {
  122.                 return age4;
  123.             }
  124.  
  125.             public double getAge5() {
  126.                 return age5;
  127.             }
  128.         }
  1.     2.编写主入口函数
  2.         
  1. public static void main(String[] args) throws Exception {
  2.               Configuration conf = new Configuration();
  3.               Job job1 = Job.getInstance(conf, "UserDrawMapReduceJob1");
  4.               job1.setJarByClass(UserDrawMapReduce.class);
  5.  
  6.               job1.setMapperClass(MyMap.class);
  7.               job1.setReducerClass(MyReduce.class);
  8.  
  9.               job1.setMapOutputKeyClass(Text.class);
  10.               job1.setMapOutputValueClass(TextArrayWritable.class);
  11.               job1.setOutputKeyClass(Text.class);
  12.               job1.setOutputValueClass(Text.class);
  13.  
  14.               job1.setInputFormatClass(TextInputFormat.class);
  15.               job1.setOutputFormatClass(TextOutputFormat.class);
  16.  
  17.               FileInputFormat.addInputPath(job1, new Path("file:///D:\\share\\project\\userdraw\\data"));// 输入路径
  18.               FileOutputFormat.setOutputPath(job1, new Path("file:///D:\\share\\project\\userdraw\\out"));// 输出路径
  19.  
  20.               Boolean state1 = job1.waitForCompletion(true);
  21.               System.out.println("job1执行成功！！！");
  22.                  if (state1) {
  23.                     conf = new Configuration();
  24.                     Job job2 = Job.getInstance(conf, "UserDrawMapReduceJob2");
  25.                     job2.setJarByClass(UserDrawMapReduce.class);
  26.  
  27.                     job2.setMapperClass(MyMap2.class);
  28.                     job2.setReducerClass(MyReduce2.class);
  29.  
  30.                     job2.setMapOutputKeyClass(Text.class);
  31.                     job2.setMapOutputValueClass(Text.class);
  32.                     job2.setOutputKeyClass(Text.class);
  33.                     job2.setOutputValueClass(Text.class);
  34.  
  35.                     job2.setInputFormatClass(TextInputFormat.class);
  36.                     job2.setOutputFormatClass(TextOutputFormat.class);
  37.  
  38.                     FileInputFormat.addInputPath(job2, new Path("file:///D:\\share\\project\\userdraw\\out"));// 输入路径
  39.                     FileOutputFormat.setOutputPath(job2, new Path("file:///D:\\share\\project\\userdraw\\out2"));// 输出路径
  40.  
  41.                     Boolean state2 = job2.waitForCompletion(true);
  42.                     System.out.println("job2执行成功！！！");
  43.                     if (state2) {
  44.                        conf = new Configuration();
  45.                        // 设置zookeeper
  46.                        conf.set(UserDrawMapReduce.conf.consite, UserDrawMapReduce.conf.hbaseip);
  47.                        // 设置hbase表名称
  48.                        conf.set(TableOutputFormat.OUTPUT_TABLE, UserDrawMapReduce.conf.tableDraw);
  49.                        // 将该值改大，防止hbase超时退出
  50.                        conf.set(UserDrawMapReduce.conf.coftime, UserDrawMapReduce.conf.time);
  51.                        Job job3 = Job.getInstance(conf,
  52.                              "UserDrawPutInHbase");
  53.                        job3.setJarByClass(UserDrawMapReduce.class);
  54.                        TableMapReduceUtil.addDependencyJars(job3);
  55.  
  56.                        FileInputFormat.setInputPaths(job3, new Path("file:///D:\\share\\project\\userdraw\\out2"));
  57.  
  58.                        job3.setMapperClass(UserDrawPutInHbaseMap.class);
  59.                        job3.setMapOutputKeyClass(Text.class);
  60.                        job3.setMapOutputValueClass(Text.class);
  61.  
  62.                        job3.setReducerClass(UserDrawPutInHbaseReduce.class);
  63.                        job3.setOutputFormatClass(TableOutputFormat.class);
  64.  
  65.                        job3.waitForCompletion(true);
  66.                     }
  67.               }
  68.            }
  1.     3.编写MR作业类
  2.         a.第一次MR作业
  3.           
  1. public class UserDrawMapReduce {
  2.            public static Config conf = new Config();
  3.  
  4.            public static class MyMap extends Mapper<LongWritable, Text, Text, TextArrayWritable> {
  5.               Text k = new Text();
  6.  
  7.               public void map(LongWritable key, Text value, Context context)
  8.                     throws IOException, InterruptedException {
  9.                  //一行文本
  10.                  String line = value.toString();
  11.                  //通过 | 进行切割
  12.                  String[] dataArray = line.split(conf.Separator);
  13.                  //唯一标识：手机号+appid
  14.                  String uiqkey = dataArray[Integer.parseInt(conf.MDN)]
  15.                           + dataArray[Integer.parseInt(conf.appID)]; // MDN + appID
  16.                  String[] val = new String[5];
  17.                  //时间
  18.                  String timenow = dataArray[Integer.parseInt(conf.Date)];
  19.                  SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd");
  20.                  val[0] = sdf.format(Long.parseLong(timenow));//时间
  21.                  val[1] = dataArray[Integer.parseInt(conf.MDN)];// 手机号
  22.                  val[2] = dataArray[Integer.parseInt(conf.appID)];// appID
  23.                  val[3] = "1";// 计数
  24.                  val[4] = dataArray[Integer.parseInt(conf.ProcedureTime)];// 使用时长
  25.                  k.set(uiqkey);
  26.                  context.write(k, new TextArrayWritable(val));
  27.  
  28.               }
  29.            }
  30.  
  31.            /**
  32.             * 统计 每个软件使用的总次数和总时间，按照unikey进行聚合
  33.             */
  34.            public static class MyReduce extends Reducer<Text, TextArrayWritable, Text, Text> {
  35.               Text v = new Text();
  36.               public void reduce(Text key, Iterable<TextArrayWritable> values,
  37.                     Context context) throws IOException, InterruptedException {
  38.                  long sum = 0;
  39.                  int count = 0;
  40.                  String[] res = new String[5];
  41.                  boolean flg = true;
  42.                  for (TextArrayWritable t : values) {
  43.                     String[] vals = t.toStrings();
  44.                     if (flg) {
  45.                        res = vals;
  46.                     }
  47.                     if (vals[3] != null) {
  48.                        count = count + 1;
  49.  
  50.                     }
  51.                     if (vals[4] != null) {
  52.                        sum += Long.valueOf(vals[4]);
  53.                     }
  54.                  }
  55.                  res[3] = String.valueOf(count);
  56.                  res[4] = String.valueOf(sum);
  57.  
  58.                  StringBuffer sb = new StringBuffer();
  59.                  sb.append(res[0]).append("|");// 时间
  60.                  sb.append(res[1]).append("|");// 手机号
  61.                  sb.append(res[2]).append("|");// appID
  62.                  sb.append(res[3]).append("|");// 计数
  63.                  sb.append(res[4]);// 使用时长
  64.                  v.set(sb.toString());
  65.                  context.write(null, v);
  66.               }
  67.            }
  68.           }
  1.         b.第二次MR作业
  2.            
  1. package userdrawmr;
  2.  
  3.            import java.io.IOException;
  4.            import java.util.HashMap;
  5.            import java.util.Map;
  6.            import java.util.Set;
  7.            import org.apache.hadoop.io.LongWritable;
  8.            import org.apache.hadoop.io.Text;
  9.            import org.apache.hadoop.mapreduce.Mapper;
  10.            import org.apache.hadoop.mapreduce.Reducer;
  11.  
  12.            import userdraw.UserDraw;
  13.            import util.LoadHdfsTable;
  14.  
  15.            public class UserDrawMapReduce2 {
  16.  
  17.                public static class MyMap2 extends Mapper<LongWritable, Text, Text, Text> {
  18.                   Text k = new Text();
  19.  
  20.                   //更改key为手机号
  21.                   public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  22.                      String line = value.toString();
  23.                      String[] dataArray = line.split("\\|");
  24.                      String newkey = dataArray[1] ; // MDN手机号
  25.                      k.set(newkey);
  26.  
  27.                      context.write(k, value);
  28.                   }
  29.                }
  30.  
  31.                public static class MyReduce2 extends Reducer<Text, Text, Text, Text> {
  32.                   Map<String, String[]> appMap = LoadHdfsTable.getAppMap();
  33.                   Text v = new Text();
  34.                   public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
  35.                      Map<String, UserDraw> userDrawMap = new HashMap<String, UserDraw>();
  36.                      Set<String> keySet = userDrawMap.keySet();
  37.                      String keyMDN = null;
  38.                      for (Text t : values) {
  39.  
  40.                         String[] dataArray = t.toString().split("\\|");
  41.                          keyMDN = dataArray[1]; // 用户MDN
  42.                         String appID = dataArray[2]; // APPID
  43.                         // 根据appID获取对应的标签信息
  44.                         if (appID.length() > 0) { // appID不能为空
  45.                            if (appMap.get(appID) == null) {
  46.                               continue;
  47.                            }
  48.                            String favourite = appMap.get(appID)[2];
  49.                            float male = Float.parseFloat(appMap.get(appID)[1]);
  50.                            float female = Float.parseFloat(appMap.get(appID)[2]);
  51.                            float age1 = Float.parseFloat(appMap.get(appID)[3]);
  52.                            float age2 = Float.parseFloat(appMap.get(appID)[4]);
  53.                            float age3 = Float.parseFloat(appMap.get(appID)[5]);
  54.                            float age4 = Float.parseFloat(appMap.get(appID)[6]);
  55.                            float age5 = Float.parseFloat(appMap.get(appID)[7]);
  56.  
  57.                            long times = Long.parseLong(dataArray[4]);
  58.                            if (userDrawMap.containsKey(keyMDN)==true) {
  59.                               UserDraw userDraw = userDrawMap.get(keyMDN);
  60.                               // 性别权重
  61.                               userDraw.protraitSex(male, female, times);
  62.                               // 年龄段权重
  63.                               userDraw.protraitAge(age1, age2, age3, age4, age5, times);
  64.  
  65.                            } else {
  66.                               userDrawMap.put(keyMDN, createDrawData(dataArray, favourite, male, female, age1, age2, age3, age4, age5, times));
  67.                            }
  68.                         }
  69.                      }
  70.                      for (String keys : keySet) {
  71.                         v.set(userDrawMap.get(keys).toString());
  72.                         context.write(null, v);
  73.  
  74.                      }
  75.                   }
  76.                }
  77.  
  78.  
  79.                // 创建画像数据
  80.                private static  UserDraw createDrawData(String[] dataArray, //
  81.                      String favourite, //兴趣爱好
  82.                      float male, float female, //性别
  83.                      float age1, float age2, float age3, float age4, float age5, //年龄
  84.                      long times) {
  85.  
  86.                   UserDraw userDraw = new UserDraw();
  87.                   userDraw.setStatTimeDay(dataArray[0]);
  88.                   userDraw.setMDN(dataArray[1]);
  89.  
  90.  
  91.                   // 初始化
  92.                   userDraw.initAge(age1, age2, age3, age4, age5);
  93.                   userDraw.initSex(male, female);
  94.  
  95.                   return userDraw;
  96.                }
  97.            }
  1.         c.第三次MR作业，将数据写入到Hbase
  2.            
  1.  package userdrawputinhbase;
  2.  
  3.             import java.io.IOException;
  4.  
  5.             import org.apache.hadoop.io.LongWritable;
  6.             import org.apache.hadoop.io.Text;
  7.             import org.apache.hadoop.mapreduce.Mapper;
  8.  
  9.  
  10.             public class UserDrawPutInHbaseMap extends Mapper<LongWritable, Text, Text, Text>{
  11.                Text k2 = new Text();
  12.                Text v2 = new Text();
  13.  
  14.                @Override
  15.                protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, Text>.Context context)
  16.                      throws IOException, InterruptedException {
  17.                   String line = value.toString();
  18.                   String[] splited = line.split("\\|");
  19.  
  20.  
  21.                      k2.set(splited[1]);
  22.                      v2.set(line);
  23.  
  24.                   context.write(k2, v2);
  25.                }
  26.             }
  1.             --------------------------------------------------------------------
  2.  
  3.            
  1.  package userdrawputinhbase;
  2.  
  3.             import java.io.IOException;
  4.  
  5.             import org.apache.hadoop.hbase.client.Durability;
  6.             import org.apache.hadoop.hbase.client.Mutation;
  7.             import org.apache.hadoop.hbase.client.Put;
  8.             import org.apache.hadoop.hbase.mapreduce.TableReducer;
  9.             import org.apache.hadoop.hbase.util.Bytes;
  10.             import org.apache.hadoop.io.NullWritable;
  11.             import org.apache.hadoop.io.Text;
  12.             import org.apache.hadoop.mapreduce.Reducer;
  13.  
  14.             public class UserDrawPutInHbaseReduce extends TableReducer<Text, Text, NullWritable>{
  15.  
  16.                 @SuppressWarnings("deprecation")
  17.                 @Override
  18.                 protected void reduce(Text k2, Iterable<Text> val,
  19.                         Reducer<Text, Text, NullWritable,Mutation>.Context context)
  20.                         throws IOException, InterruptedException {
  21.                     for (Text v2 : val) {
  22.                         String[] splited = v2.toString().split("\\|");
  23.                         //rowkey
  24.                         if(k2.toString().length()!=0){
  25.                             Put put = new Put(Bytes.toBytes(k2.toString()));
  26.  
  27.                             //跳过写入Hlog，提高写入速度
  28.                             put.setDurability(Durability.SKIP_WAL);
  29.                             put.add(Bytes.toBytes("draw"), Bytes.toBytes("mdn"), Bytes.toBytes(splited[1]));
  30.                             put.add(Bytes.toBytes("draw"), Bytes.toBytes("male"), Bytes.toBytes(splited[2]));
  31.                             put.add(Bytes.toBytes("draw"), Bytes.toBytes("female"), Bytes.toBytes(splited[3]));
  32.                             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age1"), Bytes.toBytes(splited[4]));
  33.                             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age2"), Bytes.toBytes(splited[5]));
  34.                             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age3"), Bytes.toBytes(splited[6]));
  35.                             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age4"), Bytes.toBytes(splited[7]));
  36.                             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age5"), Bytes.toBytes(splited[8]));
  37.                             context.write(NullWritable.get(), put);
  38.                         }
  39.  
  40.                     }
  41.  
  42.                 }
  43.             }
  1.     4.工具类
  2.         a.Config.class
  3.             
  1. package util;
  2.  
  3.             import java.io.IOException;
  4.             import java.io.InputStream;
  5.             import java.util.Properties;
  6.  
  7.             public class Config {
  8.                 static Properties properties;
  9.                 static{
  10.                     properties = new Properties();
  11.                     //可以获取资源文件
  12.                     InputStream inStream = Thread.currentThread().getContextClassLoader().getResourceAsStream("UserDraw.properties");
  13.                     try {
  14.                         properties.load(inStream);
  15.                     } catch (IOException e) {
  16.                         e.printStackTrace();
  17.                     }
  18.                 }
  19.  
  20.                 //UserDraw
  21.                 public String Separator = properties.getProperty("Separator");
  22.                 public String Date = properties.getProperty("Date");
  23.                 public String MDN = properties.getProperty("MDN");
  24.                 public String appID = properties.getProperty("appID");
  25.                 public String count = properties.getProperty("count");
  26.                 public String ProcedureTime = properties.getProperty("ProcedureTime");
  27.                 //Hbase
  28.                 public String consite = properties.getProperty("consite");
  29.                 public String hbaseip = properties.getProperty("hbaseip");
  30.                 public String coftime = properties.getProperty("coftime");
  31.                 public String time = properties.getProperty("time");
  32.                 public String tableDraw = properties.getProperty("tableDraw");
  33.             }
  1.         b.LoadHdfsTable.class
  2.            
  1.  package util;
  2.  
  3.             import java.io.BufferedReader;
  4.             import java.util.HashMap;
  5.             import java.util.Map;
  6.  
  7.             public class LoadHdfsTable {
  8.                public static String appTab = "/home/appTab.txt";
  9.  
  10.                private static Map<String, String[]> appMap = new HashMap<String, String[]>();
  11.                static {
  12.                   try {
  13.                      StringBuffer sb = new StringBuffer();
  14.                      String line = null;
  15.                      BufferedReader breader = ReadHdfsFile.fileReader(appTab);
  16.                      while ((line = breader.readLine()) != null) {
  17.                         String[] appArray = line.split("\\|");
  18.                         // appName
  19.                         sb.append(appArray[1]).append(",");
  20.                         // 性别权重
  21.                         sb.append(appArray[2]).append(",").append(appArray[3]).append(",");
  22.                         // 年龄段权重
  23.                         sb.append(appArray[4]).append(",").append(appArray[5]).append(",").append(appArray[6]).append(",");
  24.                         sb.append(appArray[7]).append(",").append(appArray[8]);
  25.  
  26.                         String[] appToValueArray = sb.toString().split(",");
  27.                         appMap.put(appArray[0], appToValueArray);
  28.                         sb.delete(0, sb.length());
  29.  
  30.                      }
  31.                   } catch (Exception e) {
  32.                      e.printStackTrace();
  33.                   }
  34.                }
  35.  
  36.                public static Map<String, String[]> getAppMap() {
  37.                   return appMap;
  38.                }
  39.  
  40.                public static void setAppMap(Map<String, String[]> appMap) {
  41.                   LoadHdfsTable.appMap = appMap;
  42.                }
  43.  
  44.             }
  1.         c.ReadHdfsFile.classs
  2.              
  1. package util;
  2.  
  3.              import java.io.BufferedReader;
  4.              import java.io.FileReader;
  5.              import java.io.InputStreamReader;
  6.  
  7.              import org.apache.hadoop.conf.Configuration;
  8.              import org.apache.hadoop.fs.FSDataInputStream;
  9.              import org.apache.hadoop.fs.FileSystem;
  10.              import org.apache.hadoop.fs.Path;
  11.  
  12.              /** 读取文件 **/
  13.              public class ReadHdfsFile {
  14.                public static BufferedReader fileReader(String fileName) throws Exception {
  15.              //       Configuration conf = new Configuration();
  16.              //       FileSystem fs = FileSystem.get(conf);
  17.              //       FSDataInputStream in = fs.open(new Path(fileName));
  18.              //       BufferedReader br = new BufferedReader(new InputStreamReader(in));
  19.                   FileReader fr = new FileReader("D:\\share\\project\\userdraw\\appTab.txt");
  20.                   return new BufferedReader(fr);
  21.                }
  22.              }
  1.         d.TextArrayWritable类
  2.            
  1.  package util;
  2.  
  3.             import java.util.ArrayList;
  4.  
  5.             import org.apache.hadoop.io.ArrayWritable;
  6.             import org.apache.hadoop.io.Text;
  7.  
  8.             public class TextArrayWritable extends ArrayWritable {
  9.                 public TextArrayWritable() {
  10.                     super(Text.class);
  11.                 }
  12.  
  13.                 public TextArrayWritable(String[] strings) {
  14.                     super(Text.class);
  15.                     Text[] texts = new Text[strings.length];
  16.                     for (int i = 0; i < strings.length; i++) {
  17.                         texts[i] = new Text(strings[i]);
  18.                     }
  19.                     set(texts);
  20.                 }
  21.  
  22.                 public TextArrayWritable(ArrayList<String> strings) {
  23.                     super(Text.class);
  24.  
  25.                     Text[] texts = new Text[strings.size()];
  26.                     int i = 0;
  27.                     for (String str : strings) {
  28.                         texts[i] = new Text(str);
  29.                         i++;
  30.                     }
  31.                     set(texts);
  32.                 }
  33.  
  34.                 public ArrayList<String> toArrayList(String[] writables) {
  35.                     ArrayList<String> arraylist = new ArrayList<String>();
  36.                     for (String writable : writables) {
  37.                         arraylist.add(writable.toString());
  38.                     }
  39.                     return arraylist;
  40.                 }
  41.  
  42.                 public ArrayList<String> toArrayList() {
  43.                     return toArrayList(super.toStrings());
  44.                 }
  45.  
  46.             }
  1. 三、资源文件
  2. -------------------------------------
  3.     1.Resources
  4.         [UserDraw.properties]
  5.         #用户画像配置文件：
  6.         ################################
  7.         #字段分隔符号
  8.         Separator=\\|
  9.         #日期
  10.         Date=11
  11.         #手机号
  12.         MDN=0
  13.         #appID
  14.         appID=15
  15.         #计数
  16.         count=1
  17.         #使用时长
  18.         ProcedureTime=12
  19.         #Hbase数据库配置文件：
  20.         consite=hbase.zookeeper.quorum
  21.         hbaseip=192.168.0.4,192.168.0.5,192.168.0.6
  22.         coftime=dfs.socket.timeout
  23.         time=180000
  24.         tableDraw=user_draw
  25.  
  26.     2.pom.xml依赖
  27.         <?xml version="1.0" encoding="UTF-8"?>
  28.         <project xmlns="http://maven.apache.org/POM/4.0.0"
  29.                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  30.                  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  31.             <modelVersion>4.0.0</modelVersion>
  32.  
  33.             <groupId>groupId</groupId>
  34.             <artifactId>UserDraw</artifactId>
  35.             <version>1.0-SNAPSHOT</version>
  36.  
  37.             <dependencies>
  38.                 <dependency>
  39.                     <groupId>commons-codec</groupId>
  40.                     <artifactId>commons-codec</artifactId>
  41.                     <version>1.4</version>
  42.                 </dependency>
  43.                 <dependency>
  44.                     <groupId>org.apache.hbase</groupId>
  45.                     <artifactId>hbase-client</artifactId>
  46.                     <version>1.2.4</version>
  47.                 </dependency>
  48.                 <dependency>
  49.                     <groupId>org.apache.hbase</groupId>
  50.                     <artifactId>hbase-server</artifactId>
  51.                     <version>1.2.4</version>
  52.                 </dependency>
  53.  
  54.             </dependencies>
  55.         </project>
  56.  
  57.  
  58. 四、其他
  59. ------------------------------------------------
  60.     1.创建hbase表：
  61.       create 'user_draw',{NAME=>'draw',VERSIONS=>1,BLOCKCACHE=>true,BLOOMFILTER=>'ROW',COMPRESSION=>'SNAPPY'},
  62.       {SPLITS => ['/','+','0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w',
  63.       'x','y','z','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z']}
  64.  
  65.     2.入库hbase代码：
  66.        
  67.  
  68.  
  69.  
  70.  
  1. for (Text v2 : val) {
  2.          String[] splited = v2.toString().split("\\|");
  3.          //rowkey
  4.          if(k2.toString().length()!=0){
  5.             Put put = new Put(Bytes.toBytes(k2.toString()));
  6.  
  7.             //跳过写入Hlog，提高写入速度
  8.             put.setDurability(Durability.SKIP_WAL);
  9.             put.add(Bytes.toBytes("draw"), Bytes.toBytes("mdn"), Bytes.toBytes(splited[1]));
  10.             put.add(Bytes.toBytes("draw"), Bytes.toBytes("male"), Bytes.toBytes(splited[2]));
  11.             put.add(Bytes.toBytes("draw"), Bytes.toBytes("female"), Bytes.toBytes(splited[3]));
  12.             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age1"), Bytes.toBytes(splited[4]));
  13.             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age2"), Bytes.toBytes(splited[5]));
  14.             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age3"), Bytes.toBytes(splited[6]));
  15.             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age4"), Bytes.toBytes(splited[7]));
  16.             put.add(Bytes.toBytes("draw"), Bytes.toBytes("age5"), Bytes.toBytes(splited[8]));
  17.             context.write(NullWritable.get(), put);
  18.          }
  19.  
  20.       }

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/很楠不爱3/article/detail/77752
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号