devbox
不正经
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

手机app日志分析系统（三）_usage_logs_v2啥用

作者：不正经 | 2024-03-24 08:21:50

赞

踩

usage_logs_v2啥用
  1. 一、创建hive分区表
  2. ----------------------------------------------------
  3.     1.创建数据库
  4.       $hive> create database applogsdb;
  5.  
  6.    2.创建分区表
  7.       编写脚本。
  8.       [applogs_create_table.sql]
  9.       use applogsdb;
  10.       --startup
  11.       CREATE external TABLE ext_startup_logs(createdAtMs bigint,appId string,tenantId string,deviceId string,appVersion string,appChannel string,appPlatform string,osType string,deviceStyle string,country string,province string,ipAddress string,network string,carrier string,brand string,screenSize string)PARTITIONED BY (ym string, day string,hm string) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' STORED AS TEXTFILE;
  12.  
  13.       --error
  14.       CREATE external TABLE ext_error_logs(createdAtMs bigint,appId string,tenantId string,deviceId string,appVersion string,appChannel string,appPlatform string,osType string,deviceStyle string,errorBrief string,errorDetail string)PARTITIONED BY (ym string, day string,hm string) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' STORED AS TEXTFILE;
  15.  
  16.       --event
  17.       CREATE external TABLE ext_event_logs(createdAtMs bigint,appId string,tenantId string,deviceId string,appVersion string,appChannel string,appPlatform string,osType string,deviceStyle string,eventId string,eventDurationSecs bigint,paramKeyValueMap Map<string,string>)PARTITIONED BY (ym string, day string,hm string) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' STORED AS TEXTFILE;
  18.  
  19.       --page
  20.       CREATE external TABLE ext_page_logs(createdAtMs bigint,appId string,tenantId string,deviceId string,appVersion string,appChannel string,appPlatform string,osType string,deviceStyle string,pageViewCntInSession int,pageId string,visitIndex int,nextPage string,stayDurationSecs bigint)PARTITIONED BY (ym string, day string,hm string) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' STORED AS TEXTFILE;
  21.  
  22.       --usage
  23.       CREATE external TABLE ext_usage_logs(createdAtMs bigint,appId string,tenantId string,deviceId string,appVersion string,appChannel string,appPlatform string,osType string,deviceStyle string,singleUseDurationSecs bigint,singleUploadTraffic bigint,singleDownloadTraffic bigint)PARTITIONED BY (ym string, day string,hm string) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' STORED AS TEXTFILE;
  24.  
  25.    3.执行applogs.sql脚本
  26.       $> hive -f /share/umeng/applogs_create_table.sql
  27.  
  28.  
  29. 二、使用Linux cron调度，周期性load HDFS的数据到hive的分区表
  30. ----------------------------------------------------------------
  31.     1.解释
  32.       调度就是周期运行指定的任务。
  33.  
  34.     2.Ubuntu安装Cron
  35.         apt-get install cron
  36.  
  37.     3.调度命令[Ubuntu]
  38.         $> /usr/sbin/service cron start
  39.         $> /usr/sbin/service cron status
  40.         $> /usr/sbin/service cron restart
  41.         $> /usr/sbin/service cron stop
  42.  
  43.    4.调度命令[centos]
  44.       //查看状态
  45.       $> service crond status
  46.       //停止
  47.       $>service crond stop
  48.       //启动
  49.       $> service crond start
  50.  
  51.    5.配置调度任务
  52.       a.[/etc/crontab下]
  53.             0-59|0-23 1-31 1-12 0-6
  54.              分   时  天   月   星期
  55.              *    *    *    *    *   ubuntu  source /etc/profile;echo `date` >> ~/1.log
  56.  
  57.             //五个* 表示通配，每分钟执行一次后面的命令 --> source /etc/profile;echo `date` >> ~/1.log
  58.             //Linux最小执行间隔为1分钟
  59.  
  60.     6.date操作
  61.         date -d "-3 minute" +%Y%m-%d-%H%M           //得到3分钟之前的时间
  62.         date -d "3 minute" +%Y%m-%d-%H%M           //得到3分钟之后的时间
  63.         date -d "3 hour" +%Y%m-%d-%H%M           //得到3分钟之后的时间
  64.  
  65.     7.使用sed命令编辑文件
  66.         //删除第一行
  67.         $>sed '1d' 1.log
  68.         //删除最后一行
  69.         $>sed '$d' 1.log
  70.         //删除区间行
  71.         $>sed '1,3d' 1.log
  72.         //删除所有行
  73.         $>sed '1,$d' 1.log
  74.  
  75.         //p:print -- 复制每一行，然后打印输出。也就是每一行打印两遍
  76.         $>sed '1,$p' 1.log
  77.  
  78.         //-n:安静模式，只显示处理的行 -- 只打印第一行
  79.         $>sed -n '1,$p' 1.log
  80.  
  81.         //-i:对源文件进行修改[1,$p]
  82.         $>sed -i '1,$p' 1.log
  83.  
  84.         //显示含有hello的行[/.../p]
  85.         $>sed -n '/hello/p' 1.log
  86.  
  87.         //追加内容第1行之后追加新行[1a]
  88.         $>sed -i '1ahello' 1.log
  89.  
  90.         //追加新行，指定前置字符 [1a]
  91.         $>sed -i '1a\ hello' 1.log
  92.  
  93.         //1-3行每行下面都追加新行hello[1,3a] --- append
  94.         $>sed -i '1,3ahello' 1.log
  95.  
  96.         //替换,针对整行[1,2c] -- cover
  97.         $>sed -i '1,2ckkk' 1.log
  98.  
  99.         //替换,针对特定字符串,用how替换掉hello [s/../../g]
  100.         $>sed -i 's/hello/how/g' 1.log
  101.  
  102.    8.编写脚本，周期性导入hdfs的文件到hive的分区表
  103.       [~/Downloads/.exportData.sql]
  104.         load data inpath '/data/applogs/startup/${ym}/${day}/${hm}' into table applogsdb.ext_startup_logs partition(ym='${ym}',day='${day}',hm='${hm}');
  105.         load data inpath '/data/applogs/error/${ym}/${day}/${hm}' into table applogsdb.ext_error_logs partition(ym='${ym}',day='${day}',hm='${hm}');
  106.         load data inpath '/data/applogs/event/${ym}/${day}/${hm}' into table applogsdb.ext_event_logs partition(ym='${ym}',day='${day}',hm='${hm}');
  107.         load data inpath '/data/applogs/page/${ym}/${day}/${hm}' into table applogsdb.ext_page_logs partition(ym='${ym}',day='${day}',hm='${hm}');
  108.         load data inpath '/data/applogs/usage/${ym}/${day}/${hm}' into table applogsdb.ext_usage_logs partition(ym='${ym}',day='${day}',hm='${hm}');
  109.  
  110.    9.编写执行脚本 -- 每次只拷贝1分钟时间片的数据。但是数据是3分钟前的那1分钟的数据。
  111.       [~/Downloads/exec.sh]
  112.         #!/bin/bash
  113.         systime=`date -d "-3 minute" +%Y%m-%d-%H%M`
  114.         ym=`echo ${systime} | awk -F '-' '{print $1}'`
  115.         day=`echo ${systime} | awk -F '-' '{print $2}'`
  116.         hm=`echo ${systime} | awk -F '-' '{print $3}'`
  117.  
  118.         cp ~/Downloads/.exportData.sql ~/Downloads/exportData.sql
  119.         sed -i 's/${ym}/'${ym}'/g' ~/Downloads/exportData.sql
  120.         sed -i 's/${day}/'${day}'/g' ~/Downloads/exportData.sql
  121.         sed -i 's/${hm}/'${hm}'/g' ~/Downloads/exportData.sql
  122.  
  123.         #执行hive的命令,注意此处的hive命令一定要写全路径，不然找不到hive
  124.         /soft/hive/bin/hive -f ~/Downloads/exportData.sql
  125.         rm ~/Downloads/exportData.sql
  126.  
  127.     10.设定每隔1分钟自动执行脚本exec.sh一次[生产环境一般为一天执行一次。每天的凌晨2点]
  128.         $> sudo nano /etc/crontab
  129.         *    *    *    *    *   ubuntu  source /etc/profile;~/Downloads/exec.sh
  130.  
  131.         //开启服务
  132.         $> /usr/sbin/service cron start
  133.         $> /usr/sbin/service cron status
  134.         $> /usr/sbin/service cron stop
  135.  
  136.  
  137. 三、导出web项目的war包，部署到ubuntu的tomcat上
  138. ---------------------------------------------------------------------
  139.    1.安装tomcat
  140.         a.下载安装
  141.             apache-tomcat-7.0.72.tar.gz
  142.  
  143.         b.tar开
  144.             tar -xzvf ~/Downloads/apache-tomcat-7.0.72.tar.gz -C /soft
  145.  
  146.         c.软连接
  147.             $>ln -s /soft/apache-tomcat-7.0.72 /soft/tomcat
  148.  
  149.     2.导出web项目的war包
  150.         a.找到web项目，在pom.xml中添加插件和common依赖
  151.             <?xml version="1.0" encoding="UTF-8"?>
  152.             <project xmlns="http://maven.apache.org/POM/4.0.0"
  153.                      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  154.                      xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  155.                 <modelVersion>4.0.0</modelVersion>
  156.  
  157.                 <groupId>com.test</groupId>
  158.                 <artifactId>app-logs-collect-web</artifactId>
  159.                 <version>1.0-SNAPSHOT</version>
  160.  
  161.                 <packaging>war</packaging>
  162.                 <build>
  163.                     <plugins>
  164.                         <plugin>
  165.                             <groupId>org.apache.maven.plugins</groupId>
  166.                             <artifactId>maven-surefire-plugin</artifactId>
  167.                             <version>2.12.4</version>
  168.                             <configuration>
  169.                                 <skipTests>true</skipTests>
  170.                             </configuration>
  171.                         </plugin>
  172.                         <plugin>
  173.                             <artifactId>maven-war-plugin</artifactId>
  174.                             <version>2.6</version>
  175.                             <configuration>
  176.                                 <warSourceDirectory>web</warSourceDirectory>
  177.                                 <failOnMissingWebXml>false</failOnMissingWebXml>
  178.                                 <excludes>css/*,images/*,js/*,png/*,phone/*</excludes>
  179.                             </configuration>
  180.                         </plugin>
  181.                     </plugins>
  182.                 </build>
  183.  
  184.                 <dependencies>
  185.  
  186.                     <dependency>
  187.                         <groupId>junit</groupId>
  188.                         <artifactId>junit</artifactId>
  189.                         <version>4.11</version>
  190.                     </dependency>
  191.                     <dependency>
  192.                         <groupId>com.fasterxml.jackson.core</groupId>
  193.                         <artifactId>jackson-core</artifactId>
  194.                         <version>2.8.8</version>
  195.                     </dependency>
  196.                     <dependency>
  197.                         <groupId>com.fasterxml.jackson.core</groupId>
  198.                         <artifactId>jackson-databind</artifactId>
  199.                         <version>2.8.3</version>
  200.                     </dependency>
  201.  
  202.                     <dependency>
  203.                         <groupId>com.maxmind.db</groupId>
  204.                         <artifactId>maxmind-db</artifactId>
  205.                         <version>1.0.0</version>
  206.                     </dependency>
  207.  
  208.                     <dependency>
  209.                         <groupId>org.springframework</groupId>
  210.                         <artifactId>spring-webmvc</artifactId>
  211.                         <version>4.3.5.RELEASE</version>
  212.                     </dependency>
  213.                     <dependency>
  214.                         <groupId>javax.servlet</groupId>
  215.                         <artifactId>servlet-api</artifactId>
  216.                         <version>2.5</version>
  217.                     </dependency>
  218.                     <dependency>
  219.                         <groupId>com.alibaba</groupId>
  220.                         <artifactId>fastjson</artifactId>
  221.                         <version>1.2.24</version>
  222.                     </dependency>
  223.                     <dependency>
  224.                         <groupId>com.maxmind.db</groupId>
  225.                         <artifactId>maxmind-db</artifactId>
  226.                         <version>1.0.0</version>
  227.                     </dependency>
  228.  
  229.                     <dependency>
  230.                         <groupId>org.apache.kafka</groupId>
  231.                         <artifactId>kafka_2.11</artifactId>
  232.                         <version>0.10.0.1</version>
  233.                     </dependency>
  234.                     <dependency>
  235.                         <groupId>com.test</groupId>
  236.                         <artifactId>app-analyze-common</artifactId>
  237.                         <version>1.0-SNAPSHOT</version>
  238.                     </dependency>
  239.  
  240.                 </dependencies>
  241.             </project>
  242.  
  243.         b.因为涉及到关联的公共模块common，所以，导出war包之前要先安装common模块，使common模块重新打包放到.m2仓库下。
  244.             maven --> install common module ...
  245.  
  246.         c.然后打包web服务器程序成war包app-web.war
  247.  
  248.     3.复制war文件到centos下${tomcat}/webapps
  249.  
  250.     4.启动tomcat
  251.         $>tomcat/bin/startup.sh
  252.  
  253.     5.验证
  254.         $>netstat -anop | grep 8080
  255.  
  256.     6.开启flume
  257.         flume-ng agent -f applog.conf -n a1
  258.  
  259.     7.修改手机程序连接服务器的地址。
  260.         UploadUtil.java
  261.         21行:URL url = new URL("http://s100:8080/app-web/coll/index");
  262.             http://s100:8080/app-web/coll/index
  263.  
  264.     8.至此，数据就收集并上传到hive上了
  265.  
  266.  
  267. 四、Hive查询
  268. -----------------------------------------------------------
  269.     1.通过hive查询指定app的用户数[去重]
  270.         hive> select count(distinct deviceid) from ext_startup_logs where appid = 'sdk34734';
  271.  
  272.  
  273.  
  274.  
  275.  
  276.  
  277.  
  278.  
  279.  
  280.  
  281.  
  282.  
  283.  
  284.  
  285.  
  286.  
  287.  
  288.  
  289.  
  290.  
  291.  
  292.  
  293.  
  294.  
  295.  
  296.
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/不正经/article/detail/301022
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号