微博消息分析-大数据项目_nmax死亡

[实验数据]
本实验所用数据为新浪微博数据，包含了从2013年6月1日到14日期间的12,102,744条微博。数据集已经存放在HDFS上，路径为“/data/13/3/post/post.csv”，各字段以制表符分隔。数据集还存放在了Hive上，表名为“bigdata_cases.post”。

各字段的定义为：

字段	定义
PostId	微博标识符
UserId	用户标识符
UtcTime	微博发布的标准Unix时间
Text	微博正文
RepostsCount	微博转发数
CommentsCount	微博评论数
RepostPostId	转发微博的原微博标识符，若为0则为原创微博
1
2
3
4
5
6
7
8

数据集的前5行为：

student1@master:~$ hdfs dfs -cat /data/13/3/post/post.csv | head -5
48879661    1097414213  1369920667  【试问：强奸小学女生者，势力到底有多大?】@叶海燕宝贝 举牌劝阻校长带小学女生开房，竟然摊上了大事。中午，她在微博中说，先是有人指使房东停止租房给她，然后有11男女上门打她。@迟夙生律师 微博刚才称，叶海燕现在被拘在派出所，电话是：0775-8222414。请大家呼吁一下。   52249   7620    0
36548989    2686904145  1369989873  【妻子勒死女儿后举报副局长丈夫贪污】她是用一根数据线环住女儿脖子，将其勒死的。她的副局长丈夫通过虚开增值税发票赚取巨款，包养情妇，买房生女。在杀死女儿前，她曾多次“带着三十多张存折和大量现金”举报丈夫，却始终无果。甚至在这期间，她的丈夫还被评为了“优秀党员”。http:\\t.cn\zHa60Zx   3653    698 0
33166398    1728892794  1369966040  1943年5月24日，《解放日报》发表《谈延市二流子的改造》：延安市划定二流子110人，其中女二流子39人。二流子的门上和身上被强迫佩带有二流子的徽章标志，只有在真正参加生产之后才被准许摘去。对女二流子，规定她们受家人严格束缚，帮助丈夫整顿家务，如有不改，则丈夫打骂，政府不管，也不准离婚。   18  6   0
33166399    1104150515  1369898443  今天听到了一个广告主说，现在还在包位置的网络广告主就是“傻大黑粗”，媒体游说说自己的用户有多高端，基本都是忽悠人    36  36  0
33166400    2524610164  1369962904  即使生气，也会装作淡定；即使不开心，也会努力微笑；即使悲伤，也只是偷偷的；即使在乎，也不会解释太多，这就是现在的我。  178 4   0
同时，字段“微博正文”经中文分词后的数据集也已经存放在HDFS上，路径为“/data/13/3/post_segmented/post_segmented.csv”，各字段以制表符分隔，字段“微博正文”的分词结果中各词用逗号分隔。数据集还存放在了Hive上，表名为“bigdata_cases.post_segmented”。
1
2
3
4
5
6
7

[实验步骤提示]
在以下提示步骤中，步骤1是用IKAnalyzer做中文分词，步骤2用Hive做数据分析和数据准备，所有代码在大数据计算集群上执行，步骤3是用R语言做数据可视化。

1. 用IKAnalyzer做中文分词
   对微博内容做分词，采用IKAnalyzer中文分词包。

具体Java代码如下：

package lab3.module13;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.StringReader;
import java.util.HashSet;
import java.util.Set;

import org.wltea.analyzer.core.IKSegmenter;
import org.wltea.analyzer.core.Lexeme;

public class WordTokenize {
    public static void main(String[] args) throws Exception {
        BufferedReader swbr = new BufferedReader(new InputStreamReader(new FileInputStream("stopword.dic"), "UTF-8")); 
        Set<String> stopWordSet = new HashSet<String>();  
        String stopWord = null;  
        for(; (stopWord = swbr.readLine()) != null;){  
            stopWordSet.add(stopWord);  
        } 
        
        FileInputStream fis = new FileInputStream(args[0]);
        FileOutputStream fos = new FileOutputStream(args[1]);
        BufferedReader br =new BufferedReader(new InputStreamReader(fis, "UTF-8"));
        BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fos, "UTF-8"));
        String line;
        while((line = br.readLine()) != null) {
            String[] fields = line.split("\t");
            if (fields.length != 7)
                continue;
            StringReader sr=new StringReader(fields[3]); 
            IKSegmenter iks = new IKSegmenter(sr,true);
            Lexeme t;
            StringBuilder sb = new StringBuilder();
            while ((t = iks.next()) != null) {
                if (stopWordSet.contains(t.getLexemeText()))
                    continue;
                sb.append(t.getLexemeText()).append(",");
            }
            String text;
            if (sb.length() > 0)
                text = sb.toString().substring(0, sb.length() - 1);
            else
                text = "";
            bw.write(fields[0] + "\t" + fields[1] + "\t" + fields[2] + "\t" + text + "\t" + fields[4] + "\t" + fields[5] + "\t" + fields[6] + "\r\n");
        }
        br.close();
        bw.close();
    }

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

编译成wordtokenize.jar,需要依赖IKAnalyzer的包IKAnalyzer2012_FF.jar，该包路径为“D:\packages\IK-Analyzer-2012FF\dist\”。提交到大数据计算集群运行。

java -cp IKAnalyzer2012_FF.jar:wordtokenize.jar lab3.module13.WordTokenize data/13/3/post/post.csv data/13/3/post_segmented/post_segmented.csv
1

2. 用Hive做数据分析和数据准备
   a. 统计各关键词的热度变化指标、转发数和评论数
   统计各关键词的热度变化指标、转发数和评论数。热度变化指标衡量关键词在一定时间内出现次数的变化情况，定义为

    max(N)−min(N)max(N)+min(N)
   1

其中N表示关键词在单位时间（这里是每天）内出现的次数。该指标越大，说明关键词的热度变化越大，所描述的事物可能是当时的突发热点。

hive -e \
"select Word, min(CountWord) as MinWordCount, max(CountWord) as MaxWordCount, 
(max(CountWord) - min(CountWord)) / (max(CountWord) + min(CountWord)) as Ratio,
sum(CountRepost) / sum(CountWord) as AverageRepost, sum(CountComment) / sum(CountWord) as AverageComment
from (
select Date, Word, count(1) as CountWord, 
sum(RepostsCount) as CountRepost, sum(CommentsCount) as CountComment
from (
select from_unixtime(UtcTime,'yyyy-MM-dd') as Date, Word, RepostsCount, CommentsCount
from bigdata_cases.post_segmented lateral view explode(Text) TextTable as Word
where from_unixtime(UtcTime,'yyyy-MM-dd') between '2013-06-01' and '2013-06-14') a2
group by Date, Word) a1
group by Word
having max(CountWord) > 1000
order by Ratio desc;" \
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

1.csv
得到结果的前10行：

Word	MinWordCount	MaxWordCount	Ratio	AverageRepost	AverageComment
龙舟竞渡	1	2170	0.9990787655458314	5.863933711295246	4.20409943305713
47人	1	1050	0.9980970504281637	163.6648160999306	55.10548230395559
作文题目	2	2015	0.9980168567178979	264.4995592124596	59.42932706435498
加纳	2	1903	0.9979002624671915	127.36005726556908	62.13266523502744
刘志军	5	4737	0.9978911851539435	108.75326525765851	29.62651389218713
斯诺	2	1825	0.9978106185002736	76.90679859559528	25.65432492818385
初五	3	2564	0.9976626412154266	112.04645476772616	65.81540342298288
寄往	6	4691	0.9974451777730466	6.891304347826087	4.53416149068323
聘礼	2	1509	0.9973527465254798	141.0823565700185	45.99876619370759
ios7	4	2205	0.9963784517881394	43.398148148148145	15.984825102880658
1
2
3
4
5
6
7
8
9
10
11

b. 统计热度变化前五和后五的关键词每天的出现次数、转发数和评论数
统计热度变化前五和后五的关键词每天的出现次数、转发数和评论数。

hive -e \
	"select Date, Word, count(1) as CountWord, 
	sum(RepostsCount) as CountRepost, sum(CommentsCount) as CountComment
	from (
	select from_unixtime(UtcTime,'yyyy-MM-dd') as Date, Word, RepostsCount, CommentsCount
	from bigdata_cases.post_segmented lateral view explode(Text) TextTable as Word
	where from_unixtime(UtcTime,'yyyy-MM-dd') between '2013-06-01' and '2013-06-14') a
	where Word in ('龙舟竞渡','47人','作文题目','加纳','刘志军')
	group by Date, Word" \
1
2
3
4
5
6
7
8
9

2-1.csv

hive -e \
"select Date, Word, count(1) as CountWord, 
sum(RepostsCount) as CountRepost, sum(CommentsCount) as CountComment
from (
select from_unixtime(UtcTime,'yyyy-MM-dd') as Date, Word, RepostsCount, CommentsCount
from bigdata_cases.post_segmented lateral view explode(Text) TextTable as Word
where from_unixtime(UtcTime,'yyyy-MM-dd') between '2013-06-01' and '2013-06-14') a
where Word in ('恤','情侣','鞋','广州','宝贝')
group by Date, Word" \
1
2
3
4
5
6
7
8
9

2-2.csv
得到结果的前10行：

Date	Word	CountWord	CountRepost	CountComment
2013-06-01	作文题目	8	214	80
2013-06-09	47人	170	5747	2051
2013-06-04	47人	9	306	81
2013-06-10	47人	91	7039	5056
2013-06-08	加纳	191	3949	1886
2013-06-14	加纳	21	790	144
2013-06-03	加纳	4	175	43
2013-06-05	刘志军	6	12	5
2013-06-05	龙舟竞渡	1	459	130
2013-06-08	作文题目	635	169564	30690
1
2
3
4
5
6
7
8
9
10
11

2. 用R语言做数据可视化
   a. 载入相关程序包
   载入相关程序包。将Hive输出的结果文件复制到R语言可访问的路径如“D:\workspace\”。

    > library(ggplot2)
    > library(GGally)
    Warning: package 'GGally' was built under R version 3.2.3
    b. 画出热度变化指标、转发数和评论数的直方图
    画出热度变化指标、转发数和评论数的直方图，其中横坐标为热度变化指标、转发数和评论数，纵坐标为关键词的数量，其中转发数和评论数的直方图中纵坐标用平方根尺度表示，横坐标用对数尺度表示。
    
    > data1 <- read.table("D:/workspace/1.csv", sep = "\t", fileEncoding = "UTF-8")
    Warning in scan(file, what, nmax, sep, dec, quote, skip, nlines,
    na.strings, : 输入链结'D:/workspace/1.csv'内的输入不对
    > names(data1) <- c("Word", "MinWordCount", "MaxWordCount", "HotMetric", "AverageRepost", 
    +     "AverageComment")
    > ggplot(data1, aes(x = HotMetric)) + geom_histogram(aes(fill = ..count..)) + 
    +     scale_fill_gradient("Count", low = "green", high = "red")
    stat_bin: binwidth defaulted to range/30. Use 'binwidth = x' to adjust this.
    
    
    > ggplot(data1, aes(x = AverageRepost)) + geom_histogram(aes(fill = ..count..)) + 
    +     scale_fill_gradient("Count", trans = "sqrt", low = "green", high = "red") + 
    +     scale_x_log10() + scale_y_sqrt()
    stat_bin: binwidth defaulted to range/30. Use 'binwidth = x' to adjust this.
    
    
    > ggplot(data1, aes(x = AverageComment)) + geom_histogram(aes(fill = ..count..)) + 
    +     scale_fill_gradient("Count", trans = "sqrt", low = "green", high = "red") + 
    +     scale_x_log10() + scale_y_sqrt()
    stat_bin: binwidth defaulted to range/30. Use 'binwidth = x' to adjust this.
   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   16
   17
   18
   19
   20
   21
   22
   23
   24
   25
   26

c. 画出热度变化指标、转发数和评论数的散点图矩阵和相关性
画出热度变化指标、转发数和评论数的散点图矩阵和相关性，其中转发数和评论数采用对数坐标。可以看出，转发数和评论数高度相关，但与热度变化指标相关性较小。

> ggpairs(data.frame(HotMetric = data1[[4]], AverageRepost = log10(data1[[5]]), 
+     AverageComment = log10(data1[[6]])), columnLabels = c("HotMetric", "Log(Repost)", 
+     "Log(Comment)"))
1
2
3

d. 画出热度变化前五和后五的关键词每天的出现次数、转发数和评论数
画出热度变化前五和后五的关键词每天的出现次数、转发数和评论数，其中横坐标表示日期，纵坐标表示关键词的出现次数、转发数和评论数，线条粗细表示关键词的出现次数。

> data21 <- read.table("D:/workspace/2-1.csv", sep = "\t", fileEncoding = "UTF-8")
> names(data21) <- c("Date", "Word", "CountWord", "CountRepost", "CountComment")
> data21$Date <- as.POSIXct(data21$Date)
> ggplot(data21, aes(x = Date, y = CountWord, group = Word)) + geom_line(aes(colour = Word, 
+     size = CountWord))


> ggplot(data21, aes(x = Date, y = CountRepost, group = Word)) + geom_line(aes(colour = Word, 
+     size = CountWord))


> ggplot(data21, aes(x = Date, y = CountComment, group = Word)) + geom_line(aes(colour = Word, 
+     size = CountWord))


> data22 <- read.table("D:/workspace/2-2.csv", sep = "\t", fileEncoding = "UTF-8")
> names(data22) <- c("Date", "Word", "CountWord", "CountRepost", "CountComment")
> data22$Date <- as.POSIXct(data22$Date)
> ggplot(data22, aes(x = Date, y = CountWord, group = Word)) + geom_line(aes(colour = Word, 
+     size = CountWord))


> ggplot(data22, aes(x = Date, y = CountRepost, group = Word)) + geom_line(aes(colour = Word, 
+     size = CountWord))


> ggplot(data22, aes(x = Date, y = CountComment, group = Word)) + geom_line(aes(colour = Word, 
+     size = CountWord))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28