devbox
小小林熬夜学编程
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

【hive 函数】Hive分析函数和窗口函数_hive中有没有first函数

作者：小小林熬夜学编程 | 2024-06-01 07:08:52

赞

踩

hive中有没有first函数

拿一个例子来说
数据集：

cookie1,2015-04-10 10:00:02,url2  
cookie1,2015-04-10 10:00:00,url1  
cookie1,2015-04-10 10:03:04,1url3  
cookie1,2015-04-10 10:50:05,url6  
cookie1,2015-04-10 11:00:00,url7  
cookie1,2015-04-10 10:10:00,url4  
cookie1,2015-04-10 10:50:01,url5  
cookie2,2015-04-10 10:00:02,url22  
cookie2,2015-04-10 10:00:00,url11  
cookie2,2015-04-10 10:03:04,1url33  
cookie2,2015-04-10 10:50:05,url66  
cookie2,2015-04-10 11:00:00,url77  
cookie2,2015-04-10 10:10:00,url44  
cookie2,2015-04-10 10:50:01,url55  
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14

窗口函数

  1. FIRST_VALUE 取出分组内排序后，截止到当前行，第一个值
  2. LAST_VALUE 取出分组内排序后，截止到当前行，最后一个值
  3. LEAD(col, n,DEFAULT) 用于统计窗口内往下第n行的值。第一个参数为列名，第二个参数为往下第n行(可选，默认为1)，第三个参数为默认值(当往下第n行为NULL时，取默认值)
  4. LAG(col,n,DEFAULT) 与lead相反，用于统计窗口内往下第n个值。第一个参数为列名，第二个参数为往上第n行(可选，默认为1)

FIRST_VALUE
取分组内排序后，截止到当前行，第一个值

SELECT cookieid,  
createtime,  
url,  
ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,  
FIRST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS first1  
FROM lxw1234;  

cookieid  createtime            url     rn      first1  
---------------------------------------------------------  
cookie1 2015-04-10 10:00:00     url1    1       url1  
cookie1 2015-04-10 10:00:02     url2    2       url1  
cookie1 2015-04-10 10:03:04     1url3   3       url1  
cookie1 2015-04-10 10:10:00     url4    4       url1  
cookie1 2015-04-10 10:50:01     url5    5       url1  
cookie1 2015-04-10 10:50:05     url6    6       url1  
cookie1 2015-04-10 11:00:00     url7    7       url1  
cookie2 2015-04-10 10:00:00     url11   1       url11  
cookie2 2015-04-10 10:00:02     url22   2       url11  
cookie2 2015-04-10 10:03:04     1url33  3       url11  
cookie2 2015-04-10 10:10:00     url44   4       url11  
cookie2 2015-04-10 10:50:01     url55   5       url11  
cookie2 2015-04-10 10:50:05     url66   6       url11  
cookie2 2015-04-10 11:00:00     url77   7       url11  

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24

LAST_VALUE

取分组内排序后，截止到当前行，最后一个值

SELECT cookieid,  
createtime,  
url,  
LAST_VALUE(url) OVER(PARTITION BY cookieid) AS last2    
FROM lxw1234;  

cookieid  createtime            url     last2  
----------------------------------------------  
cookie1 2015-04-10 10:00:02     url2    url5  
cookie1 2015-04-10 10:00:00     url1    url5  
cookie1 2015-04-10 10:03:04     1url3   url5  
cookie1 2015-04-10 10:50:05     url6    url5  
cookie1 2015-04-10 11:00:00     url7    url5  
cookie1 2015-04-10 10:10:00     url4    url5  
cookie1 2015-04-10 10:50:01     url5    url5  
cookie2 2015-04-10 10:00:02     url22   url55  
cookie2 2015-04-10 10:00:00     url11   url55  
cookie2 2015-04-10 10:03:04     1url33  url55  
cookie2 2015-04-10 10:50:05     url66   url55  
cookie2 2015-04-10 11:00:00     url77   url55  
cookie2 2015-04-10 10:10:00     url44   url55  
cookie2 2015-04-10 10:50:01     url55   url55  
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22

分析函数

  1. ROW_NUMBER() 从1开始，按照顺序，生成分组内记录的序列，比如，按照pv降序排列，生成分组内每天的pv名次，ROW_NUMBER()的应用场景非常多，再比如，获取分组内排序第一的记录，获取一个session中的第一条refer等
  2. RANK() 生成数据项在分组中的排名，排名相等会在名次中留下空位
  3. DENSE_RANK() 生成数据项在分组中的排名，排名相等会在名次中不会留下空位
  4. CUME_DIST() 小于等于当前值的行数除以分组内总行数。比如，统计小于等于当前薪水的人数所占总人数的比例
  5. PERCENT_RANK() 分组内当前行的RANK值-1/分组内 总行数-1
  6. NTILE(n) 用于将分组数据按照顺序切分成n片，返回当前切片值，如果切片不均匀，默认增加第一个切片的分布。NTILE不支持ROWS BETWEEN,比如NTILE(2) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PERCEDING AND CURRENT ROW)

累计操作：sum

## 创建数据表
create table orders(
    user_id string,
    device_id string,
    user_type string,
    price float,
    sales int);

## 添加数据orders.txt
zhangsa test1   new     67.1    2
lisi    test2   old     43.32   1
wanger  test3   new     88.88   3
liliu   test4   new     66.0    1
tom     test5   new     54.32   1
tomas   test6   old     77.77   2
tomson  test7   old     88.44   3
tom1    test8   new     56.55   6
tom2    test9   new     88.88   5
tom3    test10  new     66.66   5

## 开窗函数案例
select
    user_id,
    user_type,
    sales,
    -- 默认从起点到当前所有重复行
    sum(sales) over(partition by user_type order by sales asc) as sales_1,
    -- 从起点到当前所有重复行与sales_1结果相同
    sum(sales) over(partition by user_type order by sales asc range between unbounded preceding and current row) as sales_2,
    -- 从起点到当前行，结果与sale_1结果不同
    sum(sales) over(partition by user_type order by sales asc rows between unbounded preceding and current row) as sales_3,
    -- 当前行加上往前3行
    sum(sales) over(partition by user_type order by sales asc rows between 3 preceding and current row) as sales_4,
    -- 当前范围往上加3行
    sum(sales) over(partition by user_type order by sales asc range between 3 preceding and current row) as sales_5,
    -- 当前行+往前3行+往后1行
    sum(sales) over(partition by user_type order by sales asc rows between 3 preceding and 1 following) as sales_6,
    --
    sum(sales) over(partition by user_type order by sales asc range between 3 preceding and 1 following) as sales_7,
    -- 当前行+之后所有行
    sum(sales) over(partition by user_type order by sales asc rows between current row and unbounded following) as sales_8,
    --
    sum(sales) over(partition by user_type order by sales asc range between current row and unbounded following) as sales_9,
    -- 分组内所有行
    sum(sales) over(partition by user_type) as sales_10
from
    orders
order by
    user_type,
    sales,
    user_id;

##上述查询结果如下：

| user_id  | user_type  | sales  | sales_1  | sales_2  | sales_3  | sales_4  | sales_5  | sales_6  | sales_7  | sales_8  | sales_9  | sales_10  |
|----------|------------|--------|----------|----------|----------|----------|----------|----------|----------|----------|----------|-----------|
| liliu    | new        | 1      | 2        | 2        | 2        | 2        | 2        | 4        | 4        | 22       | 23       | 23        |
| tom      | new        | 1      | 2        | 2        | 1        | 1        | 2        | 2        | 4        | 23       | 23       | 23        |
| zhangsa  | new        | 2      | 4        | 4        | 4        | 4        | 4        | 7        | 7        | 21       | 21       | 23        |
| wanger   | new        | 3      | 7        | 7        | 7        | 7        | 7        | 12       | 7        | 19       | 19       | 23        |
| tom2     | new        | 5      | 17       | 17       | 17       | 15       | 15       | 21       | 21       | 11       | 16       | 23        |
| tom3     | new        | 5      | 17       | 17       | 12       | 11       | 15       | 16       | 21       | 16       | 16       | 23        |
| tom1     | new        | 6      | 23       | 23       | 23       | 19       | 19       | 19       | 19       | 6        | 6        | 23        |
| lisi     | old        | 1      | 1        | 1        | 1        | 1        | 1        | 3        | 3        | 6        | 6        | 6         |
| tomas    | old        | 2      | 3        | 3        | 3        | 3        | 3        | 6        | 6        | 5        | 5        | 6         |
| tomson   | old        | 3      | 6        | 6        | 6        | 6        | 6        | 6        | 6        | 3        | 3        | 6         |
注意
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67

RANK、ROW_NUMBER、DENSE_RANK

select
user_id,user_type,sales,
RANK() over (partition by user_type order by sales desc) as r,
ROW_NUMBER() over (partition by user_type order by sales desc) as rn,
DENSE_RANK() over (partition by user_type order by sales desc) as dr
from
orders;

##上述查询结果如下

| user_id | user_type | sales |  r  | rn  | dr  |
| ------- | --------- | ----- | --- | --- | --- |
| tom1    | new       | 6     | 1   | 1   | 1   |
| tom3    | new       | 5     | 2   | 2   | 2   |
| tom2    | new       | 5     | 2   | 3   | 2   |
| wanger  | new       | 3     | 4   | 4   | 3   |
| zhangsa | new       | 2     | 5   | 5   | 4   |
| tom     | new       | 1     | 6   | 6   | 5   |
| liliu   | new       | 1     | 6   | 7   | 5   |
| tomson  | old       | 3     | 1   | 1   | 1   |
| tomas   | old       | 2     | 2   | 2   | 2   |
| lisi    | old       | 1     | 3   | 3   | 3   |
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/小小林熬夜学编程/article/detail/656572
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号