当前位置:   article > 正文

Hive-on-MR与Hive-on-Tez的性能比较

hive on mr会读取hive所有数据吗

    实验数据:10G大小的TPCH数据(8个表);

    实验环境:1Master节点+8Slave节点,每个均为4core,4G内存,30G磁盘,磁盘读写速度50M/s;

    ①单个查询之q1_pricing_summary_report.hive

  1. SELECT
  2. L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY), SUM(L_EXTENDEDPRICE), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)), AVG(L_QUANTITY), AVG(L_EXTENDEDPRICE), AVG(L_DISCOUNT), COUNT(1)
  3. FROM
  4. lineitem
  5. WHERE
  6. L_SHIPDATE<='1998-09-02'
  7. GROUP BY L_RETURNFLAG, L_LINESTATUS
  8. ORDER BY L_RETURNFLAG, L_LINESTATUS;

    ——set hive.execution.engine=mr

  1. hive> SELECT
  2. > L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY), SUM(L_EXTENDEDPRICE), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)), AVG(L_QUANTITY), AVG(L_EXTENDEDPRICE), AVG(L_DISCOUNT), COUNT(1)
  3. > FROM
  4. > lineitem
  5. > WHERE
  6. > L_SHIPDATE<='1998-09-02'
  7. > GROUP BY L_RETURNFLAG, L_LINESTATUS
  8. > ORDER BY L_RETURNFLAG, L_LINESTATUS;
  9. 16/08/12 18:19:57 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
  10. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  11. Query ID = lwh_20160812181955_0ba24dd7-25a8-44c6-9005-f5adacfc1b08
  12. Total jobs = 2
  13. Launching Job 1 out of 2
  14. Number of reduce tasks not specified. Estimated from input data size: 31
  15. In order to change the average load for a reducer (in bytes):
  16. set hive.exec.reducers.bytes.per.reducer=<number>
  17. In order to limit the maximum number of reducers:
  18. set hive.exec.reducers.max=<number>
  19. In order to set a constant number of reducers:
  20. set mapreduce.job.reduces=<number>
  21. 16/08/12 18:20:02 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
  22. 16/08/12 18:20:03 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  23. 16/08/12 18:20:04 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  24. 16/08/12 18:20:04 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  25. 16/08/12 18:21:48 INFO input.FileInputFormat: Total input paths to process : 1
  26. 16/08/12 18:21:48 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 268435456
  27. 16/08/12 18:21:49 INFO mapreduce.JobSubmitter: number of splits:29
  28. 16/08/12 18:21:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0007
  29. 16/08/12 18:21:50 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0007
  30. 16/08/12 18:21:50 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0007/
  31. Starting Job = job_1470994648796_0007, Tracking URL = http://node643:8088/proxy/application_1470994648796_0007/
  32. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0007
  33. Hadoop job information for Stage-1: number of mappers: 29; number of reducers: 31
  34. 16/08/12 18:22:15 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  35. 2016-08-12 18:22:15,564 Stage-1 map = 0%, reduce = 0%
  36. 2016-08-12 18:22:40,147 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 40.74 sec
  37. 2016-08-12 18:22:45,548 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 189.74 sec
  38. 2016-08-12 18:22:46,625 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 211.75 sec
  39. 2016-08-12 18:22:50,940 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 298.17 sec
  40. 2016-08-12 18:22:52,019 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 306.72 sec
  41. 2016-08-12 18:22:53,097 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 313.03 sec
  42. 2016-08-12 18:22:54,173 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 341.25 sec
  43. 2016-08-12 18:22:58,499 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 408.43 sec
  44. 2016-08-12 18:23:00,651 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 458.76 sec
  45. 2016-08-12 18:23:01,724 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 463.35 sec
  46. 2016-08-12 18:23:03,873 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 474.89 sec
  47. 2016-08-12 18:23:09,249 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 485.99 sec
  48. 2016-08-12 18:23:11,405 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 492.84 sec
  49. 2016-08-12 18:23:12,498 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 509.8 sec
  50. 2016-08-12 18:23:14,676 Stage-1 map = 53%, reduce = 1%, Cumulative CPU 533.6 sec
  51. 2016-08-12 18:23:16,828 Stage-1 map = 55%, reduce = 1%, Cumulative CPU 598.25 sec
  52. 2016-08-12 18:23:20,048 Stage-1 map = 55%, reduce = 2%, Cumulative CPU 636.57 sec
  53. 2016-08-12 18:23:21,134 Stage-1 map = 56%, reduce = 2%, Cumulative CPU 649.37 sec
  54. 2016-08-12 18:23:23,298 Stage-1 map = 57%, reduce = 2%, Cumulative CPU 669.42 sec
  55. 2016-08-12 18:23:25,443 Stage-1 map = 61%, reduce = 2%, Cumulative CPU 689.89 sec
  56. 2016-08-12 18:23:26,531 Stage-1 map = 62%, reduce = 2%, Cumulative CPU 697.35 sec
  57. 2016-08-12 18:23:27,602 Stage-1 map = 64%, reduce = 2%, Cumulative CPU 715.54 sec
  58. 2016-08-12 18:23:28,671 Stage-1 map = 67%, reduce = 2%, Cumulative CPU 728.59 sec
  59. 2016-08-12 18:23:31,881 Stage-1 map = 74%, reduce = 2%, Cumulative CPU 777.91 sec
  60. 2016-08-12 18:23:32,951 Stage-1 map = 83%, reduce = 3%, Cumulative CPU 791.75 sec
  61. 2016-08-12 18:23:36,158 Stage-1 map = 83%, reduce = 4%, Cumulative CPU 809.7 sec
  62. 2016-08-12 18:23:39,366 Stage-1 map = 84%, reduce = 4%, Cumulative CPU 819.73 sec
  63. 2016-08-12 18:23:40,450 Stage-1 map = 85%, reduce = 4%, Cumulative CPU 822.8 sec
  64. 2016-08-12 18:23:42,587 Stage-1 map = 86%, reduce = 4%, Cumulative CPU 830.3 sec
  65. 2016-08-12 18:23:45,806 Stage-1 map = 89%, reduce = 4%, Cumulative CPU 865.81 sec
  66. 2016-08-12 18:23:46,880 Stage-1 map = 89%, reduce = 7%, Cumulative CPU 872.51 sec
  67. 2016-08-12 18:23:47,951 Stage-1 map = 89%, reduce = 8%, Cumulative CPU 883.72 sec
  68. 2016-08-12 18:23:49,021 Stage-1 map = 91%, reduce = 9%, Cumulative CPU 891.21 sec
  69. 2016-08-12 18:23:50,097 Stage-1 map = 93%, reduce = 9%, Cumulative CPU 893.01 sec
  70. 2016-08-12 18:23:52,247 Stage-1 map = 93%, reduce = 10%, Cumulative CPU 900.14 sec
  71. 2016-08-12 18:23:53,317 Stage-1 map = 95%, reduce = 10%, Cumulative CPU 908.03 sec
  72. 2016-08-12 18:23:57,624 Stage-1 map = 95%, reduce = 11%, Cumulative CPU 916.72 sec
  73. 2016-08-12 18:24:00,843 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 926.87 sec
  74. 2016-08-12 18:24:01,916 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 930.05 sec
  75. 2016-08-12 18:24:02,988 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 942.11 sec
  76. 2016-08-12 18:24:04,062 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 962.53 sec
  77. 2016-08-12 18:24:05,135 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 993.55 sec
  78. 2016-08-12 18:24:07,282 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 1001.0 sec
  79. 2016-08-12 18:24:11,593 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 1015.47 sec
  80. 2016-08-12 18:24:13,733 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 1020.94 sec
  81. 2016-08-12 18:24:14,800 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 1047.42 sec
  82. 2016-08-12 18:24:15,868 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1073.01 sec
  83. 2016-08-12 18:24:16,936 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1083.68 sec
  84. 2016-08-12 18:24:20,139 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 1088.29 sec
  85. 2016-08-12 18:24:21,205 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1100.29 sec
  86. 2016-08-12 18:24:23,354 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1112.48 sec
  87. MapReduce Total cumulative CPU time: 18 minutes 32 seconds 480 msec
  88. Ended Job = job_1470994648796_0007
  89. Launching Job 2 out of 2
  90. Number of reduce tasks determined at compile time: 1
  91. In order to change the average load for a reducer (in bytes):
  92. set hive.exec.reducers.bytes.per.reducer=<number>
  93. In order to limit the maximum number of reducers:
  94. set hive.exec.reducers.max=<number>
  95. In order to set a constant number of reducers:
  96. set mapreduce.job.reduces=<number>
  97. 16/08/12 18:24:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  98. 16/08/12 18:24:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  99. 16/08/12 18:24:28 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  100. 16/08/12 18:26:10 INFO input.FileInputFormat: Total input paths to process : 31
  101. 16/08/12 18:26:10 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
  102. 16/08/12 18:26:11 INFO mapreduce.JobSubmitter: number of splits:6
  103. 16/08/12 18:26:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0008
  104. 16/08/12 18:26:12 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0008
  105. 16/08/12 18:26:12 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0008/
  106. Starting Job = job_1470994648796_0008, Tracking URL = http://node643:8088/proxy/application_1470994648796_0008/
  107. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0008
  108. Hadoop job information for Stage-2: number of mappers: 6; number of reducers: 1
  109. 16/08/12 18:26:35 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  110. 2016-08-12 18:26:35,127 Stage-2 map = 0%, reduce = 0%
  111. 2016-08-12 18:26:58,048 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 11.12 sec
  112. 2016-08-12 18:27:00,187 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 21.08 sec
  113. 2016-08-12 18:27:07,672 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 26.1 sec
  114. MapReduce Total cumulative CPU time: 26 seconds 100 msec
  115. Ended Job = job_1470994648796_0008
  116. MapReduce Jobs Launched:
  117. Stage-Stage-1: Map: 29 Reduce: 31 Cumulative CPU: 1112.48 sec HDFS Read: 7776522714 HDFS Write: 3305 SUCCESS
  118. Stage-Stage-2: Map: 6 Reduce: 1 Cumulative CPU: 26.1 sec HDFS Read: 39220 HDFS Write: 720 SUCCESS
  119. Total MapReduce CPU Time Spent: 18 minutes 58 seconds 580 msec
  120. OK
  121. 16/08/12 18:27:11 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  122. 16/08/12 18:27:11 INFO mapred.FileInputFormat: Total input paths to process : 1
  123. A F 3.77518399E8 5.660657277972468E11 5.377591042780681E11 5.592766708921187E11 25.500975103007097 38237.15100895833 0.050006574540269355 14804077
  124. N F 9851614.0 1.4767438399169975E10 1.4028805792211395E10 1.459049099836673E10 25.522448302840946 38257.81066008108 0.04997336773765436 385998
  125. N O 7.43124873E8 1.1143022869018765E12 1.0585809221449628E12 1.100937000170583E12 25.498075870689316 38233.90292348169 0.0500008118209974 29144351
  126. R F 3.7773283E8 5.664310549759971E11 5.3811092266476355E11 5.59634780885086E11 25.50838478968014 38251.21927355956 0.04999679231411408 14808183
  127. Time taken: 436.074 seconds, Fetched: 4 row(s)

    ——set hive.execution.engine=tez

  1. hive> set hive.execution.engine=tez;
  2. hive> SELECT
  3. > L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY), SUM(L_EXTENDEDPRICE), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)), AVG(L_QUANTITY), AVG(L_EXTENDEDPRICE), AVG(L_DISCOUNT), COUNT(1)
  4. > FROM
  5. > lineitem
  6. > WHERE
  7. > L_SHIPDATE<='1998-09-02'
  8. > GROUP BY L_RETURNFLAG, L_LINESTATUS
  9. > ORDER BY L_RETURNFLAG, L_LINESTATUS;
  10. 16/08/12 19:01:38 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
  11. 16/08/12 19:01:38 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
  12. Query ID = lwh_20160812190132_2f9c1fad-5f40-4ad0-99bf-57a921acf812
  13. Total jobs = 1
  14. Launching Job 1 out of 1
  15. 16/08/12 19:01:39 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
  16. 16/08/12 19:03:02 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  17. 16/08/12 19:03:03 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0009
  18. 16/08/12 19:03:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  19. Status: Running (Executing on YARN cluster with App id application_1470994648796_0009)
  20. ----------------------------------------------------------------------------------------------
  21. VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
  22. ----------------------------------------------------------------------------------------------
  23. Map 1 .......... container SUCCEEDED 58 58 0 0 0 0
  24. Reducer 2 ...... container SUCCEEDED 11 11 0 0 0 0
  25. Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0
  26. ----------------------------------------------------------------------------------------------
  27. VERTICES: 03/03 [==========================>>] 100% ELAPSED TIME: 72.51 s
  28. ----------------------------------------------------------------------------------------------
  29. OK
  30. 16/08/12 19:04:42 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  31. 16/08/12 19:04:42 INFO mapred.FileInputFormat: Total input paths to process : 1
  32. A F 3.77518399E8 5.660657277972496E11 5.3775910427806433E11 5.592766708921171E11 25.500975103007097 38237.15100895852 0.05000657454031599 14804077
  33. N F 9851614.0 1.4767438399169992E10 1.4028805792211395E10 1.4590490998366734E10 25.522448302840946 38257.81066008112 0.049973367737656114 385998
  34. N O 7.43124873E8 1.1143022869018794E12 1.0585809221449642E12 1.1009370001705938E12 25.498075870689316 38233.902923481786 0.050000811821159556 29144351
  35. R F 3.7773283E8 5.664310549760012E11 5.3811092266476697E11 5.596347808850865E11 25.50838478968014 38251.219273559844 0.04999679231416023 14808183
  36. Time taken: 189.653 seconds, Fetched: 4 row(s)

    ②单个查询之q5_local_supplier_volume.hive

  1. select
  2. n_name, sum(l_extendedprice * (1 - l_discount)) as revenue
  3. from
  4. customer c join
  5. ( select n_name, l_extendedprice, l_discount, s_nationkey, o_custkey from orders o join
  6. ( select n_name, l_extendedprice, l_discount, l_orderkey, s_nationkey from lineitem l join
  7. ( select n_name, s_suppkey, s_nationkey from supplier s join
  8. ( select n_name, n_nationkey
  9. from nation n join region r
  10. on n.n_regionkey = r.r_regionkey and r.r_name = 'ASIA'
  11. ) n1 on s.s_nationkey = n1.n_nationkey
  12. ) s1 on l.l_suppkey = s1.s_suppkey
  13. ) l1 on l1.l_orderkey = o.o_orderkey and o.o_orderdate >= '1994-01-01'
  14. and o.o_orderdate < '1995-01-01'
  15. ) o1
  16. on c.c_nationkey = o1.s_nationkey and c.c_custkey = o1.o_custkey
  17. group by n_name
  18. order by revenue desc;

    ——set hive.execution.engine=mr

  1. lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q5_local_supplier_volume.hive
  2. which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
  3. SLF4J: Class path contains multiple SLF4J bindings.
  4. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  5. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  6. SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  7. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  8. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  9. 16/08/12 19:20:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  10. Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
  11. Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  12. 16/08/12 19:20:25 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
  13. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  14. Query ID = lwh_20160812192025_499bf4fc-1185-495a-b02c-e81285fd7740
  15. Total jobs = 15
  16. SLF4J: Class path contains multiple SLF4J bindings.
  17. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  18. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  19. SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  20. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  21. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  22. 16/08/12 19:20:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  23. 2016-08-12 19:20:46 Starting to launch local task to process map join; maximum memory = 477626368
  24. 16/08/12 19:20:49 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  25. 16/08/12 19:20:49 INFO mapred.FileInputFormat: Total input paths to process : 1
  26. 2016-08-12 19:20:49 Dump the side-table for tag: 1 with group count: 1 into file: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10024/HashTable-Stage-22/MapJoin-mapfile81--.hashtable
  27. 2016-08-12 19:20:49 Uploaded 1 File to: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10024/HashTable-Stage-22/MapJoin-mapfile81--.hashtable (278 bytes)
  28. 2016-08-12 19:20:49 End of local task; Time Taken: 3.023 sec.
  29. Execution completed successfully
  30. MapredLocal task succeeded
  31. Launching Job 1 out of 15
  32. Number of reduce tasks is set to 0 since there's no reduce operator
  33. 16/08/12 19:20:51 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
  34. 16/08/12 19:20:52 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  35. 16/08/12 19:20:53 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  36. 16/08/12 19:20:53 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  37. 16/08/12 19:22:38 INFO input.FileInputFormat: Total input paths to process : 1
  38. 16/08/12 19:22:38 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
  39. 16/08/12 19:22:39 INFO mapreduce.JobSubmitter: number of splits:1
  40. 16/08/12 19:22:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0011
  41. 16/08/12 19:22:40 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0011
  42. 16/08/12 19:22:40 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0011/
  43. Starting Job = job_1470994648796_0011, Tracking URL = http://node643:8088/proxy/application_1470994648796_0011/
  44. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0011
  45. Hadoop job information for Stage-22: number of mappers: 1; number of reducers: 0
  46. 16/08/12 19:23:09 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  47. 2016-08-12 19:23:09,546 Stage-22 map = 0%, reduce = 0%
  48. 2016-08-12 19:23:37,474 Stage-22 map = 100%, reduce = 0%, Cumulative CPU 6.64 sec
  49. MapReduce Total cumulative CPU time: 6 seconds 640 msec
  50. Ended Job = job_1470994648796_0011
  51. Stage-29 is selected by condition resolver.
  52. Stage-30 is filtered out by condition resolver.
  53. Stage-7 is filtered out by condition resolver.
  54. SLF4J: Class path contains multiple SLF4J bindings.
  55. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  56. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  57. SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  58. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  59. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  60. 16/08/12 19:23:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  61. 2016-08-12 19:23:52 Starting to launch local task to process map join; maximum memory = 477626368
  62. 16/08/12 19:23:54 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  63. 16/08/12 19:23:54 INFO mapred.FileInputFormat: Total input paths to process : 1
  64. 2016-08-12 19:23:54 Dump the side-table for tag: 1 with group count: 5 into file: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10020/HashTable-Stage-19/MapJoin-mapfile61--.hashtable
  65. 2016-08-12 19:23:54 Uploaded 1 File to: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10020/HashTable-Stage-19/MapJoin-mapfile61--.hashtable (391 bytes)
  66. 2016-08-12 19:23:54 End of local task; Time Taken: 2.383 sec.
  67. Execution completed successfully
  68. MapredLocal task succeeded
  69. Launching Job 3 out of 15
  70. Number of reduce tasks is set to 0 since there's no reduce operator
  71. 16/08/12 19:23:58 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  72. 16/08/12 19:23:59 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  73. 16/08/12 19:23:59 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  74. 16/08/12 19:25:43 INFO input.FileInputFormat: Total input paths to process : 1
  75. 16/08/12 19:25:43 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
  76. 16/08/12 19:25:44 INFO mapreduce.JobSubmitter: number of splits:1
  77. 16/08/12 19:25:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0012
  78. 16/08/12 19:25:45 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0012
  79. 16/08/12 19:25:45 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0012/
  80. Starting Job = job_1470994648796_0012, Tracking URL = http://node643:8088/proxy/application_1470994648796_0012/
  81. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0012
  82. Hadoop job information for Stage-19: number of mappers: 1; number of reducers: 0
  83. 16/08/12 19:26:15 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  84. 2016-08-12 19:26:15,132 Stage-19 map = 0%, reduce = 0%
  85. 2016-08-12 19:26:42,840 Stage-19 map = 100%, reduce = 0%, Cumulative CPU 9.47 sec
  86. MapReduce Total cumulative CPU time: 9 seconds 470 msec
  87. Ended Job = job_1470994648796_0012
  88. Stage-27 is selected by condition resolver.
  89. Stage-28 is filtered out by condition resolver.
  90. Stage-6 is filtered out by condition resolver.
  91. SLF4J: Class path contains multiple SLF4J bindings.
  92. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  93. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  94. SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  95. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  96. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  97. 16/08/12 19:26:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  98. 2016-08-12 19:26:57 Starting to launch local task to process map join; maximum memory = 477626368
  99. 16/08/12 19:26:59 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  100. 16/08/12 19:26:59 INFO mapred.FileInputFormat: Total input paths to process : 1
  101. 2016-08-12 19:27:00 Dump the side-table for tag: 1 with group count: 20037 into file: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10016/HashTable-Stage-16/MapJoin-mapfile41--.hashtable
  102. 2016-08-12 19:27:00 Uploaded 1 File to: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10016/HashTable-Stage-16/MapJoin-mapfile41--.hashtable (594855 bytes)
  103. 2016-08-12 19:27:00 End of local task; Time Taken: 2.755 sec.
  104. Execution completed successfully
  105. MapredLocal task succeeded
  106. Launching Job 5 out of 15
  107. Number of reduce tasks is set to 0 since there's no reduce operator
  108. 16/08/12 19:27:02 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  109. 16/08/12 19:27:02 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  110. 16/08/12 19:27:02 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  111. 16/08/12 19:28:45 INFO input.FileInputFormat: Total input paths to process : 1
  112. 16/08/12 19:28:45 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 268435456
  113. 16/08/12 19:28:46 INFO mapreduce.JobSubmitter: number of splits:29
  114. 16/08/12 19:28:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0013
  115. 16/08/12 19:28:48 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0013
  116. 16/08/12 19:28:48 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0013/
  117. Starting Job = job_1470994648796_0013, Tracking URL = http://node643:8088/proxy/application_1470994648796_0013/
  118. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0013
  119. Hadoop job information for Stage-16: number of mappers: 29; number of reducers: 0
  120. 16/08/12 19:29:15 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  121. 2016-08-12 19:29:15,011 Stage-16 map = 0%, reduce = 0%
  122. 2016-08-12 19:29:35,792 Stage-16 map = 2%, reduce = 0%, Cumulative CPU 19.35 sec
  123. 2016-08-12 19:29:39,009 Stage-16 map = 3%, reduce = 0%, Cumulative CPU 91.45 sec
  124. 2016-08-12 19:29:43,324 Stage-16 map = 7%, reduce = 0%, Cumulative CPU 199.14 sec
  125. 2016-08-12 19:29:45,479 Stage-16 map = 14%, reduce = 0%, Cumulative CPU 225.15 sec
  126. 2016-08-12 19:29:47,625 Stage-16 map = 16%, reduce = 0%, Cumulative CPU 244.67 sec
  127. 2016-08-12 19:29:48,696 Stage-16 map = 22%, reduce = 0%, Cumulative CPU 281.88 sec
  128. 2016-08-12 19:29:49,779 Stage-16 map = 31%, reduce = 0%, Cumulative CPU 293.2 sec
  129. 2016-08-12 19:29:50,846 Stage-16 map = 34%, reduce = 0%, Cumulative CPU 312.23 sec
  130. 2016-08-12 19:29:51,915 Stage-16 map = 36%, reduce = 0%, Cumulative CPU 330.85 sec
  131. 2016-08-12 19:29:52,986 Stage-16 map = 40%, reduce = 0%, Cumulative CPU 337.59 sec
  132. 2016-08-12 19:29:54,049 Stage-16 map = 41%, reduce = 0%, Cumulative CPU 341.42 sec
  133. 2016-08-12 19:29:55,129 Stage-16 map = 43%, reduce = 0%, Cumulative CPU 347.94 sec
  134. 2016-08-12 19:29:56,203 Stage-16 map = 45%, reduce = 0%, Cumulative CPU 349.7 sec
  135. 2016-08-12 19:29:57,277 Stage-16 map = 47%, reduce = 0%, Cumulative CPU 358.26 sec
  136. 2016-08-12 19:29:59,426 Stage-16 map = 48%, reduce = 0%, Cumulative CPU 361.21 sec
  137. 2016-08-12 19:30:00,506 Stage-16 map = 52%, reduce = 0%, Cumulative CPU 379.57 sec
  138. 2016-08-12 19:30:05,873 Stage-16 map = 53%, reduce = 0%, Cumulative CPU 443.3 sec
  139. 2016-08-12 19:30:06,941 Stage-16 map = 57%, reduce = 0%, Cumulative CPU 460.63 sec
  140. 2016-08-12 19:30:08,009 Stage-16 map = 60%, reduce = 0%, Cumulative CPU 483.02 sec
  141. 2016-08-12 19:30:11,219 Stage-16 map = 64%, reduce = 0%, Cumulative CPU 553.62 sec
  142. 2016-08-12 19:30:12,287 Stage-16 map = 67%, reduce = 0%, Cumulative CPU 589.04 sec
  143. 2016-08-12 19:30:13,355 Stage-16 map = 72%, reduce = 0%, Cumulative CPU 598.74 sec
  144. 2016-08-12 19:30:15,494 Stage-16 map = 74%, reduce = 0%, Cumulative CPU 622.0 sec
  145. 2016-08-12 19:30:16,565 Stage-16 map = 78%, reduce = 0%, Cumulative CPU 632.12 sec
  146. 2016-08-12 19:30:17,638 Stage-16 map = 81%, reduce = 0%, Cumulative CPU 641.87 sec
  147. 2016-08-12 19:30:18,717 Stage-16 map = 83%, reduce = 0%, Cumulative CPU 651.4 sec
  148. 2016-08-12 19:30:19,784 Stage-16 map = 84%, reduce = 0%, Cumulative CPU 654.3 sec
  149. 2016-08-12 19:30:20,851 Stage-16 map = 90%, reduce = 0%, Cumulative CPU 670.98 sec
  150. 2016-08-12 19:30:21,920 Stage-16 map = 91%, reduce = 0%, Cumulative CPU 674.26 sec
  151. 2016-08-12 19:30:22,992 Stage-16 map = 93%, reduce = 0%, Cumulative CPU 682.34 sec
  152. 2016-08-12 19:30:26,194 Stage-16 map = 98%, reduce = 0%, Cumulative CPU 697.53 sec
  153. 2016-08-12 19:30:27,261 Stage-16 map = 100%, reduce = 0%, Cumulative CPU 698.01 sec
  154. MapReduce Total cumulative CPU time: 11 minutes 38 seconds 10 msec
  155. Ended Job = job_1470994648796_0013
  156. Stage-25 is filtered out by condition resolver.
  157. Stage-26 is filtered out by condition resolver.
  158. Stage-5 is selected by condition resolver.
  159. Launching Job 6 out of 15
  160. Number of reduce tasks not specified. Estimated from input data size: 10
  161. In order to change the average load for a reducer (in bytes):
  162. set hive.exec.reducers.bytes.per.reducer=<number>
  163. In order to limit the maximum number of reducers:
  164. set hive.exec.reducers.max=<number>
  165. In order to set a constant number of reducers:
  166. set mapreduce.job.reduces=<number>
  167. 16/08/12 19:30:31 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  168. 16/08/12 19:30:31 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  169. 16/08/12 19:30:31 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  170. 16/08/12 19:32:16 INFO input.FileInputFormat: Total input paths to process : 30
  171. 16/08/12 19:32:16 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 4364567
  172. 16/08/12 19:32:16 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
  173. 16/08/12 19:32:17 INFO mapreduce.JobSubmitter: number of splits:15
  174. 16/08/12 19:32:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0014
  175. 16/08/12 19:32:18 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0014
  176. 16/08/12 19:32:18 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0014/
  177. Starting Job = job_1470994648796_0014, Tracking URL = http://node643:8088/proxy/application_1470994648796_0014/
  178. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0014
  179. Hadoop job information for Stage-5: number of mappers: 15; number of reducers: 10
  180. 16/08/12 19:32:47 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  181. 2016-08-12 19:32:47,002 Stage-5 map = 0%, reduce = 0%
  182. 2016-08-12 19:33:05,006 Stage-5 map = 2%, reduce = 0%, Cumulative CPU 16.74 sec
  183. 2016-08-12 19:33:08,223 Stage-5 map = 4%, reduce = 0%, Cumulative CPU 20.43 sec
  184. 2016-08-12 19:33:09,293 Stage-5 map = 7%, reduce = 0%, Cumulative CPU 33.74 sec
  185. 2016-08-12 19:33:14,659 Stage-5 map = 8%, reduce = 0%, Cumulative CPU 140.57 sec
  186. 2016-08-12 19:33:15,734 Stage-5 map = 19%, reduce = 0%, Cumulative CPU 157.44 sec
  187. 2016-08-12 19:33:16,804 Stage-5 map = 39%, reduce = 0%, Cumulative CPU 236.11 sec
  188. 2016-08-12 19:33:17,874 Stage-5 map = 46%, reduce = 0%, Cumulative CPU 245.0 sec
  189. 2016-08-12 19:33:20,023 Stage-5 map = 55%, reduce = 0%, Cumulative CPU 271.72 sec
  190. 2016-08-12 19:33:21,092 Stage-5 map = 58%, reduce = 0%, Cumulative CPU 278.26 sec
  191. 2016-08-12 19:33:22,172 Stage-5 map = 68%, reduce = 1%, Cumulative CPU 293.54 sec
  192. 2016-08-12 19:33:23,248 Stage-5 map = 73%, reduce = 1%, Cumulative CPU 302.53 sec
  193. 2016-08-12 19:33:25,403 Stage-5 map = 76%, reduce = 2%, Cumulative CPU 320.42 sec
  194. 2016-08-12 19:33:26,477 Stage-5 map = 79%, reduce = 2%, Cumulative CPU 330.54 sec
  195. 2016-08-12 19:33:28,621 Stage-5 map = 81%, reduce = 2%, Cumulative CPU 346.81 sec
  196. 2016-08-12 19:33:29,690 Stage-5 map = 84%, reduce = 7%, Cumulative CPU 356.93 sec
  197. 2016-08-12 19:33:31,840 Stage-5 map = 85%, reduce = 7%, Cumulative CPU 372.83 sec
  198. 2016-08-12 19:33:32,906 Stage-5 map = 86%, reduce = 8%, Cumulative CPU 377.17 sec
  199. 2016-08-12 19:33:35,047 Stage-5 map = 86%, reduce = 10%, Cumulative CPU 385.15 sec
  200. 2016-08-12 19:33:36,106 Stage-5 map = 86%, reduce = 14%, Cumulative CPU 390.71 sec
  201. 2016-08-12 19:33:37,174 Stage-5 map = 89%, reduce = 16%, Cumulative CPU 401.5 sec
  202. 2016-08-12 19:33:38,252 Stage-5 map = 93%, reduce = 16%, Cumulative CPU 417.24 sec
  203. 2016-08-12 19:33:39,325 Stage-5 map = 93%, reduce = 18%, Cumulative CPU 422.06 sec
  204. 2016-08-12 19:33:40,395 Stage-5 map = 93%, reduce = 20%, Cumulative CPU 429.3 sec
  205. 2016-08-12 19:33:41,466 Stage-5 map = 93%, reduce = 21%, Cumulative CPU 430.34 sec
  206. 2016-08-12 19:33:44,689 Stage-5 map = 93%, reduce = 24%, Cumulative CPU 445.4 sec
  207. 2016-08-12 19:33:48,979 Stage-5 map = 93%, reduce = 27%, Cumulative CPU 453.62 sec
  208. 2016-08-12 19:34:00,732 Stage-5 map = 96%, reduce = 27%, Cumulative CPU 481.33 sec
  209. 2016-08-12 19:34:01,806 Stage-5 map = 98%, reduce = 27%, Cumulative CPU 496.84 sec
  210. 2016-08-12 19:34:02,873 Stage-5 map = 98%, reduce = 29%, Cumulative CPU 498.16 sec
  211. 2016-08-12 19:34:03,940 Stage-5 map = 98%, reduce = 30%, Cumulative CPU 498.71 sec
  212. 2016-08-12 19:34:05,006 Stage-5 map = 98%, reduce = 31%, Cumulative CPU 499.47 sec
  213. 2016-08-12 19:34:17,843 Stage-5 map = 100%, reduce = 31%, Cumulative CPU 526.06 sec
  214. 2016-08-12 19:34:18,917 Stage-5 map = 100%, reduce = 32%, Cumulative CPU 528.93 sec
  215. 2016-08-12 19:34:19,987 Stage-5 map = 100%, reduce = 33%, Cumulative CPU 532.83 sec
  216. 2016-08-12 19:34:21,054 Stage-5 map = 100%, reduce = 37%, Cumulative CPU 552.13 sec
  217. 2016-08-12 19:34:22,123 Stage-5 map = 100%, reduce = 39%, Cumulative CPU 558.94 sec
  218. 2016-08-12 19:34:23,194 Stage-5 map = 100%, reduce = 41%, Cumulative CPU 565.28 sec
  219. 2016-08-12 19:34:24,272 Stage-5 map = 100%, reduce = 46%, Cumulative CPU 583.96 sec
  220. 2016-08-12 19:34:25,341 Stage-5 map = 100%, reduce = 47%, Cumulative CPU 590.27 sec
  221. 2016-08-12 19:34:26,408 Stage-5 map = 100%, reduce = 50%, Cumulative CPU 599.89 sec
  222. 2016-08-12 19:34:27,475 Stage-5 map = 100%, reduce = 54%, Cumulative CPU 615.06 sec
  223. 2016-08-12 19:34:28,552 Stage-5 map = 100%, reduce = 58%, Cumulative CPU 627.21 sec
  224. 2016-08-12 19:34:29,626 Stage-5 map = 100%, reduce = 60%, Cumulative CPU 636.3 sec
  225. 2016-08-12 19:34:30,690 Stage-5 map = 100%, reduce = 63%, Cumulative CPU 648.52 sec
  226. 2016-08-12 19:34:31,757 Stage-5 map = 100%, reduce = 65%, Cumulative CPU 659.35 sec
  227. 2016-08-12 19:34:32,824 Stage-5 map = 100%, reduce = 66%, Cumulative CPU 673.33 sec
  228. 2016-08-12 19:34:33,897 Stage-5 map = 100%, reduce = 67%, Cumulative CPU 699.63 sec
  229. 2016-08-12 19:34:34,965 Stage-5 map = 100%, reduce = 68%, Cumulative CPU 722.55 sec
  230. 2016-08-12 19:34:36,030 Stage-5 map = 100%, reduce = 71%, Cumulative CPU 754.03 sec
  231. 2016-08-12 19:34:37,098 Stage-5 map = 100%, reduce = 73%, Cumulative CPU 760.98 sec
  232. 2016-08-12 19:34:38,164 Stage-5 map = 100%, reduce = 75%, Cumulative CPU 767.18 sec
  233. 2016-08-12 19:34:39,239 Stage-5 map = 100%, reduce = 80%, Cumulative CPU 789.09 sec
  234. 2016-08-12 19:34:40,310 Stage-5 map = 100%, reduce = 82%, Cumulative CPU 795.42 sec
  235. 2016-08-12 19:34:41,379 Stage-5 map = 100%, reduce = 86%, Cumulative CPU 804.74 sec
  236. 2016-08-12 19:34:42,447 Stage-5 map = 100%, reduce = 90%, Cumulative CPU 819.99 sec
  237. 2016-08-12 19:34:43,521 Stage-5 map = 100%, reduce = 94%, Cumulative CPU 832.86 sec
  238. 2016-08-12 19:34:45,664 Stage-5 map = 100%, reduce = 97%, Cumulative CPU 842.42 sec
  239. 2016-08-12 19:34:48,876 Stage-5 map = 100%, reduce = 99%, Cumulative CPU 848.86 sec
  240. 2016-08-12 19:34:49,947 Stage-5 map = 100%, reduce = 100%, Cumulative CPU 850.51 sec
  241. MapReduce Total cumulative CPU time: 14 minutes 10 seconds 510 msec
  242. Ended Job = job_1470994648796_0014
  243. Stage-23 is filtered out by condition resolver.
  244. Stage-24 is filtered out by condition resolver.
  245. Stage-1 is selected by condition resolver.
  246. Launching Job 7 out of 15
  247. Number of reduce tasks not specified. Estimated from input data size: 2
  248. In order to change the average load for a reducer (in bytes):
  249. set hive.exec.reducers.bytes.per.reducer=<number>
  250. In order to limit the maximum number of reducers:
  251. set hive.exec.reducers.max=<number>
  252. In order to set a constant number of reducers:
  253. set mapreduce.job.reduces=<number>
  254. 16/08/12 19:34:54 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  255. 16/08/12 19:34:54 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  256. 16/08/12 19:34:54 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  257. 16/08/12 19:36:42 INFO input.FileInputFormat: Total input paths to process : 11
  258. 16/08/12 19:36:42 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 0
  259. 16/08/12 19:36:42 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
  260. 16/08/12 19:36:44 INFO mapreduce.JobSubmitter: number of splits:8
  261. 16/08/12 19:36:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0015
  262. 16/08/12 19:36:45 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0015
  263. 16/08/12 19:36:45 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0015/
  264. Starting Job = job_1470994648796_0015, Tracking URL = http://node643:8088/proxy/application_1470994648796_0015/
  265. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0015
  266. Hadoop job information for Stage-1: number of mappers: 8; number of reducers: 2
  267. 16/08/12 19:37:16 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  268. 2016-08-12 19:37:16,292 Stage-1 map = 0%, reduce = 0%
  269. 2016-08-12 19:37:35,169 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 14.56 sec
  270. 2016-08-12 19:37:41,597 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 53.87 sec
  271. 2016-08-12 19:37:43,736 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 89.11 sec
  272. 2016-08-12 19:37:44,804 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 113.89 sec
  273. 2016-08-12 19:37:45,879 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 131.11 sec
  274. 2016-08-12 19:37:48,017 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 143.2 sec
  275. 2016-08-12 19:37:49,088 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 148.04 sec
  276. 2016-08-12 19:37:50,165 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 150.29 sec
  277. 2016-08-12 19:37:53,395 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 154.92 sec
  278. 2016-08-12 19:38:07,284 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 160.38 sec
  279. 2016-08-12 19:38:10,486 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 163.47 sec
  280. 2016-08-12 19:38:11,551 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 170.58 sec
  281. 2016-08-12 19:38:13,687 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 176.84 sec
  282. 2016-08-12 19:38:15,835 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 180.33 sec
  283. 2016-08-12 19:38:16,910 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 183.4 sec
  284. 2016-08-12 19:38:19,049 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 191.22 sec
  285. 2016-08-12 19:38:20,117 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 197.44 sec
  286. 2016-08-12 19:38:22,261 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 200.83 sec
  287. 2016-08-12 19:38:23,328 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 207.33 sec
  288. 2016-08-12 19:38:25,469 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 210.45 sec
  289. 2016-08-12 19:38:26,535 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 213.69 sec
  290. 2016-08-12 19:38:27,599 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 215.77 sec
  291. 2016-08-12 19:38:29,744 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 219.08 sec
  292. MapReduce Total cumulative CPU time: 3 minutes 39 seconds 80 msec
  293. Ended Job = job_1470994648796_0015
  294. Launching Job 8 out of 15
  295. Number of reduce tasks not specified. Estimated from input data size: 1
  296. In order to change the average load for a reducer (in bytes):
  297. set hive.exec.reducers.bytes.per.reducer=<number>
  298. In order to limit the maximum number of reducers:
  299. set hive.exec.reducers.max=<number>
  300. In order to set a constant number of reducers:
  301. set mapreduce.job.reduces=<number>
  302. 16/08/12 19:38:33 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  303. 16/08/12 19:38:33 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  304. 16/08/12 19:38:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  305. 16/08/12 19:40:18 INFO input.FileInputFormat: Total input paths to process : 2
  306. 16/08/12 19:40:18 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 4, size left: 0
  307. 16/08/12 19:40:19 INFO mapreduce.JobSubmitter: number of splits:2
  308. 16/08/12 19:40:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0016
  309. 16/08/12 19:40:20 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0016
  310. 16/08/12 19:40:20 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0016/
  311. Starting Job = job_1470994648796_0016, Tracking URL = http://node643:8088/proxy/application_1470994648796_0016/
  312. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0016
  313. Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
  314. 16/08/12 19:40:44 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  315. 2016-08-12 19:40:44,851 Stage-2 map = 0%, reduce = 0%
  316. 2016-08-12 19:40:54,795 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 3.07 sec
  317. 2016-08-12 19:41:07,683 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 6.15 sec
  318. 2016-08-12 19:41:23,737 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 10.58 sec
  319. MapReduce Total cumulative CPU time: 10 seconds 580 msec
  320. Ended Job = job_1470994648796_0016
  321. Launching Job 9 out of 15
  322. Number of reduce tasks determined at compile time: 1
  323. In order to change the average load for a reducer (in bytes):
  324. set hive.exec.reducers.bytes.per.reducer=<number>
  325. In order to limit the maximum number of reducers:
  326. set hive.exec.reducers.max=<number>
  327. In order to set a constant number of reducers:
  328. set mapreduce.job.reduces=<number>
  329. 16/08/12 19:41:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  330. 16/08/12 19:41:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  331. 16/08/12 19:41:28 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  332. 16/08/12 19:43:12 INFO input.FileInputFormat: Total input paths to process : 1
  333. 16/08/12 19:43:12 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
  334. 16/08/12 19:43:13 INFO mapreduce.JobSubmitter: number of splits:1
  335. 16/08/12 19:43:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0017
  336. 16/08/12 19:43:14 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0017
  337. 16/08/12 19:43:14 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0017/
  338. Starting Job = job_1470994648796_0017, Tracking URL = http://node643:8088/proxy/application_1470994648796_0017/
  339. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0017
  340. Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
  341. 16/08/12 19:43:39 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  342. 2016-08-12 19:43:39,872 Stage-3 map = 0%, reduce = 0%
  343. 2016-08-12 19:44:04,031 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 3.2 sec
  344. 2016-08-12 19:44:26,559 Stage-3 map = 100%, reduce = 100%, Cumulative CPU 8.11 sec
  345. MapReduce Total cumulative CPU time: 8 seconds 110 msec
  346. Ended Job = job_1470994648796_0017
  347. Loading data to table default.q5_local_supplier_volume
  348. 16/08/12 19:44:30 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
  349. 16/08/12 19:44:30 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
  350. MapReduce Jobs Launched:
  351. Stage-Stage-22: Map: 1 Cumulative CPU: 6.64 sec HDFS Read: 7979 HDFS Write: 222 SUCCESS
  352. Stage-Stage-19: Map: 1 Cumulative CPU: 9.47 sec HDFS Read: 14182336 HDFS Write: 577426 SUCCESS
  353. Stage-Stage-16: Map: 29 Cumulative CPU: 698.01 sec HDFS Read: 7776171787 HDFS Write: 557334620 SUCCESS
  354. Stage-Stage-5: Map: 15 Reduce: 10 Cumulative CPU: 850.51 sec HDFS Read: 2306747760 HDFS Write: 83207781 SUCCESS
  355. Stage-Stage-1: Map: 8 Reduce: 2 Cumulative CPU: 219.08 sec HDFS Read: 328125221 HDFS Write: 514 SUCCESS
  356. Stage-Stage-2: Map: 2 Reduce: 1 Cumulative CPU: 10.58 sec HDFS Read: 7927 HDFS Write: 257 SUCCESS
  357. Stage-Stage-3: Map: 1 Reduce: 1 Cumulative CPU: 8.11 sec HDFS Read: 6074 HDFS Write: 223 SUCCESS
  358. Total MapReduce CPU Time Spent: 30 minutes 2 seconds 400 msec
  359. OK
  360. Time taken: 1446.725 seconds

    ——set hive.execution.engine=tez

    备注:在.hive文件里设置了为tez调度模式。

  1. lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q5_local_supplier_volume.hive
  2. which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
  3. SLF4J: Class path contains multiple SLF4J bindings.
  4. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  5. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  6. SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  7. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  8. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  9. 16/08/12 19:09:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  10. Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
  11. 16/08/12 19:09:31 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
  12. 16/08/12 19:09:39 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
  13. 16/08/12 19:09:39 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
  14. Query ID = lwh_20160812190931_c82a04f5-da59-4eb2-a814-4f295ace94f1
  15. Total jobs = 1
  16. Launching Job 1 out of 1
  17. 16/08/12 19:09:39 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
  18. 16/08/12 19:11:01 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  19. 16/08/12 19:11:02 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0010
  20. 16/08/12 19:11:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  21. Status: Running (Executing on YARN cluster with App id application_1470994648796_0010)
  22. ----------------------------------------------------------------------------------------------
  23. VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
  24. ----------------------------------------------------------------------------------------------
  25. Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
  26. Map 10 ......... container SUCCEEDED 1 1 0 0 0 0
  27. Map 11 ......... container SUCCEEDED 1 1 0 0 0 0
  28. Map 5 .......... container SUCCEEDED 4 4 0 0 0 0
  29. Map 7 .......... container SUCCEEDED 15 15 0 0 0 0
  30. Map 9 .......... container SUCCEEDED 1 1 0 0 0 0
  31. Reducer 2 ...... container SUCCEEDED 38 38 0 0 0 0
  32. Reducer 3 ...... container SUCCEEDED 41 41 0 0 0 0
  33. Reducer 4 ...... container SUCCEEDED 1 1 0 0 0 0
  34. Reducer 6 ...... container SUCCEEDED 35 35 0 0 0 0
  35. Reducer 8 ...... container SUCCEEDED 31 31 0 0 0 0
  36. ----------------------------------------------------------------------------------------------
  37. VERTICES: 11/11 [==========================>>] 100% ELAPSED TIME: 218.63 s
  38. ----------------------------------------------------------------------------------------------
  39. Loading data to table default.q5_local_supplier_volume
  40. 16/08/12 19:15:08 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
  41. OK
  42. Time taken: 339.425 seconds
 Hive-on-MRHive-on-Tez
q1436.074s189.653s
q51446.725s339.425s

 

②并行查询q1+q2

    ——hive.execution.engine=tez

    q1:

  1. lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q1_pricing_summary_report.hive
  2. which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
  3. SLF4J: Class path contains multiple SLF4J bindings.
  4. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  5. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  6. SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  7. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  8. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  9. 16/08/12 19:56:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  10. Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
  11. 16/08/12 19:56:08 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
  12. 16/08/12 19:56:17 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
  13. 16/08/12 19:56:17 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
  14. Query ID = lwh_20160812195608_c1cde9c4-c451-4230-8515-92f1be8758ee
  15. Total jobs = 1
  16. Launching Job 1 out of 1
  17. 16/08/12 19:56:17 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
  18. 16/08/12 19:58:51 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  19. 16/08/12 19:58:53 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0018
  20. 16/08/12 19:59:22 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  21. Status: Running (Executing on YARN cluster with App id application_1470994648796_0018)
  22. ----------------------------------------------------------------------------------------------
  23. VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
  24. ----------------------------------------------------------------------------------------------
  25. Map 1 .......... container SUCCEEDED 15 15 0 0 0 0
  26. Reducer 2 ...... container SUCCEEDED 11 11 0 0 0 0
  27. Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0
  28. ----------------------------------------------------------------------------------------------
  29. VERTICES: 03/03 [==========================>>] 100% ELAPSED TIME: 136.81 s
  30. ----------------------------------------------------------------------------------------------
  31. OK
  32. 16/08/12 20:01:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  33. 16/08/12 20:01:41 INFO mapred.FileInputFormat: Total input paths to process : 1
  34. A F 3.77518399E8 5.660657277972496E11 5.377591042780614E11 5.592766708921191E11 25.500975103007097 38237.15100895852 0.05000657454011058 14804077
  35. N F 9851614.0 1.4767438399170017E10 1.4028805792211395E10 1.459049099836676E10 25.522448302840946 38257.81066008119 0.04997336773764426 385998
  36. N O 7.43124873E8 1.1143022869018733E12 1.0585809221449624E12 1.1009370001705708E12 25.498075870689316 38233.90292348158 0.050000811821093484 29144351
  37. R F 3.7773283E8 5.664310549759983E11 5.381109226647607E11 5.596347808850894E11 25.50838478968014 38251.21927355965 0.04999679231395468 14808183
  38. Time taken: 332.832 seconds, Fetched: 4 row(s)

    q2:

输出信息过多,没有收集到。

    ——hive.execution.engine=tez

    q1:

  1. lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q1_pricing_summary_report.hive
  2. which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
  3. SLF4J: Class path contains multiple SLF4J bindings.
  4. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  5. SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  6. SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  7. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  8. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  9. 16/08/12 20:16:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  10. Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
  11. Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  12. 16/08/12 20:16:27 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
  13. 16/08/12 20:16:32 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
  14. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  15. Query ID = lwh_20160812201627_2ccd2f24-ad79-466b-8ad9-ff0f7b7cd191
  16. Total jobs = 2
  17. Launching Job 1 out of 2
  18. Number of reduce tasks not specified. Estimated from input data size: 31
  19. In order to change the average load for a reducer (in bytes):
  20. set hive.exec.reducers.bytes.per.reducer=<number>
  21. In order to limit the maximum number of reducers:
  22. set hive.exec.reducers.max=<number>
  23. In order to set a constant number of reducers:
  24. set mapreduce.job.reduces=<number>
  25. 16/08/12 20:16:37 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
  26. 16/08/12 20:16:38 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  27. 16/08/12 20:16:39 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  28. 16/08/12 20:16:40 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  29. 16/08/12 20:19:45 INFO input.FileInputFormat: Total input paths to process : 1
  30. 16/08/12 20:19:45 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 268435456
  31. 16/08/12 20:19:47 INFO mapreduce.JobSubmitter: number of splits:29
  32. 16/08/12 20:19:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0022
  33. 16/08/12 20:19:49 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0022
  34. 16/08/12 20:19:49 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0022/
  35. Starting Job = job_1470994648796_0022, Tracking URL = http://node643:8088/proxy/application_1470994648796_0022/
  36. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0022
  37. Hadoop job information for Stage-1: number of mappers: 29; number of reducers: 31
  38. 16/08/12 20:20:16 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  39. 2016-08-12 20:20:16,607 Stage-1 map = 0%, reduce = 0%
  40. 2016-08-12 20:20:40,816 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 34.83 sec
  41. 2016-08-12 20:20:47,294 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 170.63 sec
  42. 2016-08-12 20:20:53,777 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 290.64 sec
  43. 2016-08-12 20:20:55,943 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 313.98 sec
  44. 2016-08-12 20:20:57,024 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 335.71 sec
  45. 2016-08-12 20:21:00,260 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 395.76 sec
  46. 2016-08-12 20:21:01,334 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 407.72 sec
  47. 2016-08-12 20:21:02,413 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 425.55 sec
  48. 2016-08-12 20:21:03,498 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 433.11 sec
  49. 2016-08-12 20:21:08,897 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 456.3 sec
  50. 2016-08-12 20:21:09,970 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 462.23 sec
  51. 2016-08-12 20:21:12,115 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 467.86 sec
  52. 2016-08-12 20:21:13,187 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 482.36 sec
  53. 2016-08-12 20:21:15,356 Stage-1 map = 49%, reduce = 1%, Cumulative CPU 522.83 sec
  54. 2016-08-12 20:21:16,430 Stage-1 map = 52%, reduce = 1%, Cumulative CPU 528.38 sec
  55. 2016-08-12 20:21:21,816 Stage-1 map = 53%, reduce = 1%, Cumulative CPU 603.66 sec
  56. 2016-08-12 20:21:22,889 Stage-1 map = 54%, reduce = 1%, Cumulative CPU 616.1 sec
  57. 2016-08-12 20:21:23,962 Stage-1 map = 56%, reduce = 1%, Cumulative CPU 626.38 sec
  58. 2016-08-12 20:21:27,195 Stage-1 map = 57%, reduce = 1%, Cumulative CPU 665.39 sec
  59. 2016-08-12 20:21:28,270 Stage-1 map = 59%, reduce = 1%, Cumulative CPU 677.06 sec
  60. 2016-08-12 20:21:29,346 Stage-1 map = 60%, reduce = 2%, Cumulative CPU 707.06 sec
  61. 2016-08-12 20:21:31,495 Stage-1 map = 67%, reduce = 2%, Cumulative CPU 733.66 sec
  62. 2016-08-12 20:21:33,657 Stage-1 map = 72%, reduce = 2%, Cumulative CPU 753.21 sec
  63. 2016-08-12 20:21:36,878 Stage-1 map = 77%, reduce = 2%, Cumulative CPU 773.63 sec
  64. 2016-08-12 20:21:37,945 Stage-1 map = 78%, reduce = 2%, Cumulative CPU 777.17 sec
  65. 2016-08-12 20:21:39,026 Stage-1 map = 79%, reduce = 2%, Cumulative CPU 780.76 sec
  66. 2016-08-12 20:21:40,103 Stage-1 map = 82%, reduce = 2%, Cumulative CPU 783.92 sec
  67. 2016-08-12 20:21:42,241 Stage-1 map = 82%, reduce = 3%, Cumulative CPU 790.95 sec
  68. 2016-08-12 20:21:44,376 Stage-1 map = 84%, reduce = 3%, Cumulative CPU 795.82 sec
  69. 2016-08-12 20:21:46,535 Stage-1 map = 86%, reduce = 4%, Cumulative CPU 835.79 sec
  70. 2016-08-12 20:21:48,681 Stage-1 map = 86%, reduce = 5%, Cumulative CPU 851.77 sec
  71. 2016-08-12 20:21:49,756 Stage-1 map = 86%, reduce = 6%, Cumulative CPU 861.69 sec
  72. 2016-08-12 20:21:55,102 Stage-1 map = 87%, reduce = 6%, Cumulative CPU 882.42 sec
  73. 2016-08-12 20:21:56,174 Stage-1 map = 89%, reduce = 8%, Cumulative CPU 888.73 sec
  74. 2016-08-12 20:21:57,245 Stage-1 map = 89%, reduce = 9%, Cumulative CPU 890.14 sec
  75. 2016-08-12 20:22:00,466 Stage-1 map = 89%, reduce = 10%, Cumulative CPU 903.81 sec
  76. 2016-08-12 20:22:02,603 Stage-1 map = 91%, reduce = 10%, Cumulative CPU 914.72 sec
  77. 2016-08-12 20:22:04,740 Stage-1 map = 93%, reduce = 11%, Cumulative CPU 920.01 sec
  78. 2016-08-12 20:22:07,945 Stage-1 map = 94%, reduce = 11%, Cumulative CPU 924.0 sec
  79. 2016-08-12 20:22:10,087 Stage-1 map = 95%, reduce = 11%, Cumulative CPU 925.81 sec
  80. 2016-08-12 20:22:14,371 Stage-1 map = 95%, reduce = 12%, Cumulative CPU 932.02 sec
  81. 2016-08-12 20:22:16,506 Stage-1 map = 95%, reduce = 13%, Cumulative CPU 936.41 sec
  82. 2016-08-12 20:22:28,287 Stage-1 map = 98%, reduce = 13%, Cumulative CPU 950.75 sec
  83. 2016-08-12 20:22:29,355 Stage-1 map = 100%, reduce = 13%, Cumulative CPU 952.36 sec
  84. 2016-08-12 20:22:30,424 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 954.69 sec
  85. 2016-08-12 20:22:31,492 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 963.74 sec
  86. 2016-08-12 20:22:32,562 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 980.79 sec
  87. 2016-08-12 20:22:33,632 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 1017.66 sec
  88. 2016-08-12 20:22:34,704 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 1023.84 sec
  89. 2016-08-12 20:22:38,973 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 1031.16 sec
  90. 2016-08-12 20:22:40,041 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 1036.93 sec
  91. 2016-08-12 20:22:41,118 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 1042.65 sec
  92. 2016-08-12 20:22:43,278 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1049.32 sec
  93. 2016-08-12 20:22:44,355 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1077.92 sec
  94. 2016-08-12 20:22:45,432 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1099.9 sec
  95. 2016-08-12 20:22:46,501 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1105.43 sec
  96. 2016-08-12 20:22:49,707 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 1111.84 sec
  97. 2016-08-12 20:22:50,778 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1124.73 sec
  98. 2016-08-12 20:22:51,847 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1131.81 sec
  99. 2016-08-12 20:22:55,050 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1139.38 sec
  100. MapReduce Total cumulative CPU time: 18 minutes 59 seconds 380 msec
  101. Ended Job = job_1470994648796_0022
  102. Launching Job 2 out of 2
  103. Number of reduce tasks determined at compile time: 1
  104. In order to change the average load for a reducer (in bytes):
  105. set hive.exec.reducers.bytes.per.reducer=<number>
  106. In order to limit the maximum number of reducers:
  107. set hive.exec.reducers.max=<number>
  108. In order to set a constant number of reducers:
  109. set mapreduce.job.reduces=<number>
  110. 16/08/12 20:23:03 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  111. 16/08/12 20:23:03 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
  112. 16/08/12 20:23:03 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  113. 16/08/12 20:25:38 INFO input.FileInputFormat: Total input paths to process : 31
  114. 16/08/12 20:25:38 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
  115. 16/08/12 20:25:40 INFO mapreduce.JobSubmitter: number of splits:6
  116. 16/08/12 20:25:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0025
  117. 16/08/12 20:25:41 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0025
  118. 16/08/12 20:25:41 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0025/
  119. Starting Job = job_1470994648796_0025, Tracking URL = http://node643:8088/proxy/application_1470994648796_0025/
  120. Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0025
  121. Hadoop job information for Stage-2: number of mappers: 6; number of reducers: 1
  122. 16/08/12 20:26:06 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
  123. 2016-08-12 20:26:06,419 Stage-2 map = 0%, reduce = 0%
  124. 2016-08-12 20:26:16,449 Stage-2 map = 17%, reduce = 0%, Cumulative CPU 3.61 sec
  125. 2016-08-12 20:26:28,243 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 7.08 sec
  126. 2016-08-12 20:26:29,314 Stage-2 map = 67%, reduce = 0%, Cumulative CPU 14.63 sec
  127. 2016-08-12 20:26:32,526 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 21.77 sec
  128. 2016-08-12 20:26:35,737 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 27.16 sec
  129. MapReduce Total cumulative CPU time: 27 seconds 160 msec
  130. Ended Job = job_1470994648796_0025
  131. MapReduce Jobs Launched:
  132. Stage-Stage-1: Map: 29 Reduce: 31 Cumulative CPU: 1139.38 sec HDFS Read: 7776522714 HDFS Write: 3305 SUCCESS
  133. Stage-Stage-2: Map: 6 Reduce: 1 Cumulative CPU: 27.16 sec HDFS Read: 39220 HDFS Write: 723 SUCCESS
  134. Total MapReduce CPU Time Spent: 19 minutes 26 seconds 540 msec
  135. OK
  136. 16/08/12 20:26:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  137. 16/08/12 20:26:41 INFO mapred.FileInputFormat: Total input paths to process : 1
  138. A F 3.77518399E8 5.660657277972468E11 5.3775910427806824E11 5.592766708921185E11 25.500975103007097 38237.15100895833 0.05000657454026939 14804077
  139. N F 9851614.0 1.4767438399169981E10 1.4028805792211395E10 1.4590490998366728E10 25.522448302840946 38257.81066008109 0.04997336773765435 385998
  140. N O 7.43124873E8 1.1143022869018765E12 1.0585809221449629E12 1.100937000170583E12 25.498075870689316 38233.90292348169 0.050000811820997394 29144351
  141. R F 3.7773283E8 5.664310549759971E11 5.381109226647635E11 5.596347808850863E11 25.50838478968014 38251.21927355956 0.04999679231411408 14808183
  142. Time taken: 614.032 seconds, Fetched: 4 row(s)

    q2:

Time taken: 968.034 seconds, Fetched: 6351 row(s)
 Hive-on-MRHive-on-Tez
q1614.032s332.832s
q2968.034s/
total968.034s332.832s

 

转载于:https://my.oschina.net/whulyx/blog/731940

本文内容由网友自发贡献,转载请注明出处:https://www.wpsshop.cn/w/喵喵爱编程/article/detail/833845
推荐阅读
相关标签
  

闽ICP备14008679号