实验数据:10G大小的TPCH数据(8个表);
实验环境:1Master节点+8Slave节点,每个均为4core,4G内存,30G磁盘,磁盘读写速度50M/s;
①单个查询之q1_pricing_summary_report.hive
- SELECT
- L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY), SUM(L_EXTENDEDPRICE), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)), AVG(L_QUANTITY), AVG(L_EXTENDEDPRICE), AVG(L_DISCOUNT), COUNT(1)
- FROM
- lineitem
- WHERE
- L_SHIPDATE<='1998-09-02'
- GROUP BY L_RETURNFLAG, L_LINESTATUS
- ORDER BY L_RETURNFLAG, L_LINESTATUS;
——set hive.execution.engine=mr
- hive> SELECT
- > L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY), SUM(L_EXTENDEDPRICE), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)), AVG(L_QUANTITY), AVG(L_EXTENDEDPRICE), AVG(L_DISCOUNT), COUNT(1)
- > FROM
- > lineitem
- > WHERE
- > L_SHIPDATE<='1998-09-02'
- > GROUP BY L_RETURNFLAG, L_LINESTATUS
- > ORDER BY L_RETURNFLAG, L_LINESTATUS;
- 16/08/12 18:19:57 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
- WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
- Query ID = lwh_20160812181955_0ba24dd7-25a8-44c6-9005-f5adacfc1b08
- Total jobs = 2
- Launching Job 1 out of 2
- Number of reduce tasks not specified. Estimated from input data size: 31
- In order to change the average load for a reducer (in bytes):
- set hive.exec.reducers.bytes.per.reducer=<number>
- In order to limit the maximum number of reducers:
- set hive.exec.reducers.max=<number>
- In order to set a constant number of reducers:
- set mapreduce.job.reduces=<number>
- 16/08/12 18:20:02 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
- 16/08/12 18:20:03 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 18:20:04 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 18:20:04 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 18:21:48 INFO input.FileInputFormat: Total input paths to process : 1
- 16/08/12 18:21:48 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 268435456
- 16/08/12 18:21:49 INFO mapreduce.JobSubmitter: number of splits:29
- 16/08/12 18:21:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0007
- 16/08/12 18:21:50 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0007
- 16/08/12 18:21:50 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0007/
- Starting Job = job_1470994648796_0007, Tracking URL = http://node643:8088/proxy/application_1470994648796_0007/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0007
- Hadoop job information for Stage-1: number of mappers: 29; number of reducers: 31
- 16/08/12 18:22:15 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 18:22:15,564 Stage-1 map = 0%, reduce = 0%
- 2016-08-12 18:22:40,147 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 40.74 sec
- 2016-08-12 18:22:45,548 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 189.74 sec
- 2016-08-12 18:22:46,625 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 211.75 sec
- 2016-08-12 18:22:50,940 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 298.17 sec
- 2016-08-12 18:22:52,019 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 306.72 sec
- 2016-08-12 18:22:53,097 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 313.03 sec
- 2016-08-12 18:22:54,173 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 341.25 sec
- 2016-08-12 18:22:58,499 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 408.43 sec
- 2016-08-12 18:23:00,651 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 458.76 sec
- 2016-08-12 18:23:01,724 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 463.35 sec
- 2016-08-12 18:23:03,873 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 474.89 sec
- 2016-08-12 18:23:09,249 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 485.99 sec
- 2016-08-12 18:23:11,405 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 492.84 sec
- 2016-08-12 18:23:12,498 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 509.8 sec
- 2016-08-12 18:23:14,676 Stage-1 map = 53%, reduce = 1%, Cumulative CPU 533.6 sec
- 2016-08-12 18:23:16,828 Stage-1 map = 55%, reduce = 1%, Cumulative CPU 598.25 sec
- 2016-08-12 18:23:20,048 Stage-1 map = 55%, reduce = 2%, Cumulative CPU 636.57 sec
- 2016-08-12 18:23:21,134 Stage-1 map = 56%, reduce = 2%, Cumulative CPU 649.37 sec
- 2016-08-12 18:23:23,298 Stage-1 map = 57%, reduce = 2%, Cumulative CPU 669.42 sec
- 2016-08-12 18:23:25,443 Stage-1 map = 61%, reduce = 2%, Cumulative CPU 689.89 sec
- 2016-08-12 18:23:26,531 Stage-1 map = 62%, reduce = 2%, Cumulative CPU 697.35 sec
- 2016-08-12 18:23:27,602 Stage-1 map = 64%, reduce = 2%, Cumulative CPU 715.54 sec
- 2016-08-12 18:23:28,671 Stage-1 map = 67%, reduce = 2%, Cumulative CPU 728.59 sec
- 2016-08-12 18:23:31,881 Stage-1 map = 74%, reduce = 2%, Cumulative CPU 777.91 sec
- 2016-08-12 18:23:32,951 Stage-1 map = 83%, reduce = 3%, Cumulative CPU 791.75 sec
- 2016-08-12 18:23:36,158 Stage-1 map = 83%, reduce = 4%, Cumulative CPU 809.7 sec
- 2016-08-12 18:23:39,366 Stage-1 map = 84%, reduce = 4%, Cumulative CPU 819.73 sec
- 2016-08-12 18:23:40,450 Stage-1 map = 85%, reduce = 4%, Cumulative CPU 822.8 sec
- 2016-08-12 18:23:42,587 Stage-1 map = 86%, reduce = 4%, Cumulative CPU 830.3 sec
- 2016-08-12 18:23:45,806 Stage-1 map = 89%, reduce = 4%, Cumulative CPU 865.81 sec
- 2016-08-12 18:23:46,880 Stage-1 map = 89%, reduce = 7%, Cumulative CPU 872.51 sec
- 2016-08-12 18:23:47,951 Stage-1 map = 89%, reduce = 8%, Cumulative CPU 883.72 sec
- 2016-08-12 18:23:49,021 Stage-1 map = 91%, reduce = 9%, Cumulative CPU 891.21 sec
- 2016-08-12 18:23:50,097 Stage-1 map = 93%, reduce = 9%, Cumulative CPU 893.01 sec
- 2016-08-12 18:23:52,247 Stage-1 map = 93%, reduce = 10%, Cumulative CPU 900.14 sec
- 2016-08-12 18:23:53,317 Stage-1 map = 95%, reduce = 10%, Cumulative CPU 908.03 sec
- 2016-08-12 18:23:57,624 Stage-1 map = 95%, reduce = 11%, Cumulative CPU 916.72 sec
- 2016-08-12 18:24:00,843 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 926.87 sec
- 2016-08-12 18:24:01,916 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 930.05 sec
- 2016-08-12 18:24:02,988 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 942.11 sec
- 2016-08-12 18:24:04,062 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 962.53 sec
- 2016-08-12 18:24:05,135 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 993.55 sec
- 2016-08-12 18:24:07,282 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 1001.0 sec
- 2016-08-12 18:24:11,593 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 1015.47 sec
- 2016-08-12 18:24:13,733 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 1020.94 sec
- 2016-08-12 18:24:14,800 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 1047.42 sec
- 2016-08-12 18:24:15,868 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 1073.01 sec
- 2016-08-12 18:24:16,936 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1083.68 sec
- 2016-08-12 18:24:20,139 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 1088.29 sec
- 2016-08-12 18:24:21,205 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1100.29 sec
- 2016-08-12 18:24:23,354 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1112.48 sec
- MapReduce Total cumulative CPU time: 18 minutes 32 seconds 480 msec
- Ended Job = job_1470994648796_0007
- Launching Job 2 out of 2
- Number of reduce tasks determined at compile time: 1
- In order to change the average load for a reducer (in bytes):
- set hive.exec.reducers.bytes.per.reducer=<number>
- In order to limit the maximum number of reducers:
- set hive.exec.reducers.max=<number>
- In order to set a constant number of reducers:
- set mapreduce.job.reduces=<number>
- 16/08/12 18:24:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 18:24:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 18:24:28 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 18:26:10 INFO input.FileInputFormat: Total input paths to process : 31
- 16/08/12 18:26:10 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
- 16/08/12 18:26:11 INFO mapreduce.JobSubmitter: number of splits:6
- 16/08/12 18:26:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0008
- 16/08/12 18:26:12 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0008
- 16/08/12 18:26:12 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0008/
- Starting Job = job_1470994648796_0008, Tracking URL = http://node643:8088/proxy/application_1470994648796_0008/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0008
- Hadoop job information for Stage-2: number of mappers: 6; number of reducers: 1
- 16/08/12 18:26:35 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 18:26:35,127 Stage-2 map = 0%, reduce = 0%
- 2016-08-12 18:26:58,048 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 11.12 sec
- 2016-08-12 18:27:00,187 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 21.08 sec
- 2016-08-12 18:27:07,672 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 26.1 sec
- MapReduce Total cumulative CPU time: 26 seconds 100 msec
- Ended Job = job_1470994648796_0008
- MapReduce Jobs Launched:
- Stage-Stage-1: Map: 29 Reduce: 31 Cumulative CPU: 1112.48 sec HDFS Read: 7776522714 HDFS Write: 3305 SUCCESS
- Stage-Stage-2: Map: 6 Reduce: 1 Cumulative CPU: 26.1 sec HDFS Read: 39220 HDFS Write: 720 SUCCESS
- Total MapReduce CPU Time Spent: 18 minutes 58 seconds 580 msec
- OK
- 16/08/12 18:27:11 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
- 16/08/12 18:27:11 INFO mapred.FileInputFormat: Total input paths to process : 1
- A F 3.77518399E8 5.660657277972468E11 5.377591042780681E11 5.592766708921187E11 25.500975103007097 38237.15100895833 0.050006574540269355 14804077
- N F 9851614.0 1.4767438399169975E10 1.4028805792211395E10 1.459049099836673E10 25.522448302840946 38257.81066008108 0.04997336773765436 385998
- N O 7.43124873E8 1.1143022869018765E12 1.0585809221449628E12 1.100937000170583E12 25.498075870689316 38233.90292348169 0.0500008118209974 29144351
- R F 3.7773283E8 5.664310549759971E11 5.3811092266476355E11 5.59634780885086E11 25.50838478968014 38251.21927355956 0.04999679231411408 14808183
- Time taken: 436.074 seconds, Fetched: 4 row(s)
——set hive.execution.engine=tez
- hive> set hive.execution.engine=tez;
- hive> SELECT
- > L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY), SUM(L_EXTENDEDPRICE), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)), AVG(L_QUANTITY), AVG(L_EXTENDEDPRICE), AVG(L_DISCOUNT), COUNT(1)
- > FROM
- > lineitem
- > WHERE
- > L_SHIPDATE<='1998-09-02'
- > GROUP BY L_RETURNFLAG, L_LINESTATUS
- > ORDER BY L_RETURNFLAG, L_LINESTATUS;
- 16/08/12 19:01:38 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
- 16/08/12 19:01:38 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
- Query ID = lwh_20160812190132_2f9c1fad-5f40-4ad0-99bf-57a921acf812
- Total jobs = 1
- Launching Job 1 out of 1
- 16/08/12 19:01:39 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
- 16/08/12 19:03:02 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:03:03 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0009
- 16/08/12 19:03:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
-
-
- Status: Running (Executing on YARN cluster with App id application_1470994648796_0009)
-
- ----------------------------------------------------------------------------------------------
- VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
- ----------------------------------------------------------------------------------------------
- Map 1 .......... container SUCCEEDED 58 58 0 0 0 0
- Reducer 2 ...... container SUCCEEDED 11 11 0 0 0 0
- Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0
- ----------------------------------------------------------------------------------------------
- VERTICES: 03/03 [==========================>>] 100% ELAPSED TIME: 72.51 s
- ----------------------------------------------------------------------------------------------
- OK
- 16/08/12 19:04:42 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
- 16/08/12 19:04:42 INFO mapred.FileInputFormat: Total input paths to process : 1
- A F 3.77518399E8 5.660657277972496E11 5.3775910427806433E11 5.592766708921171E11 25.500975103007097 38237.15100895852 0.05000657454031599 14804077
- N F 9851614.0 1.4767438399169992E10 1.4028805792211395E10 1.4590490998366734E10 25.522448302840946 38257.81066008112 0.049973367737656114 385998
- N O 7.43124873E8 1.1143022869018794E12 1.0585809221449642E12 1.1009370001705938E12 25.498075870689316 38233.902923481786 0.050000811821159556 29144351
- R F 3.7773283E8 5.664310549760012E11 5.3811092266476697E11 5.596347808850865E11 25.50838478968014 38251.219273559844 0.04999679231416023 14808183
- Time taken: 189.653 seconds, Fetched: 4 row(s)
②单个查询之q5_local_supplier_volume.hive
- select
- n_name, sum(l_extendedprice * (1 - l_discount)) as revenue
- from
- customer c join
- ( select n_name, l_extendedprice, l_discount, s_nationkey, o_custkey from orders o join
- ( select n_name, l_extendedprice, l_discount, l_orderkey, s_nationkey from lineitem l join
- ( select n_name, s_suppkey, s_nationkey from supplier s join
- ( select n_name, n_nationkey
- from nation n join region r
- on n.n_regionkey = r.r_regionkey and r.r_name = 'ASIA'
- ) n1 on s.s_nationkey = n1.n_nationkey
- ) s1 on l.l_suppkey = s1.s_suppkey
- ) l1 on l1.l_orderkey = o.o_orderkey and o.o_orderdate >= '1994-01-01'
- and o.o_orderdate < '1995-01-01'
- ) o1
- on c.c_nationkey = o1.s_nationkey and c.c_custkey = o1.o_custkey
- group by n_name
- order by revenue desc;
——set hive.execution.engine=mr
- lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q5_local_supplier_volume.hive
- which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
- 16/08/12 19:20:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-
- Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
- Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
- 16/08/12 19:20:25 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
- WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
- Query ID = lwh_20160812192025_499bf4fc-1185-495a-b02c-e81285fd7740
- Total jobs = 15
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
- 16/08/12 19:20:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- 2016-08-12 19:20:46 Starting to launch local task to process map join; maximum memory = 477626368
- 16/08/12 19:20:49 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
- 16/08/12 19:20:49 INFO mapred.FileInputFormat: Total input paths to process : 1
- 2016-08-12 19:20:49 Dump the side-table for tag: 1 with group count: 1 into file: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10024/HashTable-Stage-22/MapJoin-mapfile81--.hashtable
- 2016-08-12 19:20:49 Uploaded 1 File to: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10024/HashTable-Stage-22/MapJoin-mapfile81--.hashtable (278 bytes)
- 2016-08-12 19:20:49 End of local task; Time Taken: 3.023 sec.
- Execution completed successfully
- MapredLocal task succeeded
- Launching Job 1 out of 15
- Number of reduce tasks is set to 0 since there's no reduce operator
- 16/08/12 19:20:51 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
- 16/08/12 19:20:52 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:20:53 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:20:53 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 19:22:38 INFO input.FileInputFormat: Total input paths to process : 1
- 16/08/12 19:22:38 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
- 16/08/12 19:22:39 INFO mapreduce.JobSubmitter: number of splits:1
- 16/08/12 19:22:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0011
- 16/08/12 19:22:40 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0011
- 16/08/12 19:22:40 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0011/
- Starting Job = job_1470994648796_0011, Tracking URL = http://node643:8088/proxy/application_1470994648796_0011/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0011
- Hadoop job information for Stage-22: number of mappers: 1; number of reducers: 0
- 16/08/12 19:23:09 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 19:23:09,546 Stage-22 map = 0%, reduce = 0%
- 2016-08-12 19:23:37,474 Stage-22 map = 100%, reduce = 0%, Cumulative CPU 6.64 sec
- MapReduce Total cumulative CPU time: 6 seconds 640 msec
- Ended Job = job_1470994648796_0011
- Stage-29 is selected by condition resolver.
- Stage-30 is filtered out by condition resolver.
- Stage-7 is filtered out by condition resolver.
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
- 16/08/12 19:23:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- 2016-08-12 19:23:52 Starting to launch local task to process map join; maximum memory = 477626368
- 16/08/12 19:23:54 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
- 16/08/12 19:23:54 INFO mapred.FileInputFormat: Total input paths to process : 1
- 2016-08-12 19:23:54 Dump the side-table for tag: 1 with group count: 5 into file: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10020/HashTable-Stage-19/MapJoin-mapfile61--.hashtable
- 2016-08-12 19:23:54 Uploaded 1 File to: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10020/HashTable-Stage-19/MapJoin-mapfile61--.hashtable (391 bytes)
- 2016-08-12 19:23:54 End of local task; Time Taken: 2.383 sec.
- Execution completed successfully
- MapredLocal task succeeded
- Launching Job 3 out of 15
- Number of reduce tasks is set to 0 since there's no reduce operator
- 16/08/12 19:23:58 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:23:59 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:23:59 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 19:25:43 INFO input.FileInputFormat: Total input paths to process : 1
- 16/08/12 19:25:43 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
- 16/08/12 19:25:44 INFO mapreduce.JobSubmitter: number of splits:1
- 16/08/12 19:25:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0012
- 16/08/12 19:25:45 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0012
- 16/08/12 19:25:45 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0012/
- Starting Job = job_1470994648796_0012, Tracking URL = http://node643:8088/proxy/application_1470994648796_0012/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0012
- Hadoop job information for Stage-19: number of mappers: 1; number of reducers: 0
- 16/08/12 19:26:15 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 19:26:15,132 Stage-19 map = 0%, reduce = 0%
- 2016-08-12 19:26:42,840 Stage-19 map = 100%, reduce = 0%, Cumulative CPU 9.47 sec
- MapReduce Total cumulative CPU time: 9 seconds 470 msec
- Ended Job = job_1470994648796_0012
- Stage-27 is selected by condition resolver.
- Stage-28 is filtered out by condition resolver.
- Stage-6 is filtered out by condition resolver.
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
- 16/08/12 19:26:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- 2016-08-12 19:26:57 Starting to launch local task to process map join; maximum memory = 477626368
- 16/08/12 19:26:59 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
- 16/08/12 19:26:59 INFO mapred.FileInputFormat: Total input paths to process : 1
- 2016-08-12 19:27:00 Dump the side-table for tag: 1 with group count: 20037 into file: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10016/HashTable-Stage-16/MapJoin-mapfile41--.hashtable
- 2016-08-12 19:27:00 Uploaded 1 File to: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10016/HashTable-Stage-16/MapJoin-mapfile41--.hashtable (594855 bytes)
- 2016-08-12 19:27:00 End of local task; Time Taken: 2.755 sec.
- Execution completed successfully
- MapredLocal task succeeded
- Launching Job 5 out of 15
- Number of reduce tasks is set to 0 since there's no reduce operator
- 16/08/12 19:27:02 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:27:02 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:27:02 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 19:28:45 INFO input.FileInputFormat: Total input paths to process : 1
- 16/08/12 19:28:45 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 268435456
- 16/08/12 19:28:46 INFO mapreduce.JobSubmitter: number of splits:29
- 16/08/12 19:28:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0013
- 16/08/12 19:28:48 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0013
- 16/08/12 19:28:48 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0013/
- Starting Job = job_1470994648796_0013, Tracking URL = http://node643:8088/proxy/application_1470994648796_0013/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0013
- Hadoop job information for Stage-16: number of mappers: 29; number of reducers: 0
- 16/08/12 19:29:15 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 19:29:15,011 Stage-16 map = 0%, reduce = 0%
- 2016-08-12 19:29:35,792 Stage-16 map = 2%, reduce = 0%, Cumulative CPU 19.35 sec
- 2016-08-12 19:29:39,009 Stage-16 map = 3%, reduce = 0%, Cumulative CPU 91.45 sec
- 2016-08-12 19:29:43,324 Stage-16 map = 7%, reduce = 0%, Cumulative CPU 199.14 sec
- 2016-08-12 19:29:45,479 Stage-16 map = 14%, reduce = 0%, Cumulative CPU 225.15 sec
- 2016-08-12 19:29:47,625 Stage-16 map = 16%, reduce = 0%, Cumulative CPU 244.67 sec
- 2016-08-12 19:29:48,696 Stage-16 map = 22%, reduce = 0%, Cumulative CPU 281.88 sec
- 2016-08-12 19:29:49,779 Stage-16 map = 31%, reduce = 0%, Cumulative CPU 293.2 sec
- 2016-08-12 19:29:50,846 Stage-16 map = 34%, reduce = 0%, Cumulative CPU 312.23 sec
- 2016-08-12 19:29:51,915 Stage-16 map = 36%, reduce = 0%, Cumulative CPU 330.85 sec
- 2016-08-12 19:29:52,986 Stage-16 map = 40%, reduce = 0%, Cumulative CPU 337.59 sec
- 2016-08-12 19:29:54,049 Stage-16 map = 41%, reduce = 0%, Cumulative CPU 341.42 sec
- 2016-08-12 19:29:55,129 Stage-16 map = 43%, reduce = 0%, Cumulative CPU 347.94 sec
- 2016-08-12 19:29:56,203 Stage-16 map = 45%, reduce = 0%, Cumulative CPU 349.7 sec
- 2016-08-12 19:29:57,277 Stage-16 map = 47%, reduce = 0%, Cumulative CPU 358.26 sec
- 2016-08-12 19:29:59,426 Stage-16 map = 48%, reduce = 0%, Cumulative CPU 361.21 sec
- 2016-08-12 19:30:00,506 Stage-16 map = 52%, reduce = 0%, Cumulative CPU 379.57 sec
- 2016-08-12 19:30:05,873 Stage-16 map = 53%, reduce = 0%, Cumulative CPU 443.3 sec
- 2016-08-12 19:30:06,941 Stage-16 map = 57%, reduce = 0%, Cumulative CPU 460.63 sec
- 2016-08-12 19:30:08,009 Stage-16 map = 60%, reduce = 0%, Cumulative CPU 483.02 sec
- 2016-08-12 19:30:11,219 Stage-16 map = 64%, reduce = 0%, Cumulative CPU 553.62 sec
- 2016-08-12 19:30:12,287 Stage-16 map = 67%, reduce = 0%, Cumulative CPU 589.04 sec
- 2016-08-12 19:30:13,355 Stage-16 map = 72%, reduce = 0%, Cumulative CPU 598.74 sec
- 2016-08-12 19:30:15,494 Stage-16 map = 74%, reduce = 0%, Cumulative CPU 622.0 sec
- 2016-08-12 19:30:16,565 Stage-16 map = 78%, reduce = 0%, Cumulative CPU 632.12 sec
- 2016-08-12 19:30:17,638 Stage-16 map = 81%, reduce = 0%, Cumulative CPU 641.87 sec
- 2016-08-12 19:30:18,717 Stage-16 map = 83%, reduce = 0%, Cumulative CPU 651.4 sec
- 2016-08-12 19:30:19,784 Stage-16 map = 84%, reduce = 0%, Cumulative CPU 654.3 sec
- 2016-08-12 19:30:20,851 Stage-16 map = 90%, reduce = 0%, Cumulative CPU 670.98 sec
- 2016-08-12 19:30:21,920 Stage-16 map = 91%, reduce = 0%, Cumulative CPU 674.26 sec
- 2016-08-12 19:30:22,992 Stage-16 map = 93%, reduce = 0%, Cumulative CPU 682.34 sec
- 2016-08-12 19:30:26,194 Stage-16 map = 98%, reduce = 0%, Cumulative CPU 697.53 sec
- 2016-08-12 19:30:27,261 Stage-16 map = 100%, reduce = 0%, Cumulative CPU 698.01 sec
- MapReduce Total cumulative CPU time: 11 minutes 38 seconds 10 msec
- Ended Job = job_1470994648796_0013
- Stage-25 is filtered out by condition resolver.
- Stage-26 is filtered out by condition resolver.
- Stage-5 is selected by condition resolver.
- Launching Job 6 out of 15
- Number of reduce tasks not specified. Estimated from input data size: 10
- In order to change the average load for a reducer (in bytes):
- set hive.exec.reducers.bytes.per.reducer=<number>
- In order to limit the maximum number of reducers:
- set hive.exec.reducers.max=<number>
- In order to set a constant number of reducers:
- set mapreduce.job.reduces=<number>
- 16/08/12 19:30:31 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:30:31 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:30:31 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 19:32:16 INFO input.FileInputFormat: Total input paths to process : 30
- 16/08/12 19:32:16 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 4364567
- 16/08/12 19:32:16 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
- 16/08/12 19:32:17 INFO mapreduce.JobSubmitter: number of splits:15
- 16/08/12 19:32:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0014
- 16/08/12 19:32:18 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0014
- 16/08/12 19:32:18 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0014/
- Starting Job = job_1470994648796_0014, Tracking URL = http://node643:8088/proxy/application_1470994648796_0014/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0014
- Hadoop job information for Stage-5: number of mappers: 15; number of reducers: 10
- 16/08/12 19:32:47 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 19:32:47,002 Stage-5 map = 0%, reduce = 0%
- 2016-08-12 19:33:05,006 Stage-5 map = 2%, reduce = 0%, Cumulative CPU 16.74 sec
- 2016-08-12 19:33:08,223 Stage-5 map = 4%, reduce = 0%, Cumulative CPU 20.43 sec
- 2016-08-12 19:33:09,293 Stage-5 map = 7%, reduce = 0%, Cumulative CPU 33.74 sec
- 2016-08-12 19:33:14,659 Stage-5 map = 8%, reduce = 0%, Cumulative CPU 140.57 sec
- 2016-08-12 19:33:15,734 Stage-5 map = 19%, reduce = 0%, Cumulative CPU 157.44 sec
- 2016-08-12 19:33:16,804 Stage-5 map = 39%, reduce = 0%, Cumulative CPU 236.11 sec
- 2016-08-12 19:33:17,874 Stage-5 map = 46%, reduce = 0%, Cumulative CPU 245.0 sec
- 2016-08-12 19:33:20,023 Stage-5 map = 55%, reduce = 0%, Cumulative CPU 271.72 sec
- 2016-08-12 19:33:21,092 Stage-5 map = 58%, reduce = 0%, Cumulative CPU 278.26 sec
- 2016-08-12 19:33:22,172 Stage-5 map = 68%, reduce = 1%, Cumulative CPU 293.54 sec
- 2016-08-12 19:33:23,248 Stage-5 map = 73%, reduce = 1%, Cumulative CPU 302.53 sec
- 2016-08-12 19:33:25,403 Stage-5 map = 76%, reduce = 2%, Cumulative CPU 320.42 sec
- 2016-08-12 19:33:26,477 Stage-5 map = 79%, reduce = 2%, Cumulative CPU 330.54 sec
- 2016-08-12 19:33:28,621 Stage-5 map = 81%, reduce = 2%, Cumulative CPU 346.81 sec
- 2016-08-12 19:33:29,690 Stage-5 map = 84%, reduce = 7%, Cumulative CPU 356.93 sec
- 2016-08-12 19:33:31,840 Stage-5 map = 85%, reduce = 7%, Cumulative CPU 372.83 sec
- 2016-08-12 19:33:32,906 Stage-5 map = 86%, reduce = 8%, Cumulative CPU 377.17 sec
- 2016-08-12 19:33:35,047 Stage-5 map = 86%, reduce = 10%, Cumulative CPU 385.15 sec
- 2016-08-12 19:33:36,106 Stage-5 map = 86%, reduce = 14%, Cumulative CPU 390.71 sec
- 2016-08-12 19:33:37,174 Stage-5 map = 89%, reduce = 16%, Cumulative CPU 401.5 sec
- 2016-08-12 19:33:38,252 Stage-5 map = 93%, reduce = 16%, Cumulative CPU 417.24 sec
- 2016-08-12 19:33:39,325 Stage-5 map = 93%, reduce = 18%, Cumulative CPU 422.06 sec
- 2016-08-12 19:33:40,395 Stage-5 map = 93%, reduce = 20%, Cumulative CPU 429.3 sec
- 2016-08-12 19:33:41,466 Stage-5 map = 93%, reduce = 21%, Cumulative CPU 430.34 sec
- 2016-08-12 19:33:44,689 Stage-5 map = 93%, reduce = 24%, Cumulative CPU 445.4 sec
- 2016-08-12 19:33:48,979 Stage-5 map = 93%, reduce = 27%, Cumulative CPU 453.62 sec
- 2016-08-12 19:34:00,732 Stage-5 map = 96%, reduce = 27%, Cumulative CPU 481.33 sec
- 2016-08-12 19:34:01,806 Stage-5 map = 98%, reduce = 27%, Cumulative CPU 496.84 sec
- 2016-08-12 19:34:02,873 Stage-5 map = 98%, reduce = 29%, Cumulative CPU 498.16 sec
- 2016-08-12 19:34:03,940 Stage-5 map = 98%, reduce = 30%, Cumulative CPU 498.71 sec
- 2016-08-12 19:34:05,006 Stage-5 map = 98%, reduce = 31%, Cumulative CPU 499.47 sec
- 2016-08-12 19:34:17,843 Stage-5 map = 100%, reduce = 31%, Cumulative CPU 526.06 sec
- 2016-08-12 19:34:18,917 Stage-5 map = 100%, reduce = 32%, Cumulative CPU 528.93 sec
- 2016-08-12 19:34:19,987 Stage-5 map = 100%, reduce = 33%, Cumulative CPU 532.83 sec
- 2016-08-12 19:34:21,054 Stage-5 map = 100%, reduce = 37%, Cumulative CPU 552.13 sec
- 2016-08-12 19:34:22,123 Stage-5 map = 100%, reduce = 39%, Cumulative CPU 558.94 sec
- 2016-08-12 19:34:23,194 Stage-5 map = 100%, reduce = 41%, Cumulative CPU 565.28 sec
- 2016-08-12 19:34:24,272 Stage-5 map = 100%, reduce = 46%, Cumulative CPU 583.96 sec
- 2016-08-12 19:34:25,341 Stage-5 map = 100%, reduce = 47%, Cumulative CPU 590.27 sec
- 2016-08-12 19:34:26,408 Stage-5 map = 100%, reduce = 50%, Cumulative CPU 599.89 sec
- 2016-08-12 19:34:27,475 Stage-5 map = 100%, reduce = 54%, Cumulative CPU 615.06 sec
- 2016-08-12 19:34:28,552 Stage-5 map = 100%, reduce = 58%, Cumulative CPU 627.21 sec
- 2016-08-12 19:34:29,626 Stage-5 map = 100%, reduce = 60%, Cumulative CPU 636.3 sec
- 2016-08-12 19:34:30,690 Stage-5 map = 100%, reduce = 63%, Cumulative CPU 648.52 sec
- 2016-08-12 19:34:31,757 Stage-5 map = 100%, reduce = 65%, Cumulative CPU 659.35 sec
- 2016-08-12 19:34:32,824 Stage-5 map = 100%, reduce = 66%, Cumulative CPU 673.33 sec
- 2016-08-12 19:34:33,897 Stage-5 map = 100%, reduce = 67%, Cumulative CPU 699.63 sec
- 2016-08-12 19:34:34,965 Stage-5 map = 100%, reduce = 68%, Cumulative CPU 722.55 sec
- 2016-08-12 19:34:36,030 Stage-5 map = 100%, reduce = 71%, Cumulative CPU 754.03 sec
- 2016-08-12 19:34:37,098 Stage-5 map = 100%, reduce = 73%, Cumulative CPU 760.98 sec
- 2016-08-12 19:34:38,164 Stage-5 map = 100%, reduce = 75%, Cumulative CPU 767.18 sec
- 2016-08-12 19:34:39,239 Stage-5 map = 100%, reduce = 80%, Cumulative CPU 789.09 sec
- 2016-08-12 19:34:40,310 Stage-5 map = 100%, reduce = 82%, Cumulative CPU 795.42 sec
- 2016-08-12 19:34:41,379 Stage-5 map = 100%, reduce = 86%, Cumulative CPU 804.74 sec
- 2016-08-12 19:34:42,447 Stage-5 map = 100%, reduce = 90%, Cumulative CPU 819.99 sec
- 2016-08-12 19:34:43,521 Stage-5 map = 100%, reduce = 94%, Cumulative CPU 832.86 sec
- 2016-08-12 19:34:45,664 Stage-5 map = 100%, reduce = 97%, Cumulative CPU 842.42 sec
- 2016-08-12 19:34:48,876 Stage-5 map = 100%, reduce = 99%, Cumulative CPU 848.86 sec
- 2016-08-12 19:34:49,947 Stage-5 map = 100%, reduce = 100%, Cumulative CPU 850.51 sec
- MapReduce Total cumulative CPU time: 14 minutes 10 seconds 510 msec
- Ended Job = job_1470994648796_0014
- Stage-23 is filtered out by condition resolver.
- Stage-24 is filtered out by condition resolver.
- Stage-1 is selected by condition resolver.
- Launching Job 7 out of 15
- Number of reduce tasks not specified. Estimated from input data size: 2
- In order to change the average load for a reducer (in bytes):
- set hive.exec.reducers.bytes.per.reducer=<number>
- In order to limit the maximum number of reducers:
- set hive.exec.reducers.max=<number>
- In order to set a constant number of reducers:
- set mapreduce.job.reduces=<number>
- 16/08/12 19:34:54 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:34:54 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:34:54 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 19:36:42 INFO input.FileInputFormat: Total input paths to process : 11
- 16/08/12 19:36:42 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 0
- 16/08/12 19:36:42 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
- 16/08/12 19:36:44 INFO mapreduce.JobSubmitter: number of splits:8
- 16/08/12 19:36:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0015
- 16/08/12 19:36:45 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0015
- 16/08/12 19:36:45 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0015/
- Starting Job = job_1470994648796_0015, Tracking URL = http://node643:8088/proxy/application_1470994648796_0015/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0015
- Hadoop job information for Stage-1: number of mappers: 8; number of reducers: 2
- 16/08/12 19:37:16 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 19:37:16,292 Stage-1 map = 0%, reduce = 0%
- 2016-08-12 19:37:35,169 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 14.56 sec
- 2016-08-12 19:37:41,597 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 53.87 sec
- 2016-08-12 19:37:43,736 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 89.11 sec
- 2016-08-12 19:37:44,804 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 113.89 sec
- 2016-08-12 19:37:45,879 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 131.11 sec
- 2016-08-12 19:37:48,017 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 143.2 sec
- 2016-08-12 19:37:49,088 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 148.04 sec
- 2016-08-12 19:37:50,165 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 150.29 sec
- 2016-08-12 19:37:53,395 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 154.92 sec
- 2016-08-12 19:38:07,284 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 160.38 sec
- 2016-08-12 19:38:10,486 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 163.47 sec
- 2016-08-12 19:38:11,551 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 170.58 sec
- 2016-08-12 19:38:13,687 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 176.84 sec
- 2016-08-12 19:38:15,835 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 180.33 sec
- 2016-08-12 19:38:16,910 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 183.4 sec
- 2016-08-12 19:38:19,049 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 191.22 sec
- 2016-08-12 19:38:20,117 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 197.44 sec
- 2016-08-12 19:38:22,261 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 200.83 sec
- 2016-08-12 19:38:23,328 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 207.33 sec
- 2016-08-12 19:38:25,469 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 210.45 sec
- 2016-08-12 19:38:26,535 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 213.69 sec
- 2016-08-12 19:38:27,599 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 215.77 sec
- 2016-08-12 19:38:29,744 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 219.08 sec
- MapReduce Total cumulative CPU time: 3 minutes 39 seconds 80 msec
- Ended Job = job_1470994648796_0015
- Launching Job 8 out of 15
- Number of reduce tasks not specified. Estimated from input data size: 1
- In order to change the average load for a reducer (in bytes):
- set hive.exec.reducers.bytes.per.reducer=<number>
- In order to limit the maximum number of reducers:
- set hive.exec.reducers.max=<number>
- In order to set a constant number of reducers:
- set mapreduce.job.reduces=<number>
- 16/08/12 19:38:33 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:38:33 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:38:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 19:40:18 INFO input.FileInputFormat: Total input paths to process : 2
- 16/08/12 19:40:18 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 4, size left: 0
- 16/08/12 19:40:19 INFO mapreduce.JobSubmitter: number of splits:2
- 16/08/12 19:40:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0016
- 16/08/12 19:40:20 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0016
- 16/08/12 19:40:20 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0016/
- Starting Job = job_1470994648796_0016, Tracking URL = http://node643:8088/proxy/application_1470994648796_0016/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0016
- Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
- 16/08/12 19:40:44 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 19:40:44,851 Stage-2 map = 0%, reduce = 0%
- 2016-08-12 19:40:54,795 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 3.07 sec
- 2016-08-12 19:41:07,683 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 6.15 sec
- 2016-08-12 19:41:23,737 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 10.58 sec
- MapReduce Total cumulative CPU time: 10 seconds 580 msec
- Ended Job = job_1470994648796_0016
- Launching Job 9 out of 15
- Number of reduce tasks determined at compile time: 1
- In order to change the average load for a reducer (in bytes):
- set hive.exec.reducers.bytes.per.reducer=<number>
- In order to limit the maximum number of reducers:
- set hive.exec.reducers.max=<number>
- In order to set a constant number of reducers:
- set mapreduce.job.reduces=<number>
- 16/08/12 19:41:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:41:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:41:28 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 19:43:12 INFO input.FileInputFormat: Total input paths to process : 1
- 16/08/12 19:43:12 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
- 16/08/12 19:43:13 INFO mapreduce.JobSubmitter: number of splits:1
- 16/08/12 19:43:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0017
- 16/08/12 19:43:14 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0017
- 16/08/12 19:43:14 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0017/
- Starting Job = job_1470994648796_0017, Tracking URL = http://node643:8088/proxy/application_1470994648796_0017/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0017
- Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
- 16/08/12 19:43:39 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 19:43:39,872 Stage-3 map = 0%, reduce = 0%
- 2016-08-12 19:44:04,031 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 3.2 sec
- 2016-08-12 19:44:26,559 Stage-3 map = 100%, reduce = 100%, Cumulative CPU 8.11 sec
- MapReduce Total cumulative CPU time: 8 seconds 110 msec
- Ended Job = job_1470994648796_0017
- Loading data to table default.q5_local_supplier_volume
- 16/08/12 19:44:30 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
- 16/08/12 19:44:30 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
- MapReduce Jobs Launched:
- Stage-Stage-22: Map: 1 Cumulative CPU: 6.64 sec HDFS Read: 7979 HDFS Write: 222 SUCCESS
- Stage-Stage-19: Map: 1 Cumulative CPU: 9.47 sec HDFS Read: 14182336 HDFS Write: 577426 SUCCESS
- Stage-Stage-16: Map: 29 Cumulative CPU: 698.01 sec HDFS Read: 7776171787 HDFS Write: 557334620 SUCCESS
- Stage-Stage-5: Map: 15 Reduce: 10 Cumulative CPU: 850.51 sec HDFS Read: 2306747760 HDFS Write: 83207781 SUCCESS
- Stage-Stage-1: Map: 8 Reduce: 2 Cumulative CPU: 219.08 sec HDFS Read: 328125221 HDFS Write: 514 SUCCESS
- Stage-Stage-2: Map: 2 Reduce: 1 Cumulative CPU: 10.58 sec HDFS Read: 7927 HDFS Write: 257 SUCCESS
- Stage-Stage-3: Map: 1 Reduce: 1 Cumulative CPU: 8.11 sec HDFS Read: 6074 HDFS Write: 223 SUCCESS
- Total MapReduce CPU Time Spent: 30 minutes 2 seconds 400 msec
- OK
- Time taken: 1446.725 seconds
——set hive.execution.engine=tez
备注:在.hive文件里设置了为tez调度模式。
- lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q5_local_supplier_volume.hive
- which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
- 16/08/12 19:09:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-
- Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
- 16/08/12 19:09:31 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
- 16/08/12 19:09:39 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
- 16/08/12 19:09:39 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
- Query ID = lwh_20160812190931_c82a04f5-da59-4eb2-a814-4f295ace94f1
- Total jobs = 1
- Launching Job 1 out of 1
- 16/08/12 19:09:39 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
- 16/08/12 19:11:01 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:11:02 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0010
- 16/08/12 19:11:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
-
-
- Status: Running (Executing on YARN cluster with App id application_1470994648796_0010)
-
- ----------------------------------------------------------------------------------------------
- VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
- ----------------------------------------------------------------------------------------------
- Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
- Map 10 ......... container SUCCEEDED 1 1 0 0 0 0
- Map 11 ......... container SUCCEEDED 1 1 0 0 0 0
- Map 5 .......... container SUCCEEDED 4 4 0 0 0 0
- Map 7 .......... container SUCCEEDED 15 15 0 0 0 0
- Map 9 .......... container SUCCEEDED 1 1 0 0 0 0
- Reducer 2 ...... container SUCCEEDED 38 38 0 0 0 0
- Reducer 3 ...... container SUCCEEDED 41 41 0 0 0 0
- Reducer 4 ...... container SUCCEEDED 1 1 0 0 0 0
- Reducer 6 ...... container SUCCEEDED 35 35 0 0 0 0
- Reducer 8 ...... container SUCCEEDED 31 31 0 0 0 0
- ----------------------------------------------------------------------------------------------
- VERTICES: 11/11 [==========================>>] 100% ELAPSED TIME: 218.63 s
- ----------------------------------------------------------------------------------------------
- Loading data to table default.q5_local_supplier_volume
- 16/08/12 19:15:08 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
- OK
- Time taken: 339.425 seconds
Hive-on-MR | Hive-on-Tez | |
q1 | 436.074s | 189.653s |
q5 | 1446.725s | 339.425s |
②并行查询q1+q2
——hive.execution.engine=tez
q1:
- lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q1_pricing_summary_report.hive
- which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
- 16/08/12 19:56:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-
- Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
- 16/08/12 19:56:08 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
- 16/08/12 19:56:17 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
- 16/08/12 19:56:17 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
- Query ID = lwh_20160812195608_c1cde9c4-c451-4230-8515-92f1be8758ee
- Total jobs = 1
- Launching Job 1 out of 1
- 16/08/12 19:56:17 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
- 16/08/12 19:58:51 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 19:58:53 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0018
- 16/08/12 19:59:22 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
-
-
- Status: Running (Executing on YARN cluster with App id application_1470994648796_0018)
-
- ----------------------------------------------------------------------------------------------
- VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
- ----------------------------------------------------------------------------------------------
- Map 1 .......... container SUCCEEDED 15 15 0 0 0 0
- Reducer 2 ...... container SUCCEEDED 11 11 0 0 0 0
- Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0
- ----------------------------------------------------------------------------------------------
- VERTICES: 03/03 [==========================>>] 100% ELAPSED TIME: 136.81 s
- ----------------------------------------------------------------------------------------------
- OK
- 16/08/12 20:01:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
- 16/08/12 20:01:41 INFO mapred.FileInputFormat: Total input paths to process : 1
- A F 3.77518399E8 5.660657277972496E11 5.377591042780614E11 5.592766708921191E11 25.500975103007097 38237.15100895852 0.05000657454011058 14804077
- N F 9851614.0 1.4767438399170017E10 1.4028805792211395E10 1.459049099836676E10 25.522448302840946 38257.81066008119 0.04997336773764426 385998
- N O 7.43124873E8 1.1143022869018733E12 1.0585809221449624E12 1.1009370001705708E12 25.498075870689316 38233.90292348158 0.050000811821093484 29144351
- R F 3.7773283E8 5.664310549759983E11 5.381109226647607E11 5.596347808850894E11 25.50838478968014 38251.21927355965 0.04999679231395468 14808183
- Time taken: 332.832 seconds, Fetched: 4 row(s)
q2:
输出信息过多,没有收集到。
——hive.execution.engine=tez
q1:
- lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q1_pricing_summary_report.hive
- which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
- 16/08/12 20:16:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-
- Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
- Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
- 16/08/12 20:16:27 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
- 16/08/12 20:16:32 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
- WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
- Query ID = lwh_20160812201627_2ccd2f24-ad79-466b-8ad9-ff0f7b7cd191
- Total jobs = 2
- Launching Job 1 out of 2
- Number of reduce tasks not specified. Estimated from input data size: 31
- In order to change the average load for a reducer (in bytes):
- set hive.exec.reducers.bytes.per.reducer=<number>
- In order to limit the maximum number of reducers:
- set hive.exec.reducers.max=<number>
- In order to set a constant number of reducers:
- set mapreduce.job.reduces=<number>
- 16/08/12 20:16:37 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
- 16/08/12 20:16:38 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 20:16:39 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 20:16:40 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 20:19:45 INFO input.FileInputFormat: Total input paths to process : 1
- 16/08/12 20:19:45 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 268435456
- 16/08/12 20:19:47 INFO mapreduce.JobSubmitter: number of splits:29
- 16/08/12 20:19:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0022
- 16/08/12 20:19:49 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0022
- 16/08/12 20:19:49 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0022/
- Starting Job = job_1470994648796_0022, Tracking URL = http://node643:8088/proxy/application_1470994648796_0022/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0022
- Hadoop job information for Stage-1: number of mappers: 29; number of reducers: 31
- 16/08/12 20:20:16 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 20:20:16,607 Stage-1 map = 0%, reduce = 0%
- 2016-08-12 20:20:40,816 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 34.83 sec
- 2016-08-12 20:20:47,294 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 170.63 sec
- 2016-08-12 20:20:53,777 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 290.64 sec
- 2016-08-12 20:20:55,943 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 313.98 sec
- 2016-08-12 20:20:57,024 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 335.71 sec
- 2016-08-12 20:21:00,260 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 395.76 sec
- 2016-08-12 20:21:01,334 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 407.72 sec
- 2016-08-12 20:21:02,413 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 425.55 sec
- 2016-08-12 20:21:03,498 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 433.11 sec
- 2016-08-12 20:21:08,897 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 456.3 sec
- 2016-08-12 20:21:09,970 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 462.23 sec
- 2016-08-12 20:21:12,115 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 467.86 sec
- 2016-08-12 20:21:13,187 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 482.36 sec
- 2016-08-12 20:21:15,356 Stage-1 map = 49%, reduce = 1%, Cumulative CPU 522.83 sec
- 2016-08-12 20:21:16,430 Stage-1 map = 52%, reduce = 1%, Cumulative CPU 528.38 sec
- 2016-08-12 20:21:21,816 Stage-1 map = 53%, reduce = 1%, Cumulative CPU 603.66 sec
- 2016-08-12 20:21:22,889 Stage-1 map = 54%, reduce = 1%, Cumulative CPU 616.1 sec
- 2016-08-12 20:21:23,962 Stage-1 map = 56%, reduce = 1%, Cumulative CPU 626.38 sec
- 2016-08-12 20:21:27,195 Stage-1 map = 57%, reduce = 1%, Cumulative CPU 665.39 sec
- 2016-08-12 20:21:28,270 Stage-1 map = 59%, reduce = 1%, Cumulative CPU 677.06 sec
- 2016-08-12 20:21:29,346 Stage-1 map = 60%, reduce = 2%, Cumulative CPU 707.06 sec
- 2016-08-12 20:21:31,495 Stage-1 map = 67%, reduce = 2%, Cumulative CPU 733.66 sec
- 2016-08-12 20:21:33,657 Stage-1 map = 72%, reduce = 2%, Cumulative CPU 753.21 sec
- 2016-08-12 20:21:36,878 Stage-1 map = 77%, reduce = 2%, Cumulative CPU 773.63 sec
- 2016-08-12 20:21:37,945 Stage-1 map = 78%, reduce = 2%, Cumulative CPU 777.17 sec
- 2016-08-12 20:21:39,026 Stage-1 map = 79%, reduce = 2%, Cumulative CPU 780.76 sec
- 2016-08-12 20:21:40,103 Stage-1 map = 82%, reduce = 2%, Cumulative CPU 783.92 sec
- 2016-08-12 20:21:42,241 Stage-1 map = 82%, reduce = 3%, Cumulative CPU 790.95 sec
- 2016-08-12 20:21:44,376 Stage-1 map = 84%, reduce = 3%, Cumulative CPU 795.82 sec
- 2016-08-12 20:21:46,535 Stage-1 map = 86%, reduce = 4%, Cumulative CPU 835.79 sec
- 2016-08-12 20:21:48,681 Stage-1 map = 86%, reduce = 5%, Cumulative CPU 851.77 sec
- 2016-08-12 20:21:49,756 Stage-1 map = 86%, reduce = 6%, Cumulative CPU 861.69 sec
- 2016-08-12 20:21:55,102 Stage-1 map = 87%, reduce = 6%, Cumulative CPU 882.42 sec
- 2016-08-12 20:21:56,174 Stage-1 map = 89%, reduce = 8%, Cumulative CPU 888.73 sec
- 2016-08-12 20:21:57,245 Stage-1 map = 89%, reduce = 9%, Cumulative CPU 890.14 sec
- 2016-08-12 20:22:00,466 Stage-1 map = 89%, reduce = 10%, Cumulative CPU 903.81 sec
- 2016-08-12 20:22:02,603 Stage-1 map = 91%, reduce = 10%, Cumulative CPU 914.72 sec
- 2016-08-12 20:22:04,740 Stage-1 map = 93%, reduce = 11%, Cumulative CPU 920.01 sec
- 2016-08-12 20:22:07,945 Stage-1 map = 94%, reduce = 11%, Cumulative CPU 924.0 sec
- 2016-08-12 20:22:10,087 Stage-1 map = 95%, reduce = 11%, Cumulative CPU 925.81 sec
- 2016-08-12 20:22:14,371 Stage-1 map = 95%, reduce = 12%, Cumulative CPU 932.02 sec
- 2016-08-12 20:22:16,506 Stage-1 map = 95%, reduce = 13%, Cumulative CPU 936.41 sec
- 2016-08-12 20:22:28,287 Stage-1 map = 98%, reduce = 13%, Cumulative CPU 950.75 sec
- 2016-08-12 20:22:29,355 Stage-1 map = 100%, reduce = 13%, Cumulative CPU 952.36 sec
- 2016-08-12 20:22:30,424 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 954.69 sec
- 2016-08-12 20:22:31,492 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 963.74 sec
- 2016-08-12 20:22:32,562 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 980.79 sec
- 2016-08-12 20:22:33,632 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 1017.66 sec
- 2016-08-12 20:22:34,704 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 1023.84 sec
- 2016-08-12 20:22:38,973 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 1031.16 sec
- 2016-08-12 20:22:40,041 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 1036.93 sec
- 2016-08-12 20:22:41,118 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 1042.65 sec
- 2016-08-12 20:22:43,278 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 1049.32 sec
- 2016-08-12 20:22:44,355 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1077.92 sec
- 2016-08-12 20:22:45,432 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1099.9 sec
- 2016-08-12 20:22:46,501 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1105.43 sec
- 2016-08-12 20:22:49,707 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 1111.84 sec
- 2016-08-12 20:22:50,778 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1124.73 sec
- 2016-08-12 20:22:51,847 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 1131.81 sec
- 2016-08-12 20:22:55,050 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1139.38 sec
- MapReduce Total cumulative CPU time: 18 minutes 59 seconds 380 msec
- Ended Job = job_1470994648796_0022
- Launching Job 2 out of 2
- Number of reduce tasks determined at compile time: 1
- In order to change the average load for a reducer (in bytes):
- set hive.exec.reducers.bytes.per.reducer=<number>
- In order to limit the maximum number of reducers:
- set hive.exec.reducers.max=<number>
- In order to set a constant number of reducers:
- set mapreduce.job.reduces=<number>
- 16/08/12 20:23:03 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 20:23:03 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
- 16/08/12 20:23:03 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
- 16/08/12 20:25:38 INFO input.FileInputFormat: Total input paths to process : 31
- 16/08/12 20:25:38 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
- 16/08/12 20:25:40 INFO mapreduce.JobSubmitter: number of splits:6
- 16/08/12 20:25:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0025
- 16/08/12 20:25:41 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0025
- 16/08/12 20:25:41 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0025/
- Starting Job = job_1470994648796_0025, Tracking URL = http://node643:8088/proxy/application_1470994648796_0025/
- Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job -kill job_1470994648796_0025
- Hadoop job information for Stage-2: number of mappers: 6; number of reducers: 1
- 16/08/12 20:26:06 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
- 2016-08-12 20:26:06,419 Stage-2 map = 0%, reduce = 0%
- 2016-08-12 20:26:16,449 Stage-2 map = 17%, reduce = 0%, Cumulative CPU 3.61 sec
- 2016-08-12 20:26:28,243 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 7.08 sec
- 2016-08-12 20:26:29,314 Stage-2 map = 67%, reduce = 0%, Cumulative CPU 14.63 sec
- 2016-08-12 20:26:32,526 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 21.77 sec
- 2016-08-12 20:26:35,737 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 27.16 sec
- MapReduce Total cumulative CPU time: 27 seconds 160 msec
- Ended Job = job_1470994648796_0025
- MapReduce Jobs Launched:
- Stage-Stage-1: Map: 29 Reduce: 31 Cumulative CPU: 1139.38 sec HDFS Read: 7776522714 HDFS Write: 3305 SUCCESS
- Stage-Stage-2: Map: 6 Reduce: 1 Cumulative CPU: 27.16 sec HDFS Read: 39220 HDFS Write: 723 SUCCESS
- Total MapReduce CPU Time Spent: 19 minutes 26 seconds 540 msec
- OK
- 16/08/12 20:26:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
- 16/08/12 20:26:41 INFO mapred.FileInputFormat: Total input paths to process : 1
- A F 3.77518399E8 5.660657277972468E11 5.3775910427806824E11 5.592766708921185E11 25.500975103007097 38237.15100895833 0.05000657454026939 14804077
- N F 9851614.0 1.4767438399169981E10 1.4028805792211395E10 1.4590490998366728E10 25.522448302840946 38257.81066008109 0.04997336773765435 385998
- N O 7.43124873E8 1.1143022869018765E12 1.0585809221449629E12 1.100937000170583E12 25.498075870689316 38233.90292348169 0.050000811820997394 29144351
- R F 3.7773283E8 5.664310549759971E11 5.381109226647635E11 5.596347808850863E11 25.50838478968014 38251.21927355956 0.04999679231411408 14808183
- Time taken: 614.032 seconds, Fetched: 4 row(s)
q2:
Time taken: 968.034 seconds, Fetched: 6351 row(s)
Hive-on-MR | Hive-on-Tez | |
q1 | 614.032s | 332.832s |
q2 | 968.034s | / |
total | 968.034s | 332.832s |