Hadoop 在reduce期间停止带有where子句的配置单元交叉连接_Hadoop_Mapreduce_Hive_Cross Join

Hadoop 在reduce期间停止带有where子句的配置单元交叉连接

hadoop mapreduce hive

Hadoop 在reduce期间停止带有where子句的配置单元交叉连接,hadoop,mapreduce,hive,cross-join,Hadoop,Mapreduce,Hive,Cross Join,我在两个表（大小：table_a~100k行，table_b~2亿行）上运行了交叉连接，如下所示： select a.id, a.first_name, a.last_name, a.street , b.pid, b.first_nam, b.last_nam, b.strt from table_a a cross join table_b b where (a.phone_number in (b.phone_1, b.phone_2) and a.birth_da

我在两个表（大小：table_a~100k行，table_b~2亿行）上运行了交叉连接，如下所示：

select a.id, a.first_name, a.last_name, a.street
     , b.pid, b.first_nam, b.last_nam, b.strt
from table_a a
  cross join table_b b
where
  (a.phone_number in (b.phone_1, b.phone_2)
   and a.birth_date in (b.birth_date_1, b.birth_date_2))
  or
  (
  cast(a.zip_code as int) = cast(b.zip_code as int)
  and
    (
      (
      upper(a.last_name) = upper(b.last_nam)
      and a.birth_date in (b.birth_date_1, b.birth_date_2)
      )
      or
      (
      a.phone_number in (b.phone_1, b.phone_2)
      )
      or
      (
      a.birth_date in (b.birth_date_1, b.birth_date_2))
      and upper(a.street) = upper(b.strt))
      )
    )
  );

作业从集群开始，显示选择了160个映射器和1个reducer来运行它。地图部分完成得相当快（大约3分钟）。降价开始，从1%、2%、4%开始缓慢但稳定地运行。。。直到它达到67%。然后它停滞了30多分钟，没有任何进展。虽然它显示累积的CPU时间增加了一些（从大约3500秒增加到4200秒）。但降幅不会超过67%。最后，我不得不取消工作，并试图重新运行它。我每次都有同样的行为。我试图通过设置以下选项来增加减速机的#：

set mapred.job.reduces=10;

但它仍然显示每当我运行查询时只使用1个reducer。你知道这里发生了什么，我如何解决这个问题吗

编辑：以下是输出

INFO  : Stage-1 is selected by condition resolver.
INFO  : Number of reduce tasks determined at compile time: 1
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : number of splits:160
INFO  : Submitting tokens for job: job_1484094252811_0005
INFO  : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN token 53 for hive)
INFO  : The url to track the job: redacted
INFO  : Starting Job = job_1484094252811_0005, redacted
INFO  : Kill Command = /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hadoop/bin/hadoop job  -kill job_1484094252811_0005
INFO  : Hadoop job information for Stage-1: number of mappers: 160; number of reducers: 1
INFO  : 2017-01-11 14:11:36,957 Stage-1 map = 0%,  reduce = 0%
INFO  : 2017-01-11 14:11:49,499 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 10.45 sec
INFO  : 2017-01-11 14:11:50,549 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 17.03 sec
INFO  : 2017-01-11 14:11:51,610 Stage-1 map = 21%,  reduce = 0%, Cumulative CPU 1100.63 sec
INFO  : 2017-01-11 14:11:52,674 Stage-1 map = 22%,  reduce = 0%, Cumulative CPU 1181.04 sec
INFO  : 2017-01-11 14:11:54,773 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 1744.29 sec
INFO  : 2017-01-11 14:11:55,819 Stage-1 map = 41%,  reduce = 0%, Cumulative CPU 1769.61 sec
INFO  : 2017-01-11 14:11:57,903 Stage-1 map = 58%,  reduce = 0%, Cumulative CPU 2444.53 sec
INFO  : 2017-01-11 14:11:59,988 Stage-1 map = 62%,  reduce = 0%, Cumulative CPU 2505.06 sec
INFO  : 2017-01-11 14:12:01,032 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2999.97 sec
INFO  : 2017-01-11 14:12:12,486 Stage-1 map = 100%,  reduce = 2%, Cumulative CPU 3051.65 sec
INFO  : 2017-01-11 14:12:21,878 Stage-1 map = 100%,  reduce = 3%, Cumulative CPU 3084.96 sec
INFO  : 2017-01-11 14:12:27,088 Stage-1 map = 100%,  reduce = 4%, Cumulative CPU 3108.88 sec
INFO  : 2017-01-11 14:12:36,477 Stage-1 map = 100%,  reduce = 5%, Cumulative CPU 3132.38 sec
INFO  : 2017-01-11 14:12:45,837 Stage-1 map = 100%,  reduce = 6%, Cumulative CPU 3153.54 sec
INFO  : 2017-01-11 14:12:55,204 Stage-1 map = 100%,  reduce = 7%, Cumulative CPU 3181.5 sec
INFO  : 2017-01-11 14:13:01,437 Stage-1 map = 100%,  reduce = 8%, Cumulative CPU 3204.99 sec
INFO  : 2017-01-11 14:13:10,781 Stage-1 map = 100%,  reduce = 9%, Cumulative CPU 3231.7 sec
INFO  : 2017-01-11 14:13:17,012 Stage-1 map = 100%,  reduce = 10%, Cumulative CPU 3255.17 sec
INFO  : 2017-01-11 14:13:25,320 Stage-1 map = 100%,  reduce = 11%, Cumulative CPU 3280.29 sec
INFO  : 2017-01-11 14:13:31,560 Stage-1 map = 100%,  reduce = 13%, Cumulative CPU 3300.07 sec
INFO  : 2017-01-11 14:13:40,903 Stage-1 map = 100%,  reduce = 14%, Cumulative CPU 3326.19 sec
INFO  : 2017-01-11 14:13:47,133 Stage-1 map = 100%,  reduce = 15%, Cumulative CPU 3345.77 sec
INFO  : 2017-01-11 14:13:56,473 Stage-1 map = 100%,  reduce = 16%, Cumulative CPU 3370.6 sec
INFO  : 2017-01-11 14:14:02,696 Stage-1 map = 100%,  reduce = 17%, Cumulative CPU 3392.17 sec
INFO  : 2017-01-11 14:14:11,142 Stage-1 map = 100%,  reduce = 18%, Cumulative CPU 3416.55 sec
INFO  : 2017-01-11 14:14:17,365 Stage-1 map = 100%,  reduce = 19%, Cumulative CPU 3436.33 sec
INFO  : 2017-01-11 14:14:26,710 Stage-1 map = 100%,  reduce = 20%, Cumulative CPU 3458.24 sec
INFO  : 2017-01-11 14:14:32,926 Stage-1 map = 100%,  reduce = 21%, Cumulative CPU 3477.13 sec
INFO  : 2017-01-11 14:14:42,262 Stage-1 map = 100%,  reduce = 22%, Cumulative CPU 3504.41 sec
INFO  : 2017-01-11 14:14:48,485 Stage-1 map = 100%,  reduce = 23%, Cumulative CPU 3532.08 sec
INFO  : 2017-01-11 14:14:56,778 Stage-1 map = 100%,  reduce = 24%, Cumulative CPU 3561.15 sec
INFO  : 2017-01-11 14:15:02,997 Stage-1 map = 100%,  reduce = 25%, Cumulative CPU 3581.73 sec
INFO  : 2017-01-11 14:15:12,319 Stage-1 map = 100%,  reduce = 26%, Cumulative CPU 3607.59 sec
INFO  : 2017-01-11 14:15:18,543 Stage-1 map = 100%,  reduce = 27%, Cumulative CPU 3631.92 sec
INFO  : 2017-01-11 14:15:27,878 Stage-1 map = 100%,  reduce = 28%, Cumulative CPU 3655.75 sec
INFO  : 2017-01-11 14:15:34,094 Stage-1 map = 100%,  reduce = 29%, Cumulative CPU 3677.59 sec
INFO  : 2017-01-11 14:15:40,369 Stage-1 map = 100%,  reduce = 30%, Cumulative CPU 3703.33 sec
INFO  : 2017-01-11 14:15:49,687 Stage-1 map = 100%,  reduce = 31%, Cumulative CPU 3726.21 sec
INFO  : 2017-01-11 14:15:52,789 Stage-1 map = 100%,  reduce = 32%, Cumulative CPU 3730.8 sec
INFO  : 2017-01-11 14:15:57,964 Stage-1 map = 100%,  reduce = 33%, Cumulative CPU 3748.92 sec
INFO  : 2017-01-11 14:16:07,269 Stage-1 map = 100%,  reduce = 59%, Cumulative CPU 3765.56 sec
INFO  : 2017-01-11 14:16:10,377 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU 3769.25 sec
INFO  : 2017-01-11 14:17:10,449 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU 3804.54 sec
INFO  : 2017-01-11 14:18:10,459 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU 3804.54 sec
INFO  : 2017-01-11 14:19:10,521 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU 3804.54 sec
INFO  : 2017-01-11 14:20:10,569 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU 3804.54 sec
INFO  : 2017-01-11 14:21:11,569 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU 3804.54 sec
INFO  : 2017-01-11 14:22:12,566 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU 3804.54 sec
INFO  : 2017-01-11 14:23:12,587 Stage-1 map = 100%,  reduce = 67%, Cumulative CPU 3804.54 sec

INFO:Stage-1由条件解析器选择。
信息：编译时确定的reduce任务数：1
信息：要更改减速器的平均负载（以字节为单位）：
信息：设置hive.exec.reducers.bytes.per.reducer=
信息：为了限制减速器的最大数量：
信息：设置hive.exec.reducers.max=
信息：为了设置恒定数量的减速器：
信息：设置mapreduce.job.reduces=
信息：拆分次数：160
信息：提交作业令牌：作业1484094252811\u 0005
信息：种类：HDFS_委派_令牌，服务：ha HDFS:nameservice1，标识：（用于配置单元的HDFS_委派_令牌53）
信息：跟踪作业的url：已编辑
信息：起始作业=Job_1484094252811_0005，已编辑
信息：Kill命令=/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hadoop/bin/hadoop作业-Kill作业
信息：第一阶段Hadoop作业信息：映射者数量：160；减速器数量：1
信息：2017-01-11 14:11:36957第一阶段地图=0%，减少=0%
信息：2017-01-11 14:11:49499第一阶段映射=1%，减少=0%，累计CPU 10.45秒
信息：2017-01-11 14:11:50549第一阶段地图=2%，减少=0%，累计CPU 17.03秒
信息：2017-01-11 14:11:51610第一阶段映射=21%，减少=0%，累计CPU 1100.63秒
信息：2017-01-11 14:11:52674第一阶段映射=22%，减少=0%，累计CPU 1181.04秒
信息：2017-01-11 14:11:54773第一阶段映射=40%，减少=0%，累计CPU 1744.29秒
信息：2017-01-11 14:11:55819第一阶段映射=41%，减少=0%，累计CPU 1769.61秒
信息：2017-01-11 14:11:57903第一阶段映射=58%，减少=0%，累计CPU 2444.53秒
信息：2017-01-11 14:11:59988第一阶段映射=62%，减少=0%，累计CPU 2505.06秒
信息：2017-01-11 14:12:01032第一阶段地图=100%，减少=0%，累计CPU 2999.97秒
信息：2017-01-11 14:12:12486第一阶段地图=100%，减少=2%，累计CPU 3051.65秒
信息：2017-01-11 14:12:21878第一阶段映射=100%，减少=3%，累计CPU 3084.96秒
信息：2017-01-11 14:12:27088第一阶段映射=100%，减少=4%，累计CPU 3108.88秒
信息：2017-01-11 14:12:36477第一阶段映射=100%，减少=5%，累计CPU 3132.38秒
信息：2017-01-11 14:12:45837第一阶段地图=100%，减少=6%，累计CPU 3153.54秒
信息：2017-01-11 14:12:55204第一阶段映射=100%，减少=7%，累计CPU 3181.5秒
信息：2017-01-11 14:13:01437第一阶段映射=100%，减少=8%，累计CPU 3204.99秒
信息：2017-01-11 14:13:10781第一阶段映射=100%，减少=9%，累计CPU 3231.7秒
信息：2017-01-11 14:13:17012第一阶段映射=100%，减少=10%，累计CPU 3255.17秒
信息：2017-01-11 14:13:25320第一阶段映射=100%，减少=11%，累计CPU 3280.29秒
信息：2017-01-11 14:13:31560第一阶段地图=100%，减少=13%，累计CPU 3300.07秒
信息：2017-01-11 14:13:40903第一阶段地图=100%，减少=14%，累计CPU 3326.19秒
信息：2017-01-11 14:13:47133第一阶段地图=100%，减少=15%，累计CPU 3345.77秒
信息：2017-01-11 14:13:56473第一阶段映射=100%，减少=16%，累计CPU 3370.6秒
信息：2017-01-11 14:14:02696第一阶段地图=100%，减少=17%，累计CPU 3392.17秒
信息：2017-01-11 14:14:11142第一阶段地图=100%，减少=18%，累计CPU 3416.55秒
信息：2017-01-11 14:14:17365第一阶段地图=100%，减少=19%，累计CPU 3436.33秒
信息：2017-01-11 14:14:26710第一阶段映射=100%，减少=20%，累计CPU 3458.24秒
信息：2017-01-11 14:14:32926第一阶段地图=100%，减少=21%，累计CPU 3477.13秒
信息：2017-01-11 14:14:42262第一阶段映射=100%，减少=22%，累计CPU 3504.41秒
信息：2017-01-11 14:14:48485第一阶段映射=100%，减少=23%，累计CPU 3532.08秒
信息：2017-01-11 14:14:56778第一阶段映射=100%，减少=24%，累计CPU 3561.15秒
信息：2017-01-11 14:15:02997第一阶段映射=100%，减少=25%，累计CPU 3581.73秒
信息：2017-01-11 14:15:12319第一阶段映射=100%，减少=26%，累计CPU 3607.59秒
信息：2017-01-11 14:15:18543第一阶段地图=100%，减少=27%，累计CPU 3631.92秒
信息：2017-01-11 14:15:27878第一阶段映射=100%，减少=28%，累计CPU 3655.75秒
信息：2017-01-11 14:15:34094第一阶段地图=100%，减少=29%，累计CPU 3677.59秒
信息：2017-01-11 14:15:40369第一阶段映射=100%，减少=30%，累计CPU 3703.33秒
信息：2017-01-11 14:15:49687第一阶段映射=100%，减少=31%，累计CPU 3726.21秒
信息：2017-01-11 14:15:52789第一阶段映射=100%，减少=32%，累计CPU 3730.8秒
信息：2017-01-11 14:15:57964第一阶段映射=100%，减少=33%，累计CPU 3748.92秒
信息：2017-01-11 14:16:07269第一阶段映射=100%，减少=59%，累计CPU 3765.56秒
信息：2017-01-11 14:16:10377第一阶段映射=100%，减少=67%，累计CPU 3769.25秒
信息：2017-01-11 14:17:10449第一阶段地图=100%，减少=67%，累计CPU 38