Hadoop 当通过配置单元将数据插入hbase时，reducer卡在99%的位置

我在amazon-ec2hadoop 1.0.1、hive-0.11.0、hbase-0.94.11、zookeeper-3.4.3上运行了两个节点的hbase集群，并使用ami-2.4.1在EMR节点上创建

所以在EMR实例中，我有一个外部表，它指向S3上的某个位置。另外，我还创建了一个hbase配置单元表modelvarlarge，modelval。现在，我正在尝试将logdata中的数据插入modelvar

但是，减速器相位卡在99%并导致以下错误： 仅供参考，thorugh，我可以从EMR连接到Ec2 zookeeper

外部表：

Hbase配置单元表：

查询： 插入到表modelvar中 选择x.cookie、点击、访问 从选择cs_Cookie作为Cookie，将*计算为来自日志数据的点击，其中cs_Cookie加入的uri（如“%.aspx%”或uri（如“%.html%”）组选择cs_Cookie作为Cookie，将countdistinct cs_Cookie作为cs_Cookie访问的日志数据组 在x.cookie=y.cookie上，按点击次数排序

错误：

---

您需要在整个EMR集群中定义主机和ip映射。假设您在Ec2上使用的是3节点hbase群集，其IP为

 ip1, ip2, ip3

我们在ec2 hbase群集的主机文件中为其指定了如下别名：

ip1 master
ip2 rgserver1
ip3 rgserver3

因此，在每个EMR节点的主机文件中，您还需要定义与上面类似的映射。否则，它将无法将数据写入hbase群集

java\.lang\.RuntimeException: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: Hive Runtime Error while processing row (tag\=0) {\"key\":{\"reducesinkkey0\":24655},\"value\":{\"_col0\":\"-\",\"_col1\":24655,\"_col2\":17},\"alias\":0}
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecReducer\.reduce(ExecReducer\.java:278)
at org\.apache\.hadoop\.mapred\.ReduceTask\.runOldReducer(ReduceTask\.java:528)
at org\.apache\.hadoop\.mapred\.ReduceTask\.run(ReduceTask\.java:429)
at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:255)
at java\.security\.AccessController\.doPrivileged(Native Method)
at javax\.security\.auth\.Subject\.doAs(Subject\.java:415)
at org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1132)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:249)
Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: Hive Runtime Error while processing row (tag\=0) {\"key\":{\"reducesinkkey0\":24655},\"value\":{\"_col0\":\"-\",\"_col1\":24655,\"_col2\":17},\"alias\":0}
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecReducer\.reduce(ExecReducer\.java:266)
\.\.\. 7 more
Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: java\.io\.IOException: org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation@10f00d3 closed
at org\.apache\.hadoop\.hive\.ql\.io\.HiveFileFormatUtils\.getHiveRecordWriter(HiveFileFormatUtils\.java:241)
at org\.apache\.hadoop\.hive\.ql\.exec\.FileSinkOperator\.createBucketFiles(FileSinkOperator\.java:539)
at org\.apache\.hadoop\.hive\.ql\.exec\.FileSinkOperator\.processOp(FileSinkOperator\.java:621)
at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.process(Operator\.java:502)
at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.forward(Operator\.java:832)
at org\.apache\.hadoop\.hive\.ql\.exec\.SelectOperator\.processOp(SelectOperator\.java:84)
at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.process(Operator\.java:502)
at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.forward(Operator\.java:832)
at org\.apache\.hadoop\.hive\.ql\.exec\.ExtractOperator\.processOp(ExtractOperator\.java:45)
at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.process(Operator\.java:502)
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecReducer\.reduce(ExecReducer\.java:257)
\.\.\. 7 more
Caused by: java\.io\.IOException: org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation@10f00d3 closed
at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.locateRegion(HConnectionManager\.java:794)
at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.locateRegion(HConnectionManager\.java:782)
at org\.apache\.hadoop\.hbase\.client\.HTable\.finishSetup(HTable\.java:249)
at org\.apache\.hadoop\.hbase\.client\.HTable\.(HTable\.java:213)
at org\.apache\.hadoop\.hbase\.client\.HTable\.(HTable\.java:171)
at org\.apache\.hadoop\.hive\.hbase\.HiveHBaseTableOutputFormat\.getHiveRecordWriter(HiveHBaseTableOutputFormat\.java:82)
at org\.apache\.hadoop\.hive\.ql\.io\.HiveFileFormatUtils\.getRecordWriter(HiveFileFormatUtils\.java:250)
at org\.apache\.hadoop\.hive\.ql\.io\.HiveFileFormatUtils\.getHiveRecordWriter(HiveFileFormatUtils\.java:238)
\.\.\. 17 more

 ip1, ip2, ip3

ip1 master
ip2 rgserver1
ip3 rgserver3