Hadoop 如何加速mapreduce作业并避免群集'；当使用importtsv时会崩溃吗？

我是Hadoop和HBase的初学者，正在学习如何使用“importtsv”将大量数据（存储在HDFS中的8GB tsv文件）导入HBase。然而，mapreduce的工作似乎非常缓慢，经过很长一段时间后，它失败了。可能文件太大，导致集群崩溃。当我改为使用一个小的tsv文件时，它工作得很好。那么，如果我坚持导入这么大的文件，如何加快mapreduce的工作速度呢？Hadoop中是否有任何缓存配置可以帮助实现这一点

我有一个macOS名称节点和两个Ubuntu数据节点

导入命令：

hbase org.apache.hadoop.hbase.mapreduce.ImportTsv  -Dimporttsv.columns=HBASE_ROW_KEY,info:action,timestamp,info:name,info:bank,info:account records /user/root

错误信息：

2017-03-20 16:48:27,136 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop/lib/native
2017-03-20 16:48:27,136 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/var/folders/nl/f3lktfgn7jg46jycx21cxfmr0000gn/T/
2017-03-20 16:48:27,136 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2017-03-20 16:48:27,137 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Mac OS X
2017-03-20 16:48:27,137 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=x86_64
2017-03-20 16:48:27,137 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=10.12.3
2017-03-20 16:48:27,137 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=haohui
2017-03-20 16:48:27,137 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/Users/haohui
2017-03-20 16:48:27,138 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/Users/haohui
2017-03-20 16:48:27,138 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,node1:2181,node2:2181 sessionTimeout=30000 watcher=hconnection-0x3fc2959f0x0, quorum=master:2181,node1:2181,node2:2181, baseZNode=/hbase
2017-03-20 16:48:27,157 INFO  [main-SendThread(master:2181)] zookeeper.ClientCnxn: Opening socket connection to server master/10.211.55.2:2181. Will not attempt to authenticate using SASL (unknown error)
2017-03-20 16:48:27,188 INFO  [main-SendThread(master:2181)] zookeeper.ClientCnxn: Socket connection established to master/10.211.55.2:2181, initiating session
2017-03-20 16:48:27,200 INFO  [main-SendThread(master:2181)] zookeeper.ClientCnxn: Session establishment complete on server master/10.211.55.2:2181, sessionid = 0x15aeae6867a0001, negotiated timeout = 30000
2017-03-20 16:48:56,396 INFO  [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2017-03-20 16:48:56,441 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15aeae6867a0001
2017-03-20 16:48:56,450 INFO  [main] zookeeper.ZooKeeper: Session: 0x15aeae6867a0001 closed
2017-03-20 16:48:56,450 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-03-20 16:48:56,524 INFO  [main] client.RMProxy: Connecting to ResourceManager at master/10.211.55.2:8032
2017-03-20 16:48:56,666 INFO  [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2017-03-20 16:48:58,818 INFO  [main] input.FileInputFormat: Total input paths to process : 1
2017-03-20 16:48:58,873 INFO  [main] mapreduce.JobSubmitter: number of splits:56
2017-03-20 16:48:58,884 INFO  [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2017-03-20 16:48:59,006 INFO  [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1489999688045_0001
2017-03-20 16:48:59,319 INFO  [main] impl.YarnClientImpl: Submitted application application_1489999688045_0001
2017-03-20 16:48:59,370 INFO  [main] mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1489999688045_0001/
2017-03-20 16:48:59,371 INFO  [main] mapreduce.Job: Running job: job_1489999688045_0001
2017-03-20 16:49:09,668 INFO  [main] mapreduce.Job: Job job_1489999688045_0001 running in uber mode : false
2017-03-20 16:49:09,670 INFO  [main] mapreduce.Job:  map 0% reduce 0%
2017-03-20 17:00:09,103 INFO  [main] mapreduce.Job: Task Id : attempt_1489999688045_0001_m_000009_0, Status : FAILED
AttemptID:attempt_1489999688045_0001_m_000009_0 Timed out after 600 secs
2017-03-20 17:00:09,127 INFO  [main] mapreduce.Job: Task Id : attempt_1489999688045_0001_m_000011_0, Status : FAILED
AttemptID:attempt_1489999688045_0001_m_000011_0 Timed out after 600 secs
2017-03-20 17:00:09,128 INFO  [main] mapreduce.Job: Task Id : attempt_1489999688045_0001_m_000010_0, Status : FAILED
AttemptID:attempt_1489999688045_0001_m_000010_0 Timed out after 600 secs
2017-03-20 17:00:09,129 INFO  [main] mapreduce.Job: Task Id : attempt_1489999688045_0001_m_000013_0, Status : FAILED
AttemptID:attempt_1489999688045_0001_m_000013_0 Timed out after 600 secs
2017-03-20 17:00:09,130 INFO  [main] mapreduce.Job: Task Id : attempt_1489999688045_0001_m_000008_0, Status : FAILED
AttemptID:attempt_1489999688045_0001_m_000008_0 Timed out after 600 secs
2017-03-20 17:00:09,131 INFO  [main] mapreduce.Job: Task Id : attempt_1489999688045_0001_m_000012_0, Status : FAILED
AttemptID:attempt_1489999688045_0001_m_000012_0 Timed out after 600 secs

2017-03-20 16:48:27136信息[main]zookeeper.zookeeper:客户端环境：java.library.path=/usr/local/hadoop/lib/native
2017-03-20 16:48:27136信息[main]zookeeper.zookeeper:客户端环境：java.io.tmpdir=/var/folders/nl/f3lktfgn7jg46jycx21cfmr0000gn/T/
2017-03-20 16:48:27136信息[主]zookeeper.zookeeper:客户端环境：java.compiler=
2017-03-20 16:48:27137信息[主]zookeeper.zookeeper:客户端环境：os.name=Mac os X
2017-03-20 16:48:27137信息[主]zookeeper.zookeeper:客户端环境：os.arch=x86_64
2017-03-20 16:48:27137信息[主]zookeeper.zookeeper:客户端环境：os.version=10.12.3
2017-03-20 16:48:27137信息[主]zookeeper.zookeeper:客户端环境：user.name=haohui
2017-03-20 16:48:27137信息[主]zookeeper.zookeeper:客户端环境：user.home=/Users/haohui
2017-03-20 16:48:27138信息[主]zookeeper.zookeeper:客户端环境：user.dir=/Users/haohui
2017-03-20 16:48:27138信息[main]zookeeper.zookeeper:启动客户端连接，connectString=master:2181，node1:2181，node2:2181 sessionTimeout=30000 watcher=hconnection-0x3fc2959f0x0，quorum=master:2181，node2:2181，baseZNode=/hbase
2017-03-20 16:48:27157信息[主发送线程（主线程：2181）]zookeeper.ClientCnxn:打开到服务器主线程的套接字连接/10.211.55.2:2181。不会尝试使用SASL进行身份验证（未知错误）
2017-03-20 16:48:27188信息[主发送线程（主线程：2181）]zookeeper.ClientCnxn：已建立到主线程的套接字连接/10.211.55.2:2181，启动会话
2017-03-20 16:48:27200信息[主发送线程（主线程：2181）]zookeeper.ClientCnxn:在服务器主节点上完成会话建立/10.211.55.2:2181，会话ID=0x15AE6867A0001，协商超时=30000
2017-03-20 16:48:56396信息[main]配置。不推荐使用：io.bytes.per.checksum不推荐使用。而是使用dfs.bytes-per-checksum
2017-03-20 16:48:56441信息[主]客户端.连接管理器$HConnectionImplementation:关闭zookeeper会话ID=0x15AE6867A0001
2017-03-20 16:48:56450信息[主]zookeeper.zookeeper:会话：0x15AE6867A0001已关闭
2017-03-20 16:48:56450信息[主事件线程]zookeeper.ClientCnxn:EventThread关闭
2017-03-20 16:48:56524 INFO[main]client.RMProxy:在master/10.211.55.2:8032连接到ResourceManager
2017-03-20 16:48:56666信息[main]配置。不推荐使用：io.bytes.per.checksum不推荐使用。而是使用dfs.bytes-per-checksum
2017-03-20 16:48:58818信息[main]输入。文件输入格式：处理的总输入路径：1
2017-03-20 16:48:58873信息[main]mapreduce.JobSubmitter:拆分数：56
2017-03-20 16:48:58884信息[main]配置。不推荐使用：io.bytes.per.checksum不推荐使用。而是使用dfs.bytes-per-checksum
2017-03-20 16:48:59006信息[main]mapreduce.JobSubmitter:为作业提交令牌：job_1489999688045_0001
2017-03-20 16:48:59319信息[main]impl.yarclientimpl:提交的申请申请
2017-03-20 16:48:59370信息[main]mapreduce.作业：跟踪作业的url:http://master:8088/proxy/application_1489999688045_0001/
2017-03-20 16:48:59371信息[主]mapreduce.作业：正在运行的作业：作业
2017-03-20 16:49:09668信息[main]mapreduce.作业：作业作业_1489999688045_0001在uber模式下运行：false
2017-03-20 16:49:09670信息[主]地图还原。作业：地图0%还原0%
2017-03-20 17:00:09103信息[main]mapreduce.作业：任务Id:trust_1489999688045_0001_m_000009_0，状态：失败
AttemptID:尝试在600秒后超时
2017-03-20 17:00:09127信息[main]mapreduce.作业：任务Id:trust_1489999688045_0001_m_000011_0，状态：失败
AttemptID:尝试_1489999688045_0001_m_000011_0在600秒后超时
2017-03-20 17:00:09128信息[main]mapreduce.作业：任务Id:trust_1489999688045_0001_m_000010_0，状态：失败
AttemptID:尝试在600秒后超时
2017-03-20 17:00:09129信息[main]mapreduce.作业：任务Id:trust_1489999688045_0001_m_000013_0，状态：失败
AttemptID:尝试在600秒后超时
2017-03-20 17:00:09130信息[main]mapreduce.作业：任务Id:trust_1489999688045_0001_m_000008_0，状态：失败
AttemptID:尝试在600秒后超时
2017-03-20 17:00:09131信息[main]mapreduce.作业：任务Id:trust_1489999688045_0001_m_000012_0，状态：失败
AttemptID:尝试在600秒后超时

我不确定如何加快您的操作，因为它实际上取决于您的模式和数据。您可以在中获得有关行的优化设计的一些信息。 至于崩溃，您的作业可能会抛出超时异常，因为在批量步骤中长时间运行计算，而没有将进度报告回由ImportTsv实用程序调度的mapreduce作业中的纱线。您可以在mapred site.xml文件中增加超时时间：

<property>
  <name>mapred.task.timeout</name>
  <value>2000000</value> <!-- A value 2000000 equals 2000 secs -->
</property>

mapred.task.timeout
2000000

---

或者您可以将其设置为0，这将禁用作业的超时，但这被认为是一种不好的做法，因为您将有风险处理集群中的潜在僵尸。

好的，将

mapred.task.timeout

设置为

        <property>
            <name>yarn.nodemanager.resource.memory-mb</name>
            <value>4096</value>
        </property>
        <property>
            <name>yarn.nodemanager.resource.cpu-vcores</name>
            <value>2</value>
        </property>
        <property>
            <name>yarn.scheduler.minimum-allocation-mb</name>
            <value>4096</value>
        </property>

        <property>
            <name>yarn.app.mapreduce.am.resource.mb</name>>
            <value>4096</value>>
        </property>>
        <property>
            <name>yarn.app.mapreduce.am.command-opts</name>>
            <value>-Xmx3768m</value>>
        </property>>
        <property>
            <name>mapreduce.map.cpu.vcores</name>>
            <value>2</value>>
        </property>>
        <property>
            <name>mapreduce.reduce.cpu.vcores</name>>
            <value>2</value>>
        </property>>