Ubuntu 16.04在PIG grunt shell中运行存储操作时崩溃
每次我在PIG grunt shell中触发LOAD命令和STORE命令时。我的系统在PIG的MapReduce阶段一开始就崩溃了 在MapReduce模式下启动PIG grunt shell之前,我触发了以下命令Ubuntu 16.04在PIG grunt shell中运行存储操作时崩溃,ubuntu,hadoop,apache-pig,hadoop2,Ubuntu,Hadoop,Apache Pig,Hadoop2,每次我在PIG grunt shell中触发LOAD命令和STORE命令时。我的系统在PIG的MapReduce阶段一开始就崩溃了 在MapReduce模式下启动PIG grunt shell之前,我触发了以下命令 start-dfs.sh start-yarn.sh mr-jobhistory-daemon.sh start historyserver 以下是上述操作的日志: 16/08/11 10:56:32 WARN util.NativeCodeLoader: Unable
start-dfs.sh
start-yarn.sh
mr-jobhistory-daemon.sh start historyserver
以下是上述操作的日志:
16/08/11 10:56:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-namenode-devesh-ThinkPad-L412.out
localhost: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-devesh-ThinkPad-L412.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-devesh-ThinkPad-L412.out
16/08/11 10:56:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-devesh-ThinkPad-L412.out
localhost: starting nodemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-devesh-ThinkPad-L412.out
starting historyserver, logging to /home/hadoop/hadoop/logs/mapred-hadoop-historyserver-devesh-ThinkPad-L412.out
在这之后,我开始吃猪壳。以下是清管器壳体的启动日志:
16/08/11 10:58:07 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
16/08/11 10:58:07 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
16/08/11 10:58:07 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2016-08-11 10:58:07,102 [main] INFO org.apache.pig.Main - Apache Pig version 0.16.0 (r1746530) compiled Jun 01 2016, 23:10:49
2016-08-11 10:58:07,103 [main] INFO org.apache.pig.Main - Logging error messages to: /home/hadoop/pig_1470893287101.log
2016-08-11 10:58:07,145 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/hadoop/.pigbootup not found
2016-08-11 10:58:07,845 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-08-11 10:58:07,876 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-08-11 10:58:07,876 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-08-11 10:58:07,876 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost:9000
2016-08-11 10:58:08,427 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-default-a50ec406-53e1-42d7-bb8c-ff49c88e9118
2016-08-11 10:58:08,427 [main] WARN org.apache.pig.PigServer - ATS is disabled since yarn.timeline-service.enabled set to false
在此之后,我在grunt shell中触发以下命令
data = LOAD '/pig/data.txt' USING PigStorage(',');
STORE data INTO '/pig/dataout.txt' USING PigStorage(',');
在此之后,当清管器开始其MapReduce阶段并重新启动时,我的系统崩溃。另外,我正在伪分布式模式下运行hadoop
我现在使用的是联想Thinkpad L412和4GB内存以及Intel i5处理器
操作系统详细信息:Ubuntu 16.04 64位版本
内核版本:4.4.0-31-generic Hadoop版本 Hadoop 2.7.2
Subversion-r b165c4fe8a74265c792ce23f546c64604acf0e41
詹金斯于2016-01-26T00:08Z编译
使用protoc 2.5.0编译
来自校验和为D0FDA26633FA762BF87EC759EBE689C的源
此命令是使用/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.7.2.jar运行的 清管器版本详细信息 Apache Pig版本0.16.0(r1746530)
编译于2016年6月1日23:10:49
提前感谢。您在任何地方都有错误吗?到目前为止,您发布的代码片段只包含警告。-一些一般性问题:1。您的集群是否可以正常工作(能否在hive中运行
count(*)
)2。猪一般会工作吗?(你能做些什么,比如描述和转储吗?或者处理不同的输入文件)3。你使用猪和/或不使用tez有关系吗?4.我不确定你所说的伪分布式模式是什么意思,但是hadoop在其他模式下工作吗?也许它与这个bug有关:你在任何地方都有错误吗?到目前为止,您发布的代码片段只包含警告。-一些一般性问题:1。您的集群是否可以正常工作(能否在hive中运行count(*)
)2。猪一般会工作吗?(你能做些什么,比如描述和转储吗?或者处理不同的输入文件)3。你使用猪和/或不使用tez有关系吗?4.不确定您所说的伪分布式模式是什么意思,但hadoop在其他模式下工作吗?可能与此bug有关: