Python mrjob bad--在Hadoop集群上使用make_runner时出现步骤错误
我试图以编程方式运行简单的wordcount示例,但我无法使代码在hadoop集群上运行 test_job.py中的作业: mr_job_test.py中的跑步者: 我可以在本地运行此代码(使用Python mrjob bad--在Hadoop集群上使用make_runner时出现步骤错误,python,hadoop,mrjob,Python,Hadoop,Mrjob,我试图以编程方式运行简单的wordcount示例,但我无法使代码在hadoop集群上运行 test_job.py中的作业: mr_job_test.py中的跑步者: 我可以在本地运行此代码(使用inline选项),但在hadoop上我得到了: > Traceback (most recent call last): File "mr_job_tester.py", line 17, > in <module> > print test_runner(ar
inline
选项),但在hadoop上我得到了:
> Traceback (most recent call last): File "mr_job_tester.py", line 17,
> in <module>
> print test_runner(args, input_dir) File "mr_job_tester.py", line 8, in test_runner
> runner.run() File "/usr/local/lib/python2.7/dist-packages/mrjob/runner.py", line 458, in
> run
> self._run() File "/usr/local/lib/python2.7/dist-packages/mrjob/hadoop.py", line 239, in
> _run
> self._run_job_in_hadoop() File "/usr/local/lib/python2.7/dist-packages/mrjob/hadoop.py", line 295, in
> _run_job_in_hadoop
> for step_num in xrange(self._num_steps()): File "/usr/local/lib/python2.7/dist-packages/mrjob/runner.py", line 742, in
> _num_steps
> return len(self._get_steps()) File "/usr/local/lib/python2.7/dist-packages/mrjob/runner.py", line 721, in
> _get_steps
> raise ValueError("Bad --steps response: \n%s" % stdout) ValueError: Bad --steps response:
>回溯(最后一次调用):文件“mr_job_tester.py”,第17行,
>在
>打印test_runner(args,input_dir)文件“mr_job_tester.py”,第8行,在test_runner中
>runner.run()文件“/usr/local/lib/python2.7/dist-packages/mrjob/runner.py”,第458行,在
>跑
>self._run()文件“/usr/local/lib/python2.7/dist-packages/mrjob/hadoop.py”,第239行,在
>_run
>self.\u运行hadoop()文件“/usr/local/lib/python2.7/dist packages/mrjob/hadoop.py”中的“作业”,第295行,在
>在hadoop中运行作业
>对于xrange(self._num_steps())中的步骤数:文件“/usr/local/lib/python2.7/dist packages/mrjob/runner.py”,第742行,在
>_num_步数
>返回len(self.\u get_steps())文件“/usr/local/lib/python2.7/dist packages/mrjob/runner.py”,第721行,在
>_获得_步骤
>raise VALUERROR(“错误--步骤响应:\n%s”%stdout)VALUE错误:错误--步骤响应:
()mrjob提交作业文件并在mapper和reducer中远程执行作业文件的方式,使得以下行必须位于作业声明文件中:
if __name__ == "__main__":
MRWordFreqCount.run()
> Traceback (most recent call last): File "mr_job_tester.py", line 17,
> in <module>
> print test_runner(args, input_dir) File "mr_job_tester.py", line 8, in test_runner
> runner.run() File "/usr/local/lib/python2.7/dist-packages/mrjob/runner.py", line 458, in
> run
> self._run() File "/usr/local/lib/python2.7/dist-packages/mrjob/hadoop.py", line 239, in
> _run
> self._run_job_in_hadoop() File "/usr/local/lib/python2.7/dist-packages/mrjob/hadoop.py", line 295, in
> _run_job_in_hadoop
> for step_num in xrange(self._num_steps()): File "/usr/local/lib/python2.7/dist-packages/mrjob/runner.py", line 742, in
> _num_steps
> return len(self._get_steps()) File "/usr/local/lib/python2.7/dist-packages/mrjob/runner.py", line 721, in
> _get_steps
> raise ValueError("Bad --steps response: \n%s" % stdout) ValueError: Bad --steps response:
if __name__ == "__main__":
MRWordFreqCount.run()