Google cloud platform 将文件名参数传递到Dataproc Hadoop作业

我有一个Hadoop作业，我正试图在GCPDataproc集群上运行。作业采用一个参数，即文件名，用于配置作业的某些方面。我似乎找不到使用gcloud CLI成功执行此操作的方法（尽管我可以在Airflow中成功运行此作业）。我试过使用一个本地文件、一个Google存储中的文件和集群本身上的文件

以下是我工作中读取文件的代码：

# local file
$ gcloud dataproc jobs submit hadoop --project my-project --region global --cluster my-cluster --jar gs://path/to/hadoop_job.jar -- props_test
Job [00653af3afaf40ea8227d5a56e3de458] submitted.
Waiting for job output...
20/01/17 19:29:55 INFO job.JobDriver: props_test
20/01/17 19:29:55 ERROR job.JobDriver: Unable to read file
Exception in thread "main" java.lang.NullPointerException
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
    at com.google.common.base.Splitter.split(Splitter.java:371)
    at job.JobDriver.parseHadoopConfig(JobDriver.java:52)
    at job.JobDriver.run(JobDriver.java:101)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at job.JobDriver.main(JobDriver.java:45)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
    at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunJarShim.main(HadoopRunJarShim.java:12)
ERROR: (gcloud.dataproc.jobs.submit.hadoop) Job [00653af3afaf40ea8227d5a56e3de458] failed with error:
Job failed with message [Exception in thread "main" java.lang.NullPointerException]. Additional details can be found at '. . .'.

# Google Storage file
$ gcloud dataproc jobs submit hadoop --project my-project --region global --cluster my-cluster --jar gs://path/to/hadoop_job.jar -- gs://path/to/props_test
Job [8ec4a775a0fb4e31b77967d99280fd6c] submitted.
Waiting for job output...
20/01/17 19:30:15 INFO job.JobDriver: gs://path/to/props_test
20/01/17 19:30:15 ERROR job.JobDriver: Unable to read file
Exception in thread "main" java.lang.NullPointerException
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
    at com.google.common.base.Splitter.split(Splitter.java:371)
    at job.JobDriver.parseHadoopConfig(JobDriver.java:52)
    at job.JobDriver.run(JobDriver.java:101)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at job.JobDriver.main(JobDriver.java:45)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
    at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunJarShim.main(HadoopRunJarShim.java:12)
ERROR: (gcloud.dataproc.jobs.submit.hadoop) Job [8ec4a775a0fb4e31b77967d99280fd6c] failed with error:
Job failed with message [Exception in thread "main" java.lang.NullPointerException]. Additional details can be found at '. . .'.

# File on Dataproc cluster
$ gcloud dataproc jobs submit hadoop --project my-project --region global --cluster my-cluster --jar gs://path/to/hadoop_job.jar -- file:///path/to/props_test
Job [fda035c5159642c294a47385d5ebb85f] submitted.
Waiting for job output...
20/01/17 19:33:23 INFO job.JobDriver: props_test
20/01/17 19:33:23 ERROR job.JobDriver: Unable to read file
Exception in thread "main" java.lang.NullPointerException
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
    at com.google.common.base.Splitter.split(Splitter.java:371)
    at job.JobDriver.parseHadoopConfig(JobDriver.java:52)
    at job.JobDriver.run(JobDriver.java:101)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at job.JobDriver.main(JobDriver.java:45)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
    at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunJarShim.main(HadoopRunJarShim.java:12)
ERROR: (gcloud.dataproc.jobs.submit.hadoop) Job [fda035c5159642c294a47385d5ebb85f] failed with error:
Job failed with message [Exception in thread "main" java.lang.NullPointerException]. Additional details can be found at '. . .'.

File infle=null；
Properties appProps=新属性（）；
infle=新文件（args[0]）；
try（FileInputStream infirestream=newfileinputstream（infie））{
附件荷载（填充梁）；
}捕获（IOE异常）{
日志错误（“无法读取文件”）；
}

下面是我运行命令的尝试（其中props_test是文件）：

---

请先尝试在GCS中暂存文件，然后通过

--files

arg按完全限定名传递，然后按args中的文件名传递：

gcloud dataproc jobs submit hadoop ... \
    --files gs://path/to/file.ini \
    -- arg1 file.ini arg3