Warning: file_get_contents(/data/phpspider/zhask/data//catemap/9/java/394.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 将hdfs文件添加到配置单元并在udf中使用时,会出现错误_Java_Hadoop_Hive_User Defined Functions_Distributed Cache - Fatal编程技术网

Java 将hdfs文件添加到配置单元并在udf中使用时,会出现错误

Java 将hdfs文件添加到配置单元并在udf中使用时,会出现错误,java,hadoop,hive,user-defined-functions,distributed-cache,Java,Hadoop,Hive,User Defined Functions,Distributed Cache,有一个名为file.txt的文件存储配置,我想在我的UDF中使用它,如下面的代码所示 公共类ParseTraceUDF扩展了GenericUDF{ 私有瞬态ObjectInspector[]argumentOIs; 公共静态字符串dataFile=“file.txt”; 公共静态最终字符串SEP=“-”; public static HashSet targetTraces=new HashSet(); public static HashSet targets=new HashSet(); 公

有一个名为
file.txt
的文件存储配置,我想在我的UDF中使用它,如下面的代码所示

公共类ParseTraceUDF扩展了GenericUDF{
私有瞬态ObjectInspector[]argumentOIs;
公共静态字符串dataFile=“file.txt”;
公共静态最终字符串SEP=“-”;
public static HashSet targetTraces=new HashSet();
public static HashSet targets=new HashSet();
公共void ReadFile(){
试一试(
FileInputStream fis=新的FileInputStream(数据文件);
InputStreamReader isr=新的InputStreamReader(fis,StandardCharsets.UTF_8);
) {
BufferedReader br=新的BufferedReader(isr);
弦线;
字符串[]tmp;
整数长度;
而((line=br.readLine())!=null){
line=line.trim();
添加(行);
tmp=直线分割(SEP);
长度=tmp.length;
addAll(Arrays.asList(tmp).subList(0,长度-1));
}
}捕获(IOE异常){
e、 printStackTrace();
}
}
公共ObjectInspector初始化(ObjectInspector[]参数)
抛出UDFArgumentException{
如果(参数长度>2)
抛出新的udfargumentlengtheexception(“运算符最多接受2个参数”);
ReadFile();
argumentOIs=args;
返回PrimitiveObjectInspectorFactory.javaStringObjectInspector;
}
...
}
当我将文件放入本地文件系统并像这样在配置单元中写入时,它运行良好

addjar/***/***/rd.JAR;
添加/**/**/file.txt;
创建临时函数parse_trace_id为'com.hive.udf.PTIdUDF';
...
但是当我想像这样在hdfs上使用配置时,它会出错

addjar/***/***/rd.JAR;
添加文件hdfs://**/**/FILE.txt;
创建临时函数parse_trace_id为'com.hive.udf.PTIdUDF';
...
错误日志如下,我想知道原因是什么。如果我能得到帮助,我将不胜感激

java.io.FileNotFoundException: file.txt (No such file or directory)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at java.io.FileInputStream.<init>(FileInputStream.java:93)
    at com.vivo.hive.udf.ParseTraceIdUDF.ReadFile(ParseTraceIdUDF.java:35)
    at com.vivo.hive.udf.ParseTraceIdUDF.initialize(ParseTraceIdUDF.java:71)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:145)
    at org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:233)
    at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:959)
    at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1176)
    at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
    at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
    at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
    at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
    at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
    at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:193)
    at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:146)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10621)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10577)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3874)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3653)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9029)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8984)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9851)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9744)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10217)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10228)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10108)
    at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:558)
    at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1356)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1473)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1275)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:226)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:175)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:389)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:324)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:726)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:634)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: NullPointerException null
java.io.FileNotFoundException:file.txt(没有这样的文件或目录)
位于java.io.FileInputStream.open0(本机方法)
在java.io.FileInputStream.open(FileInputStream.java:195)
位于java.io.FileInputStream。(FileInputStream.java:138)
位于java.io.FileInputStream。(FileInputStream.java:93)
位于com.vivo.hive.udf.ParseTraceIdUDF.ReadFile(ParseTraceIdUDF.java:35)
在com.vivo.hive.udf.ParseTraceIdUDF.initialize上(ParseTraceIdUDF.java:71)
位于org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:145)
位于org.apache.hadoop.hive.ql.plan.exprnodegenericfundesc.newInstance(exprnodegenericfundesc.java:233)
位于org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXPathFuncExprNodedesc(TypeCheckProcFactory.java:959)
位于org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1176)
位于org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
位于org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.DispatchAndreReturn(DefaultGraphWalker.java:94)
在org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)上
位于org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
位于org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
位于org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:193)
位于org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:146)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.genAllExprNodeDesc(semanticalyzer.java:10621)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.genExprNodeDesc(semanticalyzer.java:10577)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.genSelectPlan(semanticalyzer.java:3874)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.genSelectPlan(semanticalyzer.java:3653)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.genPostGroupByBodyPlan(semanticalyzer.java:9029)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.genBodyPlan(semanticalyzer.java:8984)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.genPlan(semanticalyzer.java:9851)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.genPlan(semanticalyzer.java:9744)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.genOPTree(semanticalyzer.java:10217)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.analyzeInternal(semanticalyzer.java:10228)
位于org.apache.hadoop.hive.ql.parse.semanticalyzer.analyzeInternal(semanticalyzer.java:10108)
位于org.apache.hadoop.hive.ql.parse.basesemanticalyzer.analyze(basesemanticalyzer.java:223)
位于org.apache.hadoop.hive.ql.Driver.compile(Driver.java:558)
位于org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1356)
位于org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1473)
位于org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
位于org.apache.hadoop.hive.ql.Driver.run(Driver.java:1275)
位于org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:226)
位于org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:175)
位于org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:389)
位于org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:324)
位于org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:726)
位于org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
位于org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:634)
在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处
位于sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
在sun.reflect.DelegatingMethodAccessorImpl