Hadoop 配置单元使用Talend将文件从HDFS加载到配置单元表失败

我正在使用talend 5.4/5.5连接到cdh 5.1。三节点集群

N1:CM，HIVEall服务，Datanode，Zookeeper。。。。等 N2:RM，数据节点 N3：数据节点

当我尝试将数据从hdfs加载到配置单元表时失败，而cli中的同一命令工作正常

hive> LOAD DATA  INPATH '/user/thor/test/rev_sub.txt' INTO TABLE revenue_subs;

当我使用tHiveLoad组件运行talend作业时，我遇到以下异常

[INFO ]: hive.metastore - Trying to connect to metastore with URI thrift://txwlcloud1:9083
[WARN ]: org.apache.hadoop.security.UserGroupInformation - No groups available for user thor
[INFO ]: hive.metastore - Waiting 1 seconds before next connection attempt.
[INFO ]: hive.metastore - Connected to metastore.
[ERROR]: org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException Line 1:17 Invalid path ''/user/thor/test/rev_sub.txt''
org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:17 Invalid path ''/user/thor/test/rev_sub.txt''
at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:148)
at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:229)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:459)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:355)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:82)
at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:129)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:209)
at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:154)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:191)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:197)
at big_data.hivejob_0_1.HIVEJob.tHiveLoad_1Process(HIVEJob.java:375)
at big_data.hivejob_0_1.HIVEJob.runJobInTOS(HIVEJob.java:645)
at big_data.hivejob_0_1.HIVEJob.main(HIVEJob.java:504)
Caused by: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status; Host Details : local host is: "TXWLHPW295/10.215.206.241"; destination host is: "txwlcloud2":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
at org.apache.hadoop.ipc.Client.call(Client.java:1241)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)

我为这个问题挣扎了一段时间

可能的原因是 1 jdbc驱动程序问题。我必须把jdbc驱动程序jar放在集群中的某个地方吗？还是已经存在了？ 2与远程元存储有关的一些事情

如果你们能指出负载失败的原因，这将是非常有帮助的

当我直奔>！连接jdbc:hive2://10.215.204.xyz:10000 thor org.apache.hive.jdbc.HiveDriver它正在返回正确的连接

谢谢，

---

Amit

你能发布你的工作截图并显示tHiveLoad组件的配置吗？谢谢你的回复@ydaetskcoR。我编辑了thiveload配置的帖子。我也必须把罐子放在任何地方。我为hadoop集群安装了CM，看起来它附带了必要的JAR。错误消息使它看起来像是试图转到/user/thor/test/rev_sub.txt，注意双精度“而不是”。你能检查一下你的报价是否正确吗？也许还值得切换到代码视图选项卡和Ctrl+F for/user/thor/test/rev_sub.txt。我唯一看到的是端口号。。。Talend 5.4或5.5是否也适用于cdh 5？