Hive 如何为Kerberized Hadoop集群配置ApacheNIFI

Hive 如何为Kerberized Hadoop集群配置ApacheNIFI,hive,kerberos,cloudera,hortonworks-data-platform,apache-nifi,Hive,Kerberos,Cloudera,Hortonworks Data Platform,Apache Nifi,我让ApacheNifi独立运行,工作正常。但是,当我试图设置ApacheNIFI以访问Hive或HDFS时,KerberizedClouderaHadoop集群。我遇到麻烦了 有人可以指导我设置HDFS/Hive/HBase(使用Kerberos)的文档吗 这是我在nifi.properties中给出的配置 # kerberos # nifi.kerberos.krb5.file=/etc/krb5.conf nifi.kerberos.service.principal=pseeram@JU

我让ApacheNifi独立运行,工作正常。但是,当我试图设置ApacheNIFI以访问Hive或HDFS时,KerberizedClouderaHadoop集群。我遇到麻烦了

有人可以指导我设置HDFS/Hive/HBase(使用Kerberos)的文档吗

这是我在nifi.properties中给出的配置

# kerberos #
nifi.kerberos.krb5.file=/etc/krb5.conf
nifi.kerberos.service.principal=pseeram@JUNIPER.COM
nifi.kerberos.keytab.location=/uhome/pseeram/learning/pseeram.keytab
nifi.kerberos.authentication.expiration=10 hours

我引用了各种链接,如,但没有一个是有用的。 (因为下面的链接说NiFi 0.7.1版本有问题,所以我尝试了NiFi 1.1.0版本。我也有同样的痛苦经历)

以下是我获取日志的错误:

ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.hive.SelectHiveQL 
org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://ddas1106a:10000/innovate: Peer indicated failure: Unsupported mechanism type PLAIN)
    at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:292) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
    at sun.reflect.GeneratedMethodAccessor191.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]
    at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177) ~[na:na]
    at com.sun.proxy.$Proxy83.getConnection(Unknown Source) ~[na:na]
    at org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
    at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.jar:1.1.0]
    at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.jar:1.1.0]
    at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.jar:1.1.0]
    at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.jar:1.1.0]
    at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.jar:1.1.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_51]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_51]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://ddas1106a:10000/innovate: Peer indicated failure: Unsupported mechanism type PLAIN)
    at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549) ~[commons-dbcp-1.4.jar:1.4]
    at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) ~[commons-dbcp-1.4.jar:1.4]
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[commons-dbcp-1.4.jar:1.4]
    at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:288) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
    ... 18 common frames omitted
Caused by: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://ddas1106a:10000/innovate: Peer indicated failure: Unsupported mechanism type PLAIN
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:231) ~[hive-jdbc-1.2.1.jar:1.2.1]
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) ~[hive-jdbc-1.2.1.jar:1.2.1]
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[hive-jdbc-1.2.1.jar:1.2.1]
    at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) ~[commons-dbcp-1.4.jar:1.4]
    at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) ~[commons-dbcp-1.4.jar:1.4]
    at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556) ~[commons-dbcp-1.4.jar:1.4]
    at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545) ~[commons-dbcp-1.4.jar:1.4]
    ... 21 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: Unsupported mechanism type PLAIN
    at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199) ~[hive-exec-1.2.1.jar:1.2.1]
    at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307) ~[hive-exec-1.2.1.jar:1.2.1]
    at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[hive-exec-1.2.1.jar:1.2.1]
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204) ~[hive-jdbc-1.2.1.jar:1.2.1]
    ... 27 common frames omitted


WARN [NiFi Web Server-29] o.a.nifi.dbcp.hive.HiveConnectionPool HiveConnectionPool[id=278beb67-0159-1000-cffa-8c8534c285c8] Configuration does not have security enabled, Keytab and Principal will be ignored
ERROR[Timer-Driven Process Thread-7]o.a.nifi.processors.hive.SelectHiveQL
org.apache.nifi.processor.exception.ProcessException:org.apache.commons.dbcp.SQLNestedException:无法创建PoolableConnectionFactory(无法使用JDBC Uri打开客户端传输:JDBC:hive2://ddas1106a:10000/innovate:Peer指示的故障:不支持的机制类型PLAIN)
在org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:292)~[nifi-hive-processors-1.1.0.jar:1.1.0]
at sun.reflect.GeneratedMethodAccessor191.invoke(未知源)~[na:na]
在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)~[na:1.8.051]
在java.lang.reflect.Method.invoke(Method.java:497)~[na:1.8.0_51]
在org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177)~[na:na]
在com.sun.proxy.$Proxy83.getConnection(未知源)~[na:na]
在org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158)~[nifi-hive-processors-1.1.0.jar:1.1.0]
在org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)[nifi-api-1.1.0.jar:1.1.0]
位于org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)[nifi-framework-core-1.1.0.jar:1.1.0]
在org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)[nifi-framework-core-1.1.0.jar:1.1.0]
在org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)[nifi-framework-core-1.1.0.jar:1.1.0]
位于org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)[nifi-framework-core-1.1.0.jar:1.1.0]
在java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[na:1.8.051]
在java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[na:1.8.051]
在java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[na:1.8.051]
在java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[na:1.8.051]
位于java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[na:1.8.051]
在java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[na:1.8.051]
在java.lang.Thread.run(Thread.java:745)[na:1.8.0_51]
原因:org.apache.commons.dbcp.SQLNestedException:无法创建PoolableConnectionFactory(无法使用JDBC Uri打开客户端传输:JDBC:hive2://ddas1106a:10000/innovate:Peer指示的故障:不支持的机制类型PLAIN)
在org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)~[commons-dbcp-1.4.jar:1.4]
在org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)~[commons-dbcp-1.4.jar:1.4]
在org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)~[commons-dbcp-1.4.jar:1.4]
在org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:288)~[nifi-hive-processors-1.1.0.jar:1.1.0]
... 省略18个公共框架
原因:java.sql.SQLException:无法使用JDBC Uri打开客户端传输:JDBC:hive2://ddas1106a:10000/innovate:Peer指示故障:不支持的机制类型普通
在org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:231)~[hive-jdbc-1.2.1.jar:1.2.1]
在org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)~[hive-jdbc-1.2.1.jar:1.2.1]
在org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)~[hive-jdbc-1.2.1.jar:1.2.1]
在org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)~[commons-dbcp-1.4.jar:1.4]
在org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)~[commons-dbcp-1.4.jar:1.4]
在org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)~[commons-dbcp-1.4.jar:1.4]
在org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)~[commons-dbcp-1.4.jar:1.4]
... 省略21个公共帧
原因:org.apache.thrift.transport.ttTransportException:对等指示故障:不支持的机制类型
在org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199)~[hive-exec-1.2.1.jar:1.2.1]
在org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)~[hive-exec-1.2.1.jar:1.2.1]
在org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)~[hive-exec-1.2.1.jar:1.2.1]
在org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)~[hive-jdbc-1.2.1.jar:1.2.1]
... 省略了27个公共框架
警告[NiFi Web Server-29]o.a.NiFi.dbcp.hive.HiveConnectionPool HiveConnectionPool[id=278beb67-0159-1000-cffa-8c8534c285c8]配置未启用安全性,将忽略键表和主体

您在nifi.properties文件中添加的内容对于Kerberizing nifi集群非常有用。为了访问kerberized hadoop集群,您需要提供适当的