Hadoop Spark应用程序中Kerberos中的HDFS写入问题

我有一个spark应用程序，它从Kafka读取数据并将数据写入HDFS。我的应用程序在几分钟内运行正常，但一段时间后，它开始出现以下错误并失败

2018-01-02 17:59:20 LeaseRenewer:username@nameservicename [WARN ] UserGroupInformation - PriviledgedActionException as:username@REALM_NAME (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]
2018-01-02 17:59:20 Spark Context Cleaner [INFO ] ContextCleaner - Cleaned accumulator 3480439
2018-01-02 17:59:20 LeaseRenewer:username@nameservicename [WARN ] Client - Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]
2018-01-02 17:59:20 LeaseRenewer:username@nameservicename [WARN ] UserGroupInformation - PriviledgedActionException as:username@REALM_NAME (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]
2018-01-02 17:59:20 Spark Context Cleaner [INFO ] ContextCleaner - Cleaned accumulator 3480438
2018-01-02 17:59:20 LeaseRenewer:username@nameservicename [INFO ] RetryInvocationHandler - Exception while invoking renewLease of class ClientNamenodeProtocolTranslatorPB over namenode1/10.12.2.2:8020. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]; Host Details : local host is: "edgenode/10.12.2.1"; destination host is: "namenode1":8020; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1508)
    at org.apache.hadoop.ipc.Client.call(Client.java:1441)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
    at com.sun.proxy.$Proxy41.renewLease(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:590)
    at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
    at com.sun.proxy.$Proxy42.renewLease(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:945)
    at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)
    at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)
    at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
    at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]
    at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
    at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)
    at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
    at org.apache.hadoop.ipc.Client.call(Client.java:1480)
    ... 16 more

---

如果有人知道此问题的解决方案，请告诉我。

问题已得到解决。我遇到此问题是因为其中一台计算机上缺少键表文件。

是否正在写入一些初始数据？启动期间是否指定了主体和键选项卡？或者你正在使用驱动程序上的本地票证缓存？是的，在几分钟内，或者我可以说20分钟内，它工作正常，并在hdfs中写入数据，但在那之后，它开始给出上述错误并失败。我的keytab和jaas登录配置在每个节点上都是本地可用的，我在启动期间提供principal和keytab。您是否使用Ranger/sentry，并且权限配置在飞行中更改，因此不再授予访问权限？另外，您是否只尝试使用本地票证缓存中的票证，即在spark submit上未指定票证？另外，在少量测试数据上尝试使用sparks本地执行模型。这有帮助吗？我在标准Kerberos消息列表中找不到

客户端凭据已被吊销

；这可能是一条特定的Active Directory消息，与自定义设置相关=>请与AD管理员联系。此外，您是否为Kafka使用者（通过指向键表的JAAS conf文件）和标准Spark到Hadoop接口（通过

--principal

/

--keytab

）注入了不同的Kerberos设置？Kerberos的Hadoop实现（基于Java impl.但是有一些黑客和肮脏的把戏）与Kerberos的标准Java实现不兼容。。。