24小时后运行spark流时出现HBase kerberos错误

spark streaming应用程序在Thread中运行，并使用Kerberos访问HBase。如果在24小时后运行，它将被挂起，并出现以下错误：

Caused by: GSSException: No valid credentials provided (Mechanism level: Requested start time is later than end time (11) - Requested start time is later than end time)
    at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
    ... 25 more
Caused by: KrbException: Requested start time is later than end time (11) - Requested start time is later than end time
    at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
    at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192)
    at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203)
    at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309)
    at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
    at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
    at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
    ... 28 more
Caused by: KrbException: Identifier doesn't match expected value (906)
    at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143)
    at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66)
    at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61)
    at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
    ... 34 more
Kerberos username [yarn]:

---

这是因为在krb5.conf中，您的

default_life

和

ticket_life

设置为24小时。尝试将其增加到24小时以上。尝试传递

--principal

和

--keytab

参数。Spark应以这种方式自动更新Kerberos票证。如果您将继续遇到问题，请添加“spark.hadoop.fs.hdfs.impl.disable.cache”：“true”。

val ugi:UserGroupInformation = UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal,keytab)
ugi.checkTGTAndReloginFromKeytab()
p1 = ugi.doAs(new PrivilegedAction[Iterator[Row]] {
  override def run(): Iterator[Row] = {
      dataprocess() // error occurs in hbase access in here
    }
  }
)