Fatal编程技术网
  • hadoop/
  • WebHDFS在安全hadoop群集上不工作

WebHDFS在安全hadoop群集上不工作

hadoop curl

WebHDFS在安全hadoop群集上不工作,hadoop,curl,kerberos,webhdfs,Hadoop,Curl,Kerberos,Webhdfs,我正在尝试使用Kerberos保护我的HDP2 Hadoop集群 到目前为止，Hdfs、Hive、Hbase、色调蜂蜡和色调作业/任务浏览器工作正常；但是，色调的文件浏览器不工作，它会回答： WebHdfsException at /filebrowser/ AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] (error 500) Request Method: G

我正在尝试使用Kerberos保护我的HDP2 Hadoop集群

到目前为止，Hdfs、Hive、Hbase、色调蜂蜡和色调作业/任务浏览器工作正常；但是，色调的文件浏览器不工作，它会回答：

WebHdfsException at /filebrowser/
AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS] (error 500)
Request Method: GET
Request URL:    http://bt1svlmy:8000/filebrowser/
Django Version: 1.2.3
Exception Type: WebHdfsException
Exception Value:    
AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS] (error 500)
Exception Location: /usr/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py in _stats, line 208
Python Executable:  /usr/bin/python2.6
Python Version: 2.6.6
(...)

HTTP/1.1 500 Internal Server Error
Server: Apache-Coyote/1.1
Set-Cookie: hadoop.auth=u=theuser&p=theuser&t=simple&e=1412735529027&s=rQAfgMdExsQjx6N8cQ10JKWb2kM=; Path=/; Expires=Wed, 08-Oct-2014 02:32:09 GMT; HttpOnly
Content-Type: application/json
Transfer-Encoding: chunked
Date: Tue, 07 Oct 2014 16:32:09 GMT
Connection: close

{"RemoteException":{"message":"SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]","exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException"}}
我的
hue.ini
文件配置了所有
security\u enabled=true
和其他相关参数

我相信问题在于WebHDFS

我尝试了在上给出的curl命令

答复:

WebHdfsException at /filebrowser/
AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS] (error 500)
Request Method: GET
Request URL:    http://bt1svlmy:8000/filebrowser/
Django Version: 1.2.3
Exception Type: WebHdfsException
Exception Value:    
AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS] (error 500)
Exception Location: /usr/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py in _stats, line 208
Python Executable:  /usr/bin/python2.6
Python Version: 2.6.6
(...)

HTTP/1.1 403 Forbidden
Server: Apache-Coyote/1.1
Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
Content-Type: text/html;charset=utf-8
Content-Length: 1027
Date: Wed, 08 Oct 2014 06:55:51 GMT

<html><head><title>Apache Tomcat/6.0.37 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 403 - Anonymous requests are disallowed</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Anonymous requests are disallowed</u></p><p><b>description</b> <u>Access to the specified resource has been forbidden.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.37</h3></body></html>

HTTP/1.1 500 Internal Server Error
Server: Apache-Coyote/1.1
Set-Cookie: hadoop.auth=u=theuser&p=theuser&t=simple&e=1412735529027&s=rQAfgMdExsQjx6N8cQ10JKWb2kM=; Path=/; Expires=Wed, 08-Oct-2014 02:32:09 GMT; HttpOnly
Content-Type: application/json
Transfer-Encoding: chunked
Date: Tue, 07 Oct 2014 16:32:09 GMT
Connection: close

{"RemoteException":{"message":"SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]","exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException"}}
它回答：

WebHdfsException at /filebrowser/
AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS] (error 500)
Request Method: GET
Request URL:    http://bt1svlmy:8000/filebrowser/
Django Version: 1.2.3
Exception Type: WebHdfsException
Exception Value:    
AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS] (error 500)
Exception Location: /usr/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py in _stats, line 208
Python Executable:  /usr/bin/python2.6
Python Version: 2.6.6
(...)

HTTP/1.1 500 Internal Server Error
Server: Apache-Coyote/1.1
Set-Cookie: hadoop.auth=u=theuser&p=theuser&t=simple&e=1412735529027&s=rQAfgMdExsQjx6N8cQ10JKWb2kM=; Path=/; Expires=Wed, 08-Oct-2014 02:32:09 GMT; HttpOnly
Content-Type: application/json
Transfer-Encoding: chunked
Date: Tue, 07 Oct 2014 16:32:09 GMT
Connection: close

{"RemoteException":{"message":"SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]","exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException"}}
WebHDFS和curl之间似乎没有Kerberos协商

我期待着这样的事情：

HTTP/1.1 401 UnauthorizedContent-Type: text/html; charset=utf-8
WWW-Authenticate: Negotiate
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 307 TEMPORARY_REDIRECT
Content-Type: application/octet-stream
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: hadoop.auth="u=exampleuser&p=exampleuser@MYCOMPANY.COM&t=kerberos&e=1375144834763&s=iY52iRvjuuoZ5iYG8G5g12O2Vwo=";Path=/
Location: http://hadoopnamenode.mycompany.com:1006/webhdfs/v1/user/release/docexample/test.txt?op=OPEN&delegation=JAAHcmVsZWFzZQdyZWxlYXNlAIoBQCrfpdGKAUBO7CnRju3TbBSlID_osB658jfGfRpEt8-u9WHymRJXRUJIREZTIGRlbGVnYXRpb24SMTAuMjAuMTAwLjkxOjUwMDcw&offset=0
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 16
Server: Jetty(6.1.26)
A|1|2|3
B|4|5|6
你知道会出什么问题吗

在我的
hdfs site.xml中，每个节点上都有：


<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>

<property>
  <name>dfs.web.authentication.kerberos.principal</name>
  <value>HTTP/_HOST@MY-REALM.COM</value>
</property>

<property>
  <name>dfs.web.authentication.kerberos.keytab</name>
  <value>/etc/hadoop/conf/HTTP.keytab</value> <!-- path to the HTTP keytab -->
</property>




dfs.webhdfs.enabled
真的
dfs.web.authentication.kerberos.principal
HTTP/_HOST@MY-REALM.COM
dfs.web.authentication.kerberos.keytab
/etc/hadoop/conf/HTTP.keytab

看起来您不是访问WebHDFS（默认端口=50070），而是访问HttpFS（默认端口=14000），这是一个“普通”webapp，没有以相同的方式进行保护


WebHDFS url通常类似于
http://namenode:50070/webhdfs/v1
；尝试使用该参数修改hue.ini（建议使用WebHDFS而不是HttpFS）
但是，我有一个活动/备用配置，无法确定哪一个是namenode。。。如何处理？来自Cloudera doc：“WebHDF和HttpFS都使用HTTP REST API，因此它们完全可互操作，但色调必须配置为使用其中一种。对于HDFS HA部署，您必须使用HttpFS。”（）嗨，Arnaud，这很酷；我正在尝试使用Kerberos保护hadoop环境。你能详细告诉我你选择的过程/方式吗？好吧，那将是一个很长的答案，对于这个论坛来说太长了。简而言之，您必须在某处安装KDC，创建hadoop系统超级用户和凭据，并考虑将KDC与遗留企业LDAP系统互连以进行用户标识。我使用Ambari+一些手工制作的shell脚本来实际配置hadoop集群；告诉你细节。好的，阿诺，让我看看；我将在2台ubuntu机器上使用一个可用的hadoop集群，并从Ambari安装开始。