在Mesos上运行k8s时，如何通过响应错误找到匹配的http请求？

我参考并尝试在同一本地

Ubuntu

上部署

k8s

和

Mesos

。执行以下命令：

$ km controller-manager \
  --master=${KUBERNETES_MASTER_IP}:8888 \
  --cloud-provider=mesos \
  --cloud-config=./mesos-cloud.conf  \
  --v=8 >controller.log 2>&1 &

从

控制器.log

：

$ more controller.log
I1210 05:33:13.570789   12082 controllermanager.go:217] Creating hostIP:hostPort endpoint controller
I1210 05:33:13.571253   12082 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1210 05:33:13.571294   12082 standalone.go:46] creating new standalone detector for &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16
.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.571469   12082 standalone.go:67] Detect()
I1210 05:33:13.571485   12082 standalone.go:69] spinning up asyc master detector poller
I1210 05:33:13.571501   12082 standalone.go:78] spawning asyc master detector listener
I1210 05:33:13.571608   12082 nodecontroller.go:132] Sending events to api server.
I1210 05:33:13.571706   12082 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1210 05:33:13.571729   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/services
I1210 05:33:13.571757   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.571772   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.571831   12082 standalone.go:80] waiting for polled to send updates
I1210 05:33:13.571986   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/replicationcontrollers
I1210 05:33:13.572010   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572026   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
E1210 05:33:13.571994   12082 controllermanager.go:152] Failed to start service controller: the cloud provider does not support TCP load ba
lancers.
I1210 05:33:13.572124   12082 round_trippers.go:222] GET http://16.187.250.141:8888/apis/extensions/v1beta1/daemonsets
I1210 05:33:13.572151   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572167   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572208   12082 persistentvolume_claim_binder_controller.go:346] Starting PersistentVolumeClaimBinder
I1210 05:33:13.572262   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572268   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572300   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572314   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572305   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572331   12082 plugins.go:265] Loaded volume plugin "kubernetes.io/host-path"
I1210 05:33:13.572371   12082 plugins.go:265] Loaded volume plugin "kubernetes.io/nfs"
I1210 05:33:13.572393   12082 persistentvolume_recycler_controller.go:211] Starting PersistentVolumeRecycler
I1210 05:33:13.572452   12082 resource_quota_controller.go:133] Resource quota controller queued all resource quota for full calculation of
 usage
I1210 05:33:13.572479   12082 client.go:77] Reloading cached Mesos state
I1210 05:33:13.572283   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572499   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572346   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572508   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572531   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572535   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572547   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572571   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572584   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572521   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572612   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572632   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.572662   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572676   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572684   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/resourcequotas
I1210 05:33:13.572702   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572714   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572756   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumeclaims
I1210 05:33:13.572551   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572791   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572815   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572329   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572846   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572858   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572864   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/serviceaccounts?fieldSelector=metadata.name%3Dde
fault
I1210 05:33:13.572869   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572885   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572892   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572777   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572948   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572167   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573010   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573019   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572834   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573072   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572876   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573120   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572641   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.573193   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573202   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.573286   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573304   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573313   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.576806   12082 standalone.go:214] Got mesos state, content length 1795
I1210 05:33:13.577013   12082 standalone.go:147] detected leadership change from '<nil>' to 'master@16.187.250.141:5050'
I1210 05:33:13.577098   12082 standalone.go:185] master leader poller sleeping for 29.994633799s
I1210 05:33:13.577118   12082 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:
5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.577218   12082 client.go:155] cloud master changed to '16.187.250.141:5050'
I1210 05:33:14.192579   12082 round_trippers.go:247] Response Status: 403 Forbidden in 619 milliseconds
I1210 05:33:14.192610   12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192621   12082 round_trippers.go:253]     Proxy-Connection: Keep-Alive
E1210 05:33:14.192639   12082 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
I1210 05:33:14.192667   12082 round_trippers.go:253]     Connection: Keep-Alive
I1210 05:33:14.192692   12082 round_trippers.go:253]     Content-Length: 606
I1210 05:33:14.192703   12082 round_trippers.go:253]     Cache-Control: no-cache
I1210 05:33:14.192712   12082 round_trippers.go:253]     Pragma: no-cache
I1210 05:33:14.192722   12082 round_trippers.go:253]     Content-Type: text/html; charset=utf-8
I1210 05:33:14.192765   12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.192800   12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192812   12082 round_trippers.go:253]     Pragma: no-cache
I1210 05:33:14.192823   12082 round_trippers.go:253]     Content-Type: text/html; charset=utf-8
I1210 05:33:14.192833   12082 round_trippers.go:253]     Proxy-Connection: Keep-Alive
I1210 05:33:14.192842   12082 round_trippers.go:253]     Connection: Keep-Alive
I1210 05:33:14.192851   12082 round_trippers.go:253]     Content-Length: 606
I1210 05:33:14.192861   12082 round_trippers.go:253]     Cache-Control: no-cache
I1210 05:33:14.192905   12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192935   12082 request.go:861] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192785   12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.193025   12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.193034   12082 request.go:861] Response Body: <HTML><HEAD>

$more controller.log
I1210 05:33:13.570789 12082 controllermanager.go:217]创建hostIP:hostPort端点控制器
I1210 05:33:13.571253 12082中云。go:75]新中云，master='16.187.250.141:5050'
I1210 05:33:13.571294 12082独立。转到：46]为&MasterInfo创建新的独立检测器{Id:*主机，Ip:*0，端口：*5050，Pid:*master@16
.187.250.141:5050，主机名：*16.187.250.141，版本：nil，地址：nil，XXX_未识别：[]，}
I1210 05:33:13.571469 12082独立。转到：67]检测（）
I1210 05:33:13.57148512082独立。转到：69]旋转asyc主探测器轮询器
I1210 05:33:13.5715012082独立。转到：78]生成asyc主检测器侦听器
I1210 05:33:13.571608 12082节点控制器。转到：132]向api服务器发送事件。
I1210 05:33:13.571706 12082独立。go:140]在“16.187.250.141:5050”进行主领导投票
I1210 05:33:13.571729 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/services
I1210 05:33:13.571757 12082往返行程。go:229]请求标题：
I1210 05:33:13.571772 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.571831 12082独立。转到：80]等待轮询发送更新
I1210 05:33:13.571986 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/replicationcontrollers
I1210 05:33:13.5720112082往返行程。go:229]请求标题：
I1210 05:33:13.572026 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
E1210 05:33:13.571994 12082 controllermanager.go:152]无法启动服务控制器：云提供程序不支持TCP加载
兰瑟斯。
I1210 05:33:13.572124 12082往返者。出发：222]出发http://16.187.250.141:8888/apis/extensions/v1beta1/daemonsets
I1210 05:33:13.5721112082往返者。go:229]请求标题：
I1210 05:33:13.57216712082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.57220812082持久卷声明绑定器控制器。转到：346]启动持久卷声明绑定器
I1210 05:33:13.57262 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572268 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.5723012082往返者。go:229]请求标题：
I1210 05:33:13.572314 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.57230512082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572331 12082插件。转到：265]加载的卷插件“kubernetes.io/host path”
I1210 05:33:13.572371 12082插件。转到：265]加载卷插件“kubernetes.io/nfs”
I1210 05:33:13.572393 12082持久卷回收器控制器。转到：211]启动持久卷回收器
I1210 05:33:13.572452 12082资源_配额_控制器。go:133]资源配额控制器将所有资源配额排入队列，以便完全计算
使用
I1210 05:33:13.572479 12082客户端。转到：77]重新加载缓存的Mesos状态
I1210 05:33:13.572283 12082往返者。go:229]请求标题：
I1210 05:33:13.572499 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572346 12082往返行程。go:229]请求标题：
I1210 05:33:13.572508 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.572531 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.572535 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572547 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572571 12082往返者。go:229]请求标题：
I1210 05:33:13.572584 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.572521 12082往返行程。go:229]请求标题：
I1210 05:33:13.572612 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.572632 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.572662 12082往返者。go:229]请求标题：
I1210 05:33:13.572676 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.572684 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/resourcequotas
I1210 05:33:13.572702 12082往返行程。go:229]请求标题：
I1210 05:33:13.572714 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.572756 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/persistentvolumeclaims
I1210 05:33:13.572551 12082往返者。go:229]请求标题：
I1210 05:33:13.572791 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.572815 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572329 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572846 12082往返者。go:229]请求标题：
I1210 05:33:13.572858 12082往返者。go:232]用户代理：km/v1.2.0（linux/amd64）kubernetes/c2cfcd5
I1210 05:33:13.572864 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/serviceaccounts?fieldSelector=metadata.name%3Dde
过错
I1210 05:33:13.572869 12082往返者。出发：222]出发http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572885 12082往返者。go:229]请求标题：
I1210 05:33:13.572892 12082往返脱扣器
lsof -p pid -P -n

sudo tcpdump -vv -A -s 0 'tcp port xxxx' -i em1 -w capture.pcap