并发docker yocto构建导致docker挂起_Docker_Jenkins_Yocto

并发docker yocto构建导致docker挂起

docker jenkins yocto

并发docker yocto构建导致docker挂起,docker,jenkins,yocto,Docker,Jenkins,Yocto,我正在Jenkins Azure动态代理中运行yocto构建。我已经将代理配置为执行这些构建，但当我设置Jenkins为每个代理使用多个执行器时，我发现docker有时会挂起 09:30:53 u20-a683f0 does not seem to be running inside a container 09:30:54 $ docker run -t -d -u 1001:1002 -v ${WORKSPACE}:/data/meta-domino -e SHARED_MOUNT_PA

我正在Jenkins Azure动态代理中运行yocto构建。我已经将代理配置为执行这些构建，但当我设置Jenkins为每个代理使用多个执行器时，我发现docker有时会挂起

09:30:53  u20-a683f0 does not seem to be running inside a container
09:30:54  $ docker run -t -d -u 1001:1002 -v ${WORKSPACE}:/data/meta-domino -e SHARED_MOUNT_PATH -v ${SHARED_MOUNT_PATH}${CACHE_WORKING_DIR}:${SHARED_MOUNT_PATH}${CACHE_WORKING_DIR} -v ${SHARED_MOUNT_PATH}${DOWNLOADS_WORKING_DIR}:${SHARED_MOUNT_PATH}${DOWNLOADS_WORKING_DIR} -w /home/jslave/workspace/ps-438-rightsizing-dynamic-agent-5 -v /home/jslave/workspace/ps-438-rightsizing-dynamic-agent-5:/home/jslave/workspace/ps-438-rightsizing-dynamic-agent-5:rw,z -v /home/jslave/workspace/ps-438-rightsizing-dynamic-agent-5-tmp:/home/jslave/workspace/ps-438-rightsizing-dynamic-agent-5-tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** myregistry.azurecr.io/livia-yocto-builder:0.6.2 cat
09:33:54  ERROR: Timeout after 180 seconds

当这种情况发生时，已经有另一个执行者在代理上运行一些docker构建步骤。有时我看到呼叫堆栈

[ 7492.572146] INFO: task dockerd:2768 blocked for more than 120 seconds.
[ 7492.576685]       Not tainted 4.15.0-1113-azure #126~16.04.1-Ubuntu
[ 7492.580161] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7492.584920] dockerd         D    0  2768      1 0x00000000
[ 7492.584923] Call Trace:
[ 7492.584931]  __schedule+0x3d6/0x8b0
[ 7492.584932]  schedule+0x36/0x80
[ 7492.584934]  rwsem_down_write_failed+0x1fc/0x390
[ 7492.584937]  call_rwsem_down_write_failed+0x17/0x30
[ 7492.584939]  ? call_rwsem_down_write_failed+0x17/0x30
[ 7492.584940]  down_write+0x2d/0x40
[ 7492.584943]  sync_inodes_sb+0x9b/0x2b0
[ 7492.584945]  __sync_filesystem+0x1b/0x60
[ 7492.584945]  sync_filesystem+0x3c/0x50
[ 7492.584949]  ovl_sync_fs+0x37/0x50 [overlay]
[ 7492.584950]  __sync_filesystem+0x33/0x60
[ 7492.584951]  sync_filesystem+0x3c/0x50
[ 7492.584954]  generic_shutdown_super+0x27/0x120
[ 7492.584955]  kill_anon_super+0x12/0x20
[ 7492.584955]  deactivate_locked_super+0x48/0x80
[ 7492.584956]  deactivate_super+0x5a/0x60
[ 7492.584958]  cleanup_mnt+0x3f/0x80
[ 7492.584959]  __cleanup_mnt+0x12/0x20
[ 7492.584962]  task_work_run+0x8a/0xb0
[ 7492.584964]  exit_to_usermode_loop+0xc4/0xd0
[ 7492.584965]  do_syscall_64+0x1af/0x1e0
[ 7492.584967]  entry_SYSCALL_64_after_hwframe+0x41/0xa6
[ 7492.584968] RIP: 0033:0x5570f753ff30
[ 7492.584969] RSP: 002b:000000c422baa360 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
[ 7492.584970] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00005570f753ff30
[ 7492.584970] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000c43008c360
[ 7492.584971] RBP: 000000c422baa3c0 R08: 0000000000000000 R09: 0000000000000000
[ 7492.584971] R10: 0000000000000000 R11: 0000000000000202 R12: ffffffffffffffff
[ 7492.584972] R13: 000000000000000a R14: 0000000000000009 R15: 0000000000000055
[ 7492.584974] INFO: task dockerd:2789 blocked for more than 120 seconds.
[ 7492.587461]       Not tainted 4.15.0-1113-azure #126~16.04.1-Ubuntu
[ 7492.591110] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7492.595526] dockerd         D    0  2789      1 0x00000000

我认为这是某种docker并发问题，与两个容器同时使用覆盖文件系统有关

我尝试过抵消容器构建的启动（这并没有解决问题），查看journalctl、syslog和kern.log文件，但没有收集到任何明显的线索……我可以看到一些与管道破裂相关的错误，但我认为这是因为Jenkins在容器处理完后停止容器

May 26 06:49:26 u20-4bea00 dockerd[1295]: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
May 26 06:49:26 u20-4bea00 dockerd[1295]: time="2021-05-26T06:49:26.140551381Z" level=error msg="Handler for GET /containers/8ffe49d082653f4177ec9a41c56ecd2c0a4874557a1cf1c18c3e365137b714f5/json returned error: write unix /run/docker.sock->@: write: broken pipe"
May 26 06:49:26 u20-4bea00 dockerd[1295]: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
May 26 06:56:13 u20-4bea00 dockerd[1295]: time="2021-05-26T06:56:13.965795692Z" level=info msg="Container 277f6befc932e0fdb4ac652949d2f6deeb5361593208bc082bb7ef10dfd61b79 failed to exit within 1 seconds of signal 15 - using the force"
May 26 06:56:18 u20-4bea00 dockerd[1295]: time="2021-05-26T06:56:18.538936014Z" level=info msg="ignoring event" container=277f6befc932e0fdb4ac652949d2f6deeb5361593208bc082bb7ef10dfd61b79 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 26 06:59:18 u20-4bea00 dockerd[1295]: time="2021-05-26T06:59:18.506977973Z" level=info msg="Container 32993389d58854c2f05273b6eb7fa2c2a0043f5d409e1df9d2efc21859894e8c failed to exit within 1 seconds of signal 15 - using the force"
May 26 06:59:41 u20-4bea00 dockerd[1295]: time="2021-05-26T06:59:41.956941220Z" level=info msg="ignoring event" container=32993389d58854c2f05273b6eb7fa2c2a0043f5d409e1df9d2efc21859894e8c module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 26 07:01:55 u20-4bea00 dockerd[1295]: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11)
May 26 07:01:55 u20-4bea00 dockerd[1295]: time="2021-05-26T07:01:55.367219564Z" level=error msg="Handler for GET /containers/277f6befc932e0fdb4ac652949d2f6deeb5361593208bc082bb7ef10dfd61b79/json returned error: write unix /run/docker.sock->@: write: broken pipe"
May 26 07:01:55 u20-4bea00 dockerd[1295]: time="2021-05-26T07:01:55.367663378Z" level=error msg="Handler for GET /containers/277f6befc932e0fdb4ac652949d2f6deeb5361593208bc082bb7ef10dfd61b79/json returned error: write unix /run/docker.sock->@: write: broken pipe"
May 26 07:01:55 u20-4bea00 dockerd[1295]: time="2021-05-26T07:01:55.367956887Z" level=error msg="Handler for GET /containers/277f6befc932e0fdb4ac652949d2f6deeb5361593208bc082bb7ef10dfd61b79/json returned error: write unix /run/docker.sock->@: write: broken pipe"
May 26 07:01:55 u20-4bea00 dockerd[1295]: time="2021-05-26T07:01:55.368414302Z" level=error msg="Handler for GET /containers/277f6befc932e0fdb4ac652949d2f6deeb5361593208bc082bb7ef10dfd61b79/json returned error: write unix /run/docker.sock->@: write: broken pipe"
May 26 07:01:55 u20-4bea00 dockerd[1295]: time="2021-05-26T07:01:55.368781714Z" level=error msg="Handler for GET /containers/277f6befc932e0fdb4ac652949d2f6deeb5361593208bc082bb7ef10dfd61b79/json returned error: write unix /run/docker.sock->@: write: broken pipe"
May 26 07:01:55 u20-4bea00 dockerd[1295]: time="2021-05-26T07:01:55.368935419Z" level=error msg="Handler for GET /containers/277f6befc932e0fdb4ac652949d2f6deeb5361593208bc082bb7ef10dfd61b79/json returned error: write unix /run/docker.sock->@: write: broken pipe"

集装箱277f6befc932e0fdb4ac652949d2f6deeb5361593208bc082bb7ef10dfd61b79和32993389d58854c2f05273b6eb7fa2c2a0043f5d409e1df9d2efc21859894e8c是两个在Jenkins日志中报告这些超时的集装箱

如何防止docker上吊