Virtual machine 自动禁用/启用1节点ProxBox
我有一个包含3个节点的proxmox集群（node1、node2、node3），但两天前我开始看到集群如何丢失node2的图片。这个节点是活动的，它工作，虚拟机继续工作，我可以通过ssh和web访问它。所有节点都可以在网络上看到它。此问题会阻止VM复制和VM迁移。 登录节点2：Virtual machine 自动禁用/启用1节点ProxBox,virtual-machine,high-availability,proxmox,Virtual Machine,High Availability,Proxmox,我有一个包含3个节点的proxmox集群（node1、node2、node3），但两天前我开始看到集群如何丢失node2的图片。这个节点是活动的，它工作，虚拟机继续工作，我可以通过ssh和web访问它。所有节点都可以在网络上看到它。此问题会阻止VM复制和VM迁移。 登录节点2： Oct 28 08:08:00 node2 systemd[1]: Starting Proxmox VE replication runner... Oct 28 08:08:00 node2 pvesr[50099]
Oct 28 08:08:00 node2 systemd[1]: Starting Proxmox VE replication runner...
Oct 28 08:08:00 node2 pvesr[50099]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:08:01 node2 pvestatd[3594]: storage 'nfs-back' is not online
Oct 28 08:08:01 node2 corosync[3573]: [KNET ] rx: host: 4 link: 0 is up
Oct 28 08:08:01 node2 corosync[3573]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 28 08:08:01 node2 pvesr[50099]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:08:02 node2 pvesr[50099]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:08:03 node2 corosync[3573]: [TOTEM ] A new membership (2.bb9e) was formed. Members
Oct 28 08:08:03 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:03 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:03 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:03 node2 pvesr[50099]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:08:04 node2 pvesr[50099]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:08:05 node2 corosync[3573]: [TOTEM ] A new membership (2.bba2) was formed. Members
Oct 28 08:08:05 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:05 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:05 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:05 node2 pvesr[50099]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:08:06 node2 pvesr[50099]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:08:07 node2 corosync[3573]: [TOTEM ] A new membership (2.bba6) was formed. Members
Oct 28 08:08:07 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:07 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:07 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:07 node2 pvesr[50099]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:08:08 node2 pvesr[50099]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:08:09 node2 corosync[3573]: [TOTEM ] A new membership (2.bbaa) was formed. Members
Oct 28 08:08:09 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:09 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:09 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:09 node2 pvesr[50099]: error with cfs lock 'file-replication_cfg': no quorum!
Oct 28 08:08:09 node2 systemd[1]: pvesr.service: Main process exited, code=exited, status=13/n/a
Oct 28 08:08:09 node2 systemd[1]: pvesr.service: Failed with result 'exit-code'.
Oct 28 08:08:09 node2 systemd[1]: Failed to start Proxmox VE replication runner.
Oct 28 08:08:10 node2 pvestatd[3594]: storage 'nfs-back' is not online
Oct 28 08:08:11 node2 corosync[3573]: [TOTEM ] A new membership (2.bbae) was formed. Members
Oct 28 08:08:11 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:11 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:11 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:13 node2 corosync[3573]: [TOTEM ] A new membership (2.bbb2) was formed. Members
Oct 28 08:08:13 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:13 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:13 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:15 node2 corosync[3573]: [TOTEM ] A new membership (2.bbb6) was formed. Members
Oct 28 08:08:15 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:15 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:15 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:17 node2 corosync[3573]: [TOTEM ] A new membership (2.bbba) was formed. Members
Oct 28 08:08:17 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:17 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:17 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:19 node2 corosync[3573]: [TOTEM ] A new membership (2.bbbe) was formed. Members
Oct 28 08:08:19 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:19 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:19 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:21 node2 corosync[3573]: [TOTEM ] A new membership (2.bbc2) was formed. Members
Oct 28 08:08:21 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:21 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:21 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:23 node2 corosync[3573]: [TOTEM ] A new membership (2.bbc6) was formed. Members
Oct 28 08:08:23 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:23 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:23 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:25 node2 corosync[3573]: [TOTEM ] A new membership (2.bbca) was formed. Members
Oct 28 08:08:25 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:25 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:25 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:27 node2 corosync[3573]: [TOTEM ] A new membership (2.bbce) was formed. Members
Oct 28 08:08:27 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:27 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:27 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:27 node2 corosync[3573]: [KNET ] rx: host: 1 link: 0 is up
Oct 28 08:08:27 node2 corosync[3573]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 28 08:08:27 node2 corosync[3573]: [TOTEM ] A new membership (1.bbd2) was formed. Members joined: 1 4
Oct 28 08:08:27 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:27 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:27 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: members: 1/2737, 2/3565, 4/14943
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: starting data syncronisation
Oct 28 08:08:27 node2 pmxcfs[3565]: [status] notice: members: 1/2737, 2/3565, 4/14943
Oct 28 08:08:27 node2 pmxcfs[3565]: [status] notice: starting data syncronisation
Oct 28 08:08:27 node2 corosync[3573]: [QUORUM] This node is within the primary component and will provide service.
Oct 28 08:08:27 node2 corosync[3573]: [QUORUM] Members[3]: 1 2 4
Oct 28 08:08:27 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:27 node2 pmxcfs[3565]: [status] notice: node has quorum
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: received sync request (epoch 1/2737/00000CB2)
Oct 28 08:08:27 node2 pmxcfs[3565]: [status] notice: received sync request (epoch 1/2737/00000CDA)
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: received all states
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: leader is 1/2737
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: synced members: 1/2737, 4/14943
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: waiting for updates from leader
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: dfsm_deliver_queue: queue length 6
Oct 28 08:08:27 node2 pmxcfs[3565]: [status] notice: received all states
Oct 28 08:08:27 node2 pmxcfs[3565]: [status] notice: all data is up to date
Oct 28 08:08:27 node2 pmxcfs[3565]: [status] notice: dfsm_deliver_queue: queue length 3
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: update complete - trying to commit (got 8 inode updates)
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: all data is up to date
Oct 28 08:08:27 node2 pmxcfs[3565]: [dcdb] notice: dfsm_deliver_sync_queue: queue length 6
Oct 28 08:08:37 node2 pve-ha-crm[3627]: status change wait_for_quorum => slave
Oct 28 08:08:48 node2 corosync[3573]: [KNET ] link: host: 4 link: 0 is down
Oct 28 08:08:48 node2 corosync[3573]: [KNET ] link: host: 1 link: 0 is down
Oct 28 08:08:48 node2 corosync[3573]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 28 08:08:48 node2 corosync[3573]: [KNET ] host: host: 4 has no active links
Oct 28 08:08:48 node2 corosync[3573]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 28 08:08:48 node2 corosync[3573]: [KNET ] host: host: 1 has no active links
Oct 28 08:08:48 node2 corosync[3573]: [TOTEM ] Token has not been received in 1237 ms
Oct 28 08:08:49 node2 corosync[3573]: [TOTEM ] A processor failed, forming new configuration.
Oct 28 08:08:50 node2 pvestatd[3594]: storage 'nfs-back' is not online
Oct 28 08:08:51 node2 corosync[3573]: [TOTEM ] A new membership (2.bbd6) was formed. Members left: 1 4
Oct 28 08:08:51 node2 corosync[3573]: [TOTEM ] Failed to receive the leave message. failed: 1 4
Oct 28 08:08:51 node2 corosync[3573]: [CPG ] downlist left_list: 2 received
Oct 28 08:08:51 node2 pmxcfs[3565]: [dcdb] notice: members: 2/3565
Oct 28 08:08:51 node2 corosync[3573]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Oct 28 08:08:51 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:08:51 node2 pmxcfs[3565]: [status] notice: members: 2/3565
Oct 28 08:08:51 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:08:51 node2 pmxcfs[3565]: [status] notice: node lost quorum
Oct 28 08:08:51 node2 pmxcfs[3565]: [dcdb] crit: received write while not quorate - trigger resync
Oct 28 08:08:51 node2 pmxcfs[3565]: [dcdb] crit: leaving CPG group
Oct 28 08:08:51 node2 pmxcfs[3565]: [dcdb] notice: start cluster connection
Oct 28 08:08:51 node2 pmxcfs[3565]: [dcdb] crit: cpg_join failed: 14
Oct 28 08:08:51 node2 pmxcfs[3565]: [dcdb] crit: can't initialize service
Oct 28 08:08:53 node2 pve-ha-crm[3627]: status change slave => wait_for_quorum
Oct 28 08:08:57 node2 pmxcfs[3565]: [dcdb] notice: members: 2/3565
Oct 28 08:08:57 node2 pmxcfs[3565]: [dcdb] notice: all data is up to date
Oct 28 08:09:00 node2 pvestatd[3594]: storage 'nfs-back' is not online
Oct 28 08:09:00 node2 systemd[1]: Starting Proxmox VE replication runner...
Oct 28 08:09:00 node2 pvesr[51468]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:09:01 node2 pvesr[51468]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:09:02 node2 pvesr[51468]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:09:03 node2 pvesr[51468]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:09:04 node2 pvesr[51468]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:09:05 node2 pvesr[51468]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:09:06 node2 pvesr[51468]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:09:07 node2 pvesr[51468]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:09:08 node2 pvesr[51468]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:09:09 node2 pvesr[51468]: error with cfs lock 'file-replication_cfg': no quorum!
Oct 28 08:09:09 node2 systemd[1]: pvesr.service: Main process exited, code=exited, status=13/n/a
Oct 28 08:09:09 node2 systemd[1]: pvesr.service: Failed with result 'exit-code'.
Oct 28 08:09:09 node2 systemd[1]: Failed to start Proxmox VE replication runner.
Oct 28 08:09:10 node2 pvestatd[3594]: storage 'nfs-back' is not online
Oct 28 08:09:19 node2 corosync[3573]: [KNET ] rx: host: 1 link: 0 is up
Oct 28 08:09:19 node2 corosync[3573]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 28 08:09:20 node2 pvestatd[3594]: storage 'nfs-back' is not online
Oct 28 08:09:20 node2 corosync[3573]: [TOTEM ] Token has not been received in 1439 ms
Oct 28 08:09:22 node2 corosync[3573]: [TOTEM ] Token has not been received in 3089 ms
Oct 28 08:09:25 node2 corosync[3573]: [TOTEM ] A new membership (2.bbe2) was formed. Members
Oct 28 08:09:25 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:09:25 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:09:25 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:09:26 node2 corosync[3573]: [TOTEM ] Token has not been received in 1238 ms
Oct 28 08:09:27 node2 corosync[3573]: [TOTEM ] Token has not been received in 2888 ms
Oct 28 08:09:28 node2 corosync[3573]: [KNET ] rx: host: 4 link: 0 is up
Oct 28 08:09:28 node2 corosync[3573]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 28 08:09:28 node2 corosync[3573]: [TOTEM ] A new membership (2.bbf2) was formed. Members
Oct 28 08:09:28 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:09:28 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:09:28 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:09:28 node2 corosync[3573]: [TOTEM ] A new membership (1.bbf6) was formed. Members joined: 1 4
Oct 28 08:09:28 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:09:28 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:09:28 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:09:28 node2 pmxcfs[3565]: [dcdb] notice: members: 1/2737, 2/3565, 4/14943
Oct 28 08:09:28 node2 pmxcfs[3565]: [dcdb] notice: starting data syncronisation
Oct 28 08:09:28 node2 pmxcfs[3565]: [status] notice: members: 1/2737, 2/3565, 4/14943
Oct 28 08:09:28 node2 pmxcfs[3565]: [status] notice: starting data syncronisation
Oct 28 08:09:28 node2 corosync[3573]: [QUORUM] This node is within the primary component and will provide service.
Oct 28 08:09:28 node2 corosync[3573]: [QUORUM] Members[3]: 1 2 4
Oct 28 08:09:28 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:09:28 node2 pmxcfs[3565]: [status] notice: node has quorum
Oct 28 08:09:29 node2 pmxcfs[3565]: [dcdb] notice: received sync request (epoch 1/2737/00000CB4)
Oct 28 08:09:29 node2 pmxcfs[3565]: [status] notice: received sync request (epoch 1/2737/00000CDC)
Oct 28 08:09:29 node2 pmxcfs[3565]: [dcdb] notice: received all states
Oct 28 08:09:29 node2 pmxcfs[3565]: [dcdb] notice: leader is 1/2737
Oct 28 08:09:29 node2 pmxcfs[3565]: [dcdb] notice: synced members: 1/2737, 4/14943
Oct 28 08:09:29 node2 pmxcfs[3565]: [dcdb] notice: waiting for updates from leader
Oct 28 08:09:29 node2 pmxcfs[3565]: [status] notice: received all states
Oct 28 08:09:29 node2 pmxcfs[3565]: [status] notice: all data is up to date
Oct 28 08:09:29 node2 pmxcfs[3565]: [dcdb] notice: update complete - trying to commit (got 7 inode updates)
Oct 28 08:09:29 node2 pmxcfs[3565]: [dcdb] notice: all data is up to date
Oct 28 08:09:38 node2 pve-ha-crm[3627]: status change wait_for_quorum => slave
Oct 28 08:09:53 node2 corosync[3573]: [KNET ] link: host: 4 link: 0 is down
Oct 28 08:09:53 node2 corosync[3573]: [KNET ] link: host: 1 link: 0 is down
Oct 28 08:09:53 node2 corosync[3573]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 28 08:09:53 node2 corosync[3573]: [KNET ] host: host: 4 has no active links
Oct 28 08:09:53 node2 corosync[3573]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 28 08:09:53 node2 corosync[3573]: [KNET ] host: host: 1 has no active links
Oct 28 08:09:54 node2 corosync[3573]: [TOTEM ] Token has not been received in 1237 ms
Oct 28 08:09:55 node2 corosync[3573]: [TOTEM ] A processor failed, forming new configuration.
Oct 28 08:09:57 node2 corosync[3573]: [TOTEM ] A new membership (2.bbfa) was formed. Members left: 1 4
Oct 28 08:09:57 node2 corosync[3573]: [TOTEM ] Failed to receive the leave message. failed: 1 4
Oct 28 08:09:57 node2 corosync[3573]: [CPG ] downlist left_list: 2 received
Oct 28 08:09:57 node2 pmxcfs[3565]: [dcdb] notice: members: 2/3565
Oct 28 08:09:57 node2 corosync[3573]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Oct 28 08:09:57 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:09:57 node2 pmxcfs[3565]: [status] notice: members: 2/3565
Oct 28 08:09:57 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:09:57 node2 pmxcfs[3565]: [status] notice: node lost quorum
Oct 28 08:09:57 node2 pmxcfs[3565]: [dcdb] crit: received write while not quorate - trigger resync
Oct 28 08:09:57 node2 pmxcfs[3565]: [dcdb] crit: leaving CPG group
Oct 28 08:09:57 node2 pmxcfs[3565]: [dcdb] notice: start cluster connection
Oct 28 08:09:57 node2 pmxcfs[3565]: [dcdb] crit: cpg_join failed: 14
Oct 28 08:09:57 node2 pmxcfs[3565]: [dcdb] crit: can't initialize service
Oct 28 08:09:57 node2 pve-ha-lrm[3636]: unable to write lrm status file - unable to open file '/etc/pve/nodes/node2/lrm_status.tmp.3636' - Permission denied
Oct 28 08:10:00 node2 systemd[1]: Starting Proxmox VE replication runner...
Oct 28 08:10:00 node2 pvesr[52774]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:10:01 node2 pvestatd[3594]: storage 'nfs-back' is not online
Oct 28 08:10:01 node2 pvesr[52774]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:10:02 node2 pve-ha-crm[3627]: status change slave => wait_for_quorum
Oct 28 08:10:02 node2 pvesr[52774]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:10:03 node2 pmxcfs[3565]: [dcdb] notice: members: 2/3565
Oct 28 08:10:03 node2 pmxcfs[3565]: [dcdb] notice: all data is up to date
Oct 28 08:10:03 node2 pvesr[52774]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:10:04 node2 pvesr[52774]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:10:05 node2 pvesr[52774]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:10:06 node2 pvesr[52774]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:10:07 node2 pvesr[52774]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:10:08 node2 pvesr[52774]: trying to acquire cfs lock 'file-replication_cfg' ...
Oct 28 08:10:09 node2 pvesr[52774]: error with cfs lock 'file-replication_cfg': no quorum!
Oct 28 08:10:09 node2 systemd[1]: pvesr.service: Main process exited, code=exited, status=13/n/a
Oct 28 08:10:09 node2 systemd[1]: pvesr.service: Failed with result 'exit-code'.
Oct 28 08:10:09 node2 systemd[1]: Failed to start Proxmox VE replication runner.
Oct 28 08:10:10 node2 pvestatd[3594]: storage 'nfs-back' is not online
Oct 28 08:10:10 node2 corosync[3573]: [KNET ] rx: host: 4 link: 0 is up
Oct 28 08:10:10 node2 corosync[3573]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Oct 28 08:10:13 node2 corosync[3573]: [TOTEM ] A new membership (2.bbfe) was formed. Members
Oct 28 08:10:13 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:10:13 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:10:13 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:10:15 node2 corosync[3573]: [TOTEM ] A new membership (2.bc02) was formed. Members
Oct 28 08:10:15 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:10:15 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:10:15 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:10:17 node2 corosync[3573]: [TOTEM ] A new membership (2.bc06) was formed. Members
Oct 28 08:10:17 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:10:17 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:10:17 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:10:19 node2 corosync[3573]: [TOTEM ] A new membership (2.bc0a) was formed. Members
Oct 28 08:10:19 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:10:19 node2 corosync[3573]: [QUORUM] Members[1]: 2
Oct 28 08:10:19 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:10:20 node2 corosync[3573]: [KNET ] rx: host: 1 link: 0 is up
Oct 28 08:10:20 node2 corosync[3573]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 28 08:10:20 node2 corosync[3573]: [TOTEM ] A new membership (1.bc0e) was formed. Members joined: 1 4
Oct 28 08:10:20 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:10:20 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:10:20 node2 corosync[3573]: [CPG ] downlist left_list: 0 received
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: members: 1/2737, 2/3565, 4/14943
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: starting data syncronisation
Oct 28 08:10:20 node2 pmxcfs[3565]: [status] notice: members: 1/2737, 2/3565, 4/14943
Oct 28 08:10:20 node2 pmxcfs[3565]: [status] notice: starting data syncronisation
Oct 28 08:10:20 node2 corosync[3573]: [QUORUM] This node is within the primary component and will provide service.
Oct 28 08:10:20 node2 corosync[3573]: [QUORUM] Members[3]: 1 2 4
Oct 28 08:10:20 node2 corosync[3573]: [MAIN ] Completed service synchronization, ready to provide service.
Oct 28 08:10:20 node2 pmxcfs[3565]: [status] notice: node has quorum
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: received sync request (epoch 1/2737/00000CB6)
Oct 28 08:10:20 node2 pmxcfs[3565]: [status] notice: received sync request (epoch 1/2737/00000CDE)
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: received all states
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: leader is 1/2737
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: synced members: 1/2737, 4/14943
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: waiting for updates from leader
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: dfsm_deliver_queue: queue length 5
Oct 28 08:10:20 node2 pmxcfs[3565]: [status] notice: received all states
Oct 28 08:10:20 node2 pmxcfs[3565]: [status] notice: all data is up to date
Oct 28 08:10:20 node2 pmxcfs[3565]: [status] notice: dfsm_deliver_queue: queue length 6
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: update complete - trying to commit (got 7 inode updates)
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: all data is up to date
Oct 28 08:10:20 node2 pmxcfs[3565]: [dcdb] notice: dfsm_deliver_sync_queue: queue length 5
Oct 28 08:10:27 node2 pve-ha-crm[3627]: status change wait_for_quorum => slave
Oct 28 08:11:00 node2 systemd[1]: Starting Proxmox VE replication runner...
Oct 28 08:11:00 node2 systemd[1]: pvesr.service: Succeeded.
Oct 28 08:11:00 node2 systemd[1]: Started Proxmox VE replication runner.
Oct 28 08:12:00 node2 systemd[1]: Starting Proxmox VE replication runner...
Oct 28 08:12:00 node2 systemd[1]: pvesr.service: Succeeded.
Oct 28 08:12:00 node2 systemd[1]: Started Proxmox VE replication runner.
Oct 28 08:12:01 node2 sshd[55659]: Accepted publickey for root from 172.16.100.10 port 60110 ssh2: RSA SHA256:6w/SKpMwegZ2NibghYuUTJWUZRg7k+PO9d2eyex6Rm0
Oct 28 08:12:01 node2 sshd[55659]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 28 08:12:01 node2 systemd-logind[2958]: New session 38678 of user root.
Oct 28 08:12:01 node2 systemd[1]: Started Session 38678 of user root.
Oct 28 08:12:01 node2 sshd[55659]: Received disconnect from 172.16.100.10 port 60110:11: disconnected by user
Oct 28 08:12:01 node2 sshd[55659]: Disconnected from user root 172.16.100.10 port 60110
Oct 28 08:12:01 node2 sshd[55659]: pam_unix(sshd:session): session closed for user root
Oct 28 08:12:01 node2 systemd[1]: session-38678.scope: Succeeded.
Oct 28 08:12:01 node2 systemd-logind[2958]: Session 38678 logged out. Waiting for processes to exit.
Oct 28 08:12:01 node2 systemd-logind[2958]: Removed session 38678.
Oct 28 08:12:01 node2 sshd[55677]: Accepted publickey for root from 172.16.100.10 port 60116 ssh2: RSA SHA256:6w/SKpMwegZ2NibghYuUTJWUZRg7k+PO9d2eyex6Rm0
Oct 28 08:12:01 node2 sshd[55677]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 28 08:12:01 node2 systemd-logind[2958]: New session 38679 of user root.
Oct 28 08:12:01 node2 systemd[1]: Started Session 38679 of user root.
Oct 28 08:12:02 node2 zed[55796]: eid=99008 class=history_event pool_guid=0xE8B91311C244C8FC
Oct 28 10:19:10 node2 systemd[1]: Started PVE Local HA Resource Manager Daemon.
Oct 28 10:20:17 node2 pve-ha-lrm[51076]: unable to write lrm status file - unable to open file '/etc/pve/nodes/node2/lrm_status.tmp.51076' - Permission denied
Oct 28 10:21:20 node2 pve-ha-lrm[51076]: unable to write lrm status file - unable to open file '/etc/pve/nodes/node2/lrm_status.tmp.51076' - Permission denied
可能是什么？权限可能被拒绝，因为您的群集丢失了仲裁。如果丢失仲裁，文件将变为只读。当一个节点无法被其他节点访问时，仲裁将丢失
您应该查看
pvecm状态