OpenStack过云节点'；反省总是超时的
我正在使用Rhosp16.1在RHEL8.2环境中部署Openstack。 对于这些过程，我使用了RHOSP 16.1的官方RedHat文档： 我已经成功安装了Undercloud，但在对Overcloud节点进行内省时遇到了错误。 在内省（使用--provide选项）之后，节点应该处于“可用”状态，但是在我们的执行过程中，它保持在“可用”状态。 已达到内省重试限制。错误是“内省超时”，如Mistral日志中所示。 你知道是什么导致了这个错误吗 我尝试了这些对策，但都无济于事：OpenStack过云节点'；反省总是超时的,openstack,openstack-nova,openstack-neutron,rhel8,tripleo,Openstack,Openstack Nova,Openstack Neutron,Rhel8,Tripleo,我正在使用Rhosp16.1在RHEL8.2环境中部署Openstack。 对于这些过程，我使用了RHOSP 16.1的官方RedHat文档： 我已经成功安装了Undercloud，但在对Overcloud节点进行内省时遇到了错误。 在内省（使用--provide选项）之后，节点应该处于“可用”状态，但是在我们的执行过程中，它保持在“可用”状态。 已达到内省重试限制。错误是“内省超时”，如Mistral日志中所示。 你知道是什么导致了这个错误吗 我尝试了这些对策，但都无济于事： 禁用seli
- 禁用selinux
- 重新创建过云节点（通过virsh）
- 确认可以从云下（通过ipmitool）访问和控制过云节点
undercloud.confovercloud stackenv.json(undercloud) [stack@undercloud ~]$ cat undercloud.conf
[DEFAULT]
container_images_file = /home/stack/containers-prepare-parameter.yaml
deployment_user = stack
enabled_hardware_types = ipmi,redfish,ilo,idrac,manual-management
generate_service_certificate = false
inspection_interface = br-ctlplane
ipxe_enabled = true
local_interface = ens4
local_ip = 10.10.10.10/24
local_subnet = ctlplane-subnet
overcloud_domain_name = overcloud.domain
subnets = ctlplane-subnet
undercloud_hostname = undercloud.home
undercloud_log_file = install-undercloud.log
[ctlplane-subnet]
cidr = 10.10.10.0/24
dhcp_end = 10.10.10.100
dhcp_start = 10.10.10.20
gateway = 10.10.10.10
inspection_iprange = 10.10.10.101,10.10.10.150
masquerade = true
(undercloud) [stack@undercloud ~]$ cat overcloud-stackenv.json
{
"nodes": [
{
"arch": "x86_64",
"disk": "80",
"memory": "12288",
"name": "new-overcloud-compute",
"pm_user": "admin",
"pm_addr": "10.10.10.10",
"pm_password": "password",
"pm_port": "6001",
"pm_type": "pxe_ipmitool",
"mac": [
"52:54:00:b4:c6:f0"
],
"cpu": "2",
"capabilities": "profile:compute,boot_option:local"
},
{
"arch": "x86_64",
"disk": "80",
"memory": "12288",
"name": "new-overcloud-controller",
"pm_user": "admin",
"pm_addr": "10.10.10.10",
"pm_password": "password",
"pm_port": "6002",
"pm_type": "pxe_ipmitool",
"mac": [
"52:54:00:b4:c6:f8"
],
"cpu": "2",
"capabilities": "profile:control,boot_option:local"
}
]
}
pm_typeipmitoolvbmc列表的输出(undercloud) [stack@undercloud ~]$ vbmc list
+--------------------------+---------+-------------+------+
| Domain name | Status | Address | Port |
+--------------------------+---------+-------------+------+
| new-overcloud-compute | running | 10.10.10.10 | 6001 |
| new-overcloud-controller | running | 10.10.10.10 | 6002 |
+--------------------------+---------+-------------+------+
(undercloud) [stack@undercloud ~]$ openstack overcloud node import --introspect --provide overcloud-stackenv.json
Waiting for messages on queue 'tripleo' with no timeout.
2 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID 2b4db183-7bbf-488c-8754-54174f0ba47e
Successfully registered node UUID 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e
Waiting for introspection to finish...
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node attempt failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
Introspection of node attempt failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
Retrying 2 nodes that failed introspection. Attempt 1 of 3
Introspection of node attempt failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
Introspection of node attempt failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
Retrying 2 nodes that failed introspection. Attempt 2 of 3
Introspection of node attempt failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
Introspection of node attempt failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
Retrying 2 nodes that failed introspection. Attempt 3 of 3
Introspection of node attempt failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
Introspection of node attempt failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
Retry limit reached with 2 nodes still failing introspection
Introspection completed with errors: Retry limit reached with 2 nodes still failing introspection
sys:1: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.10.10.10', 60614)>
sys:1: ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.10.10.10', 48436)>
(undercloud) [stack@undercloud ~]$ openstack baremetal node list
+--------------------------------------+--------------------------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------------------+---------------+-------------+--------------------+-------------+
| 2b4db183-7bbf-488c-8754-54174f0ba47e | new-overcloud-compute | None | power on | manageable | False |
| 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e | new-overcloud-controller | None | power on | manageable | False |
+--------------------------------------+--------------------------+---------------+-------------+--------------------+-------------+
(undercloud) [stack@undercloud ~]$ nova service-list
+--------------------------------------+----------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+----------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| a0e54463-0afa-49b0-bc1b-2af14ad7eb63 | nova-conductor | undercloud.overcloud.domain | internal | enabled | up | 2020-09-28T07:39:51.000000 | - | False |
| 88f4bf62-e052-4e3a-8114-f6d492001cf6 | nova-scheduler | undercloud.overcloud.domain | internal | enabled | up | 2020-09-28T07:39:48.000000 | - | False |
| ef9ccd89-f74e-4628-a1fc-4f4aed8bb422 | nova-compute | undercloud.overcloud.domain | nova | enabled | up | 2020-09-28T07:39:50.000000 | - | False |
+--------------------------------------+----------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
(undercloud) [stack@undercloud ~]$ nova hypervisor-list
+--------------------------------------+--------------------------------------+-------+---------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+--------------------------------------+-------+---------+
| 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e | 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e | up | enabled |
| 2b4db183-7bbf-488c-8754-54174f0ba47e | 2b4db183-7bbf-488c-8754-54174f0ba47e | up | enabled |
+--------------------------------------+--------------------------------------+-------+---------+
(undercloud) [stack@undercloud ~]$ openstack baremetal node list
+--------------------------------------+--------------------------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------------------+---------------+-------------+--------------------+-------------+
| 2b4db183-7bbf-488c-8754-54174f0ba47e | new-overcloud-compute | None | power on | manageable | False |
| 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e | new-overcloud-controller | None | power on | manageable | False |
+--------------------------------------+--------------------------+---------------+-------------+--------------------+-------------+
(undercloud) [stack@undercloud ~]$ nova service-list
+--------------------------------------+----------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+----------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| a0e54463-0afa-49b0-bc1b-2af14ad7eb63 | nova-conductor | undercloud.overcloud.domain | internal | enabled | up | 2020-09-28T07:39:51.000000 | - | False |
| 88f4bf62-e052-4e3a-8114-f6d492001cf6 | nova-scheduler | undercloud.overcloud.domain | internal | enabled | up | 2020-09-28T07:39:48.000000 | - | False |
| ef9ccd89-f74e-4628-a1fc-4f4aed8bb422 | nova-compute | undercloud.overcloud.domain | nova | enabled | up | 2020-09-28T07:39:50.000000 | - | False |
+--------------------------------------+----------------+-----------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
(undercloud) [stack@undercloud ~]$ nova hypervisor-list
+--------------------------------------+--------------------------------------+-------+---------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+--------------------------------------+-------+---------+
| 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e | 8aa29d4b-dd52-42a7-9b25-05d0999fbe3e | up | enabled |
| 2b4db183-7bbf-488c-8754-54174f0ba47e | 2b4db183-7bbf-488c-8754-54174f0ba47e | up | enabled |
+--------------------------------------+--------------------------------------+-------+---------+
[root@undercloud containers]# grep -iE "(fail|error|warn|denied)" mistral/engine.log
...
2020-09-28 13:28:19.014 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% $.get('status') = "FAILED" %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:19.019 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% "Workflow failed due to message status. Status:{} Message:{}".format($.get('status'), $.get('message')) %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:19.026 6 DEBUG mistral.workflow.direct_workflow [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Found commands: [Fail [workflow=tripleo.messaging.v1.send]] _find_next_commands_for_task /usr/lib/python3.6/site-packages/mistral/workflow/direct_workflow.py:157
2020-09-28 13:28:19.058 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Workflow 'tripleo.messaging.v1.send' [RUNNING -> ERROR, msg=Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:2b4db183-7bbf-488c-8754-54174f0ba47e.] (execution_id=ff5cf8ee-52f6-4b3c-b4a3-0a9ec220c52a)
2020-09-28 13:28:19.111 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=ff5cf8ee-52f6-4b3c-b4a3-0a9ec220c52a, result=Result [data=None, error=Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:2b..., cancel=False]]
2020-09-28 13:28:19.146 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Task 'send_message' (c4b3466b-34b1-4889-a84a-f38a8ec913db) [RUNNING -> ERROR, msg=Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:2b4db183-7bbf-488c-8754-54174f0ba47e.] (execution_id=5e0c77f6-9f54-4fa7-bbfe-908ceab17e7e)
2020-09-28 13:28:19.284 6 WARNING oslo_db.sqlalchemy.utils [-] Unique keys not in sort_keys. The sorting order may be unstable.
2020-09-28 13:28:19.330 6 INFO workflow_trace [-] Workflow 'tripleo.baremetal.v1._introspect' [RUNNING -> ERROR, msg=Failure caused by error in tasks: send_message
send_message [task_ex_id=c4b3466b-34b1-4889-a84a-f38a8ec913db] -> Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
[wf_ex_id=ff5cf8ee-52f6-4b3c-b4a3-0a9ec220c52a, idx=0]: Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:2b4db183-7bbf-488c-8754-54174f0ba47e.
2020-09-28 13:28:19.349 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=5e0c77f6-9f54-4fa7-bbfe-908ceab17e7e, result=Result [data=None, error=Failure caused by error in tasks: send_message
2020-09-28 13:28:20.126 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=0d02fd62-adf9-4db8-a942-df47c7236e65, result=Result [data=None, error=None, cancel=False]]
2020-09-28 13:28:20.148 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Action 'std.noop' (0d02fd62-adf9-4db8-a942-df47c7236e65)(task=check_status) [RUNNING -> SUCCESS, Result [data=None, error=None, cancel=False]]
2020-09-28 13:28:20.192 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% $.get('status') = "FAILED" %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:20.196 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% "Workflow failed due to message status. Status:{} Message:{}".format($.get('status'), $.get('message')) %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:20.203 6 DEBUG mistral.workflow.direct_workflow [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Found commands: [Fail [workflow=tripleo.messaging.v1.send]] _find_next_commands_for_task /usr/lib/python3.6/site-packages/mistral/workflow/direct_workflow.py:157
2020-09-28 13:28:20.243 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Workflow 'tripleo.messaging.v1.send' [RUNNING -> ERROR, msg=Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.] (execution_id=83375607-678b-4654-acc9-a8c36edace4b)
2020-09-28 13:28:20.275 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=83375607-678b-4654-acc9-a8c36edace4b, result=Result [data=None, error=Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:8a..., cancel=False]]
2020-09-28 13:28:20.299 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Task 'send_message' (e339574e-2cd8-4c0a-9d26-d5826d90bdbc) [RUNNING -> ERROR, msg=Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.] (execution_id=39407fb5-354f-4989-88e5-cf627b12f2d6)
2020-09-28 13:28:20.389 6 WARNING oslo_db.sqlalchemy.utils [-] Unique keys not in sort_keys. The sorting order may be unstable.
2020-09-28 13:28:20.434 6 INFO workflow_trace [-] Workflow 'tripleo.baremetal.v1._introspect' [RUNNING -> ERROR, msg=Failure caused by error in tasks: send_message
send_message [task_ex_id=e339574e-2cd8-4c0a-9d26-d5826d90bdbc] -> Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
[wf_ex_id=83375607-678b-4654-acc9-a8c36edace4b, idx=0]: Workflow failed due to message status. Status:FAILED Message:Introspection of node attempt failed:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e.
2020-09-28 13:28:20.455 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=39407fb5-354f-4989-88e5-cf627b12f2d6, result=Result [data=None, error=Failure caused by error in tasks: send_message
2020-09-28 13:28:21.336 6 INFO workflow_trace [req-02272ea3-36ec-49a7-9042-e7917691b87c 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Task 'introspect_nodes' (79017565-04d9-499a-93f9-cd15aca38765) [RUNNING -> ERROR, msg=One or more actions had failed.] (execution_id=43188fcc-e4a0-489e-9032-a33fd8ae6902)
2020-09-28 13:28:21.356 6 DEBUG mistral.workflow.direct_workflow [req-02272ea3-36ec-49a7-9042-e7917691b87c 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Found commands: [Run task [workflow=tripleo.baremetal.v1.introspect, task=max_retry_attempts_reached, waif_flag=False, triggered_by=[{'task_id': '79017565-04d9-499a-93f9-cd15aca38765', 'event': 'on-error'}]]] _find_next_commands_for_task /usr/lib/python3.6/site-packages/mistral/workflow/direct_workflow.py:157
2020-09-28 13:28:21.418 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=0c23a7fb-af8e-44f5-b0c4-a9f0094d599c, result=Result [data=None, error=None, cancel=False]]
2020-09-28 13:28:21.434 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Action 'std.noop' (0c23a7fb-af8e-44f5-b0c4-a9f0094d599c)(task=max_retry_attempts_reached) [RUNNING -> SUCCESS, Result [data=None, error=None, cancel=False]]
2020-09-28 13:28:21.460 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% 'Retry limit reached with {0} nodes still failing introspection'.format($.failed_introspection.len()) %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:21.517 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% { introspected_nodes => $.get('introspected_nodes', []), failed_introspection => $.get('failed_introspection', []) } %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:21.546 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting workflow [name=tripleo.messaging.v1.send, input={'queue_name': 'tripleo', 'type': 'tripleo.baremetal.v1.introspect', 'status': 'FAILED', 'message...]
...
2020-09-28 13:28:23.874 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% $.get('status') = "FAILED" %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:23.881 6 DEBUG mistral.expressions.yaql_expression [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Starting to evaluate YAQL expression. [expression='<% "Workflow failed due to message status. Status:{} Message:{}".format($.get('status'), $.get('message')) %>'] evaluate /usr/lib/python3.6/site-packages/mistral/expressions/yaql_expression.py:175
2020-09-28 13:28:23.887 6 DEBUG mistral.workflow.direct_workflow [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Found commands: [Fail [workflow=tripleo.messaging.v1.send]] _find_next_commands_for_task /usr/lib/python3.6/site-packages/mistral/workflow/direct_workflow.py:157
2020-09-28 13:28:23.918 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Workflow 'tripleo.messaging.v1.send' [RUNNING -> ERROR, msg=Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection] (execution_id=89c35cc0-c179-490e-ad7c-b50bc2a1c36c)
2020-09-28 13:28:23.945 6 INFO mistral.engine.engine_server [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'on_action_complete'[action_ex_id=89c35cc0-c179-490e-ad7c-b50bc2a1c36c, result=Result [data=None, error=Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still ..., cancel=False]]
2020-09-28 13:28:23.966 6 INFO workflow_trace [req-b0904acf-a4e2-41bb-a82c-8d4e4e7c68de 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Task 'send_message' (c7218062-acc2-4baa-be01-ad7ebf5fef27) [RUNNING -> ERROR, msg=Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection] (execution_id=43188fcc-e4a0-489e-9032-a33fd8ae6902)
2020-09-28 13:28:24.052 6 WARNING oslo_db.sqlalchemy.utils [-] Unique keys not in sort_keys. The sorting order may be unstable.
2020-09-28 13:28:24.099 6 INFO workflow_trace [-] Workflow 'tripleo.baremetal.v1.introspect' [RUNNING -> ERROR, msg=Failure caused by error in tasks: send_message
send_message [task_ex_id=c7218062-acc2-4baa-be01-ad7ebf5fef27] -> Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection
[wf_ex_id=89c35cc0-c179-490e-ad7c-b50bc2a1c36c, idx=0]: Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection
[root@undercloud containers]# grep -iE "(fail|error|warn|denied)" mistral/executor.log
2020-09-28 12:06:52.602 6 WARNING tripleo_common.utils.nodes [req-331aca0d-d9b8-4a7c-8836-3fcdc9eea72b 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Replacing deprecated driver pxe_ipmitool with the hardware type ipmi, please update your inventory
2020-09-28 12:06:54.870 6 WARNING tripleo_common.utils.nodes [req-331aca0d-d9b8-4a7c-8836-3fcdc9eea72b 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Replacing deprecated driver pxe_ipmitool with the hardware type ipmi, please update your inventory
...
2020-09-28 12:06:58.866 6 DEBUG mistral.executors.executor_server [req-331aca0d-d9b8-4a7c-8836-3fcdc9eea72b 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'run_action'[action_ex_id=299c5f9c-ecd0-4fd1-9879-bbd4ec501b7a, action_cls_str=mistral.actions.openstack.actions.IronicAction, action_cls_attrs={'client_method_name': 'node.get'}, params={'node_id': '2b4db183-7bbf-488c-8754-54174f0ba47e', 'fields': ['provision_state', 'last_error']}, timeout=1200] run_action /usr/lib/python3.6/site-packages/mistral/executors/executor_server.py:94
2020-09-28 12:07:02.854 6 DEBUG mistral.executors.executor_server [req-fda60dfa-6dfb-41b3-b7c8-ff0baa8ff9bc 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'run_action'[action_ex_id=515881f8-c2e1-4b35-a53e-36ba4bd4c431, action_cls_str=mistral.actions.openstack.actions.IronicAction, action_cls_attrs={'client_method_name': 'node.get'}, params={'node_id': '2b4db183-7bbf-488c-8754-54174f0ba47e', 'fields': ['provision_state', 'last_error']}, timeout=1200] run_action /usr/lib/python3.6/site-packages/mistral/executors/executor_server.py:94
2020-09-28 12:07:03.063 6 DEBUG mistral.executors.executor_server [req-fda60dfa-6dfb-41b3-b7c8-ff0baa8ff9bc 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Received RPC request 'run_action'[action_ex_id=51e88369-a9d1-4469-b714-b8580c5be64d, action_cls_str=mistral.actions.openstack.actions.IronicAction, action_cls_attrs={'client_method_name': 'node.get'}, params={'node_id': '8aa29d4b-dd52-42a7-9b25-05d0999fbe3e', 'fields': ['provision_state', 'last_error']}, timeout=1200] run_action /usr/lib/python3.6/site-packages/mistral/executors/executor_server.py:94
2020-09-28 12:27:18.382 6 WARNING mistral.actions.openstack.base [req-fda60dfa-6dfb-41b3-b7c8-ff0baa8ff9bc 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Traceback (most recent call last):
...
2020-09-28 13:28:15.618 6 WARNING mistral.actions.openstack.base [req-02272ea3-36ec-49a7-9042-e7917691b87c 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] Traceback (most recent call last):
ironic_inspector_client.v1.WaitTimeoutError: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
: ironic_inspector_client.v1.WaitTimeoutError: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
2020-09-28 13:28:15.619 6 WARNING mistral.executors.default_executor [req-02272ea3-36ec-49a7-9042-e7917691b87c 51b4a4f00e7c402ab3671cbbb83efab9 d12e223fba6945e6ac286da962231bcb - default default] The action raised an exception [action_ex_id=7aefe54e-46c7-41f1-abfe-cd8dca9336c9, msg='BaremetalIntrospectionAction.wait_for_finish failed: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']', action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{'client_method_name': 'wait_for_finish'}', params='{'uuids': ['2b4db183-7bbf-488c-8754-54174f0ba47e'], 'max_retries': 120, 'retry_interval': 10}']: mistral.exceptions.ActionException: BaremetalIntrospectionAction.wait_for_finish failed: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor Traceback (most recent call last):
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py", line 117, in run
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor result = method(**self._kwargs_for_run)
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/ironic_inspector_client/v1.py", line 265, in wait_for_finish
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor "of nodes %s") % new_active_node_ids)
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor ironic_inspector_client.v1.WaitTimeoutError: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor During handling of the above exception, another exception occurred:
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor Traceback (most recent call last):
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/mistral/executors/default_executor.py", line 133, in _do_run_action
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor result = action.run(action_ctx)
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py", line 130, in run
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor (self.__class__.__name__, self.client_method_name, str(e))
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor mistral.exceptions.ActionException: BaremetalIntrospectionAction.wait_for_finish failed: Timeout while waiting for introspection of nodes ['2b4db183-7bbf-488c-8754-54174f0ba47e']
2020-09-28 13:28:15.619 6 ERROR mistral.executors.default_executor
[root@undercloud containers]# grep -iE "(fail|error|warn|denied)" nova/nova-compute.log
2020-09-28 12:07:15.103 6 ERROR nova.compute.manager [req-76b53cba-540c-4069-bb12-85b5f2b71f14 - - - - -] No compute node record for host undercloud.overcloud.domain: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host undercloud.overcloud.domain could not be found.
2020-09-28 12:07:15.143 6 WARNING nova.compute.resource_tracker [req-76b53cba-540c-4069-bb12-85b5f2b71f14 - - - - -] No compute node record for undercloud.overcloud.domain:8aa29d4b-dd52-42a7-9b25-05d0999fbe3e: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host undercloud.overcloud.domain could not be found.
2020-09-28 12:07:16.805 6 WARNING nova.compute.resource_tracker [req-76b53cba-540c-4069-bb12-85b5f2b71f14 - - - - -] No compute node record for undercloud.overcloud.domain:2b4db183-7bbf-488c-8754-54174f0ba47e: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host undercloud.overcloud.domain could not be found.