Python 使用pyautogui定位opencv检测
因此,我正在制作一个机器人,它可以检测屏幕上的图标,并将鼠标移动到用户选择的检测到的图标。这是代码的样子:Python 使用pyautogui定位opencv检测,python,opencv,computer-vision,artificial-intelligence,pyautogui,Python,Opencv,Computer Vision,Artificial Intelligence,Pyautogui,因此,我正在制作一个机器人,它可以检测屏幕上的图标,并将鼠标移动到用户选择的检测到的图标。这是代码的样子: import numpy as np import pyautogui import cv2 from PIL import ImageGrab fourcc = cv2.VideoWriter_fourcc(*'XVID') face_csc = cv2.CascadeClassifier('improved_cascade.xml') out = cv2.VideoWriter(
import numpy as np
import pyautogui
import cv2
from PIL import ImageGrab
fourcc = cv2.VideoWriter_fourcc(*'XVID')
face_csc = cv2.CascadeClassifier('improved_cascade.xml')
out = cv2.VideoWriter("output.avi", fourcc, 5.0, (1366, 768))
while True:
img = ImageGrab.grab(bbox=None)
# convert image to numpy array
img_np = np.array(img)
# convert color space from BGR to RGB
frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2RGB)
# show image on OpenCV frame
faces = face_csc.detectMultiScale(frame, 1.1 , 15)
for (x,y,w,h) in faces:
detected_icon = cv2.rectangle(frame,(x,y),(x+w,y+h), (255,0,0), 2)
roi_gray = frame[y:y+h, x:x+w]
roi_color = img_np[y:y+h,x:x+w]
cv2.putText(frame,'icon',(x,y),cv2.FONT_HERSHEY_TRIPLEX,0.8,(0,0,255),1)
cv2.imshow("stream", frame)
# write frame to video writer
out.write(frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
out.release()
cv2.destroyAllWindows()
但是我很难让我的鼠标点击opencv检测到的图标。例如:假设我设置了我的程序,当它在屏幕上检测到chrome时,它会自动将鼠标悬停在图标上并单击它。我怎么能做到呢?多亏了我没有一个方便的Windows框来运行ImageGrab,但假设它生成的屏幕截图与实际屏幕的宽度和高度(以像素为单位)相同,鉴于Pyautogui和OpenCV都将原点放在左上角,转换为Pyautogui应该很简单:
for (x,y,w,h) in faces:
center_x = x + 0.5 * w
center_y = y + 0.5 * h
pyautogui.click(center_x, center_y)
将鼠标指针传送到对象矩形的中心并单击它。如果您想模拟更人性化的鼠标移动和单击,Pyautogui就有了相应的功能