使用Python自动复制图像以提取内部黑边界ROI

我一直在研究OpenCV和Pillow（尤其是Python之外的ImageMagick），以实现以下目标：

自动识别扫描图像中的内部黑色边框，并将图像裁剪到该边框。这是一张黑屏示例图像，第一张是“原始”图像，第二张在黑色边框周围有一个红色高光，这是我希望实现的目标：

问题是，边界不在图像的外部，扫描质量差异很大，这意味着边界永远不会在同一点上，也不可能通过像素进行裁剪

编辑：我正在寻找一种方法来裁剪图像，只保留黑色边框内的所有内容（目前模糊的内容）

我正在寻求有关以下方面的帮助：a）是否可以进行这种裁剪；b）如何更好地使用Python进行裁剪

---

谢谢

在Imagemagick中，有一种非常简单的方法可以做到这一点

Get the center coordinates

Clone the image and do the following on the clone

Threshold the image so that the inside of the black lines is white. 

(If necessary use -connected-components to merge smaller black features into the white in the center)

Apply some morphology open to make sure that the black lines are continuous

Floodfill the image with red starting in the center

Convert non-red to black and red to white

Put the processed clone into the alpha channel of the input

输入：

以下是与上述代码等效的Python Wand代码：

#!/bin/python3.7

from wand.image import Image
from wand.drawing import Drawing
from wand.color import Color
from wand.display import display

with Image(filename='black_rect.jpg') as img:
    with img.clone() as copied:
        copied.auto_level()
        copied.threshold(threshold=0.35)
        copied.morphology(method='open', kernel='disk:5')
        centx=round(0.5*copied.width)
        centy=round(0.5*copied.height)
        with Drawing() as draw:
            draw.fill_color='red'
            draw.color(x=centx, y=centy, paint_method='floodfill')
            draw(copied)
        copied.opaque_paint(target='red', fill='black', fuzz=0.0, invert=True)
        copied.opaque_paint(target='red', fill='white', fuzz=0.0, invert=False)
        display(copied)
        copied.alpha_channel = 'copy'
        img.composite(copied, left=0, top=0, operator='copy_alpha')
        img.format='png'
        display(img)
        img.save(filename='black_rect_interior.png')

对于OpenCV，我建议下面的处理可以是一种方法。对不起，我不精通OpenCV

Threshold the image so that the inside of the black lines is white. 

Apply some morphology open to make sure that the black lines are continuous

Get the contours of the white regions.

Get the largest interior contour and fill the inside with white

Put that result into the alpha channel of the input

补充：

对于那些感兴趣的人，这里有一个较长的方法，有助于透视校正。我做了一些类似于纳森西所做的事情，但用的是Imagemagick

首先，对图像进行阈值设置并打开形态学，以确保黑线是连续的

然后做连接组件，得到最大白色区域的ID号

然后提取该区域

id=$(convert img.jpg -auto-level -threshold 35% \
-morphology open disk:5 -type bilevel \
-define connected-components:mean-color=true \
-define connected-components:verbose=true \
-connected-components 8 null: | grep "gray(255)" | head -n 1 | awk '{print $1}' | sed 's/[:]*$//')
echo $id

convert img.jpg -auto-level -threshold 35% \
-morphology open disk:5 -type bilevel \
-define connected-components:mean-color=true \
-define connected-components:keep=$id \
-connected-components 8 \
-alpha extract -morphology erode disk:5 \
region.png

现在做Canny边缘检测和hough线变换。在这里，我将canny图像、hough线保存为红线，并将覆盖在图像上的线和线信息保存在.mvg文件中

convert region.png \
\( +clone -canny 0x1+10%+30% +write region_canny.png \
-background none -fill red -stroke red -strokewidth 2 \
-hough-lines 9x9+400 +write region_lines.png +write lines.mvg \) \
-compose over -composite region_hough.png

convert region_lines.png -alpha extract region_bw_lines.png

# Hough line transform: 9x9+400
viewbox 0 0 2000 2829
# x1,y1  x2,y2 # count angle distance
line 0,202.862 2000,272.704  # 763 92 824
line 204.881,0 106.09,2829  # 990 2 1156
line 1783.84,0 1685.05,2829  # 450 2 2734
line 0,2620.34 2000,2690.18  # 604 92 3240

接下来，我使用我编写的脚本来执行角点检测。这里我使用哈里斯探测器

corners=$(corners -m harris -t 40 -d 5 -p yes region_bw_lines.png region_bw_lines_corners.png)
echo "$corners"

pt=1 coords=195.8,207.8
pt=2 coords=1772.8,262.8
pt=3 coords=111.5,2622.5
pt=4 coords=1688.5,2677.5

接下来，我以顺时针方式提取和排序角点。下面是我编写的一些代码，这些代码是我从中转换而来的

最后，我使用角坐标在黑色背景上绘制一个白色填充多边形，并将结果放入输入图像的alpha通道

convert img.jpg \
\( +clone -fill black -colorize 100 \
-fill white -draw "polygon $sort_corners" \) \
-alpha off -compose copy_opacity -composite result.png

---

好的，长柱进线系好安全带。这是策略

• 将图像转换为灰度和中值模糊
• 执行canny边缘检测
• 执行形态学变换以平滑图像
• 放大以增强和连接轮廓
• 执行线检测并在掩模上绘制所需的矩形ROI
• 执行此操作以检测4个角
• 顺时针排列角点
• 在第二个遮罩上绘制角点，并找到轮廓以获得完美的ROI
• 对ROI执行透视变换以获得鸟瞰视图
• 旋转图像以获得最终结果

---

Canny边缘检测

接下来，我们执行形态学变换以闭合间隙并平滑图像（左）。然后放大以增强轮廓（右）

从这里开始，我们使用

cv2.HoughLinesP（）

和最小线长度和最大线间距过滤器执行线检测，以获得较大的矩形ROI。我们把这个ROI画在面具上

minLineLength = 150
maxLineGap = 250
lines = cv2.HoughLinesP(dilate,1,np.pi/180,100,minLineLength,maxLineGap)
for line in lines:
    for x1,y1,x2,y2 in line:
        cv2.line(mask,(x1,y1),(x2,y2),(255,255,255),2)

mask = cv2.dilate(mask, kernel, iterations=2)
mask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)

现在，我们使用

cv2.goodFeaturesToTrack（）

执行Shi-Tomasi角点检测，以检测四个角点坐标

corners = cv2.goodFeaturesToTrack(mask,4,0.5,1000)

c_list = []
for corner in corners:
    x,y = corner.ravel()
    c_list.append([int(x), int(y)])
    cv2.circle(image,(x,y),40,(36,255,12),-1)

无序角坐标

corners = cv2.goodFeaturesToTrack(mask,4,0.5,1000)

c_list = []
for corner in corners:
    x,y = corner.ravel()
    c_list.append([int(x), int(y)])
    cv2.circle(image,(x,y),40,(36,255,12),-1)

[16902693]、[1132622]、[1766269]、[197212]]

从这里开始，我们顺时针对四个角点重新排序，方法是对坐标进行排序并按

（左上、右上、右下、左下）

顺序重新排列。这一步对于在执行透视变换时获得ROI的自顶向下视图非常重要

有序角坐标

corners = cv2.goodFeaturesToTrack(mask,4,0.5,1000)

c_list = []
for corner in corners:
    x,y = corner.ravel()
    c_list.append([int(x), int(y)])
    cv2.circle(image,(x,y),40,(36,255,12),-1)

[197212],， [1766,269], [1690,2693], [1132622]]

重新排序后，我们将点绘制到第二个掩模上，以获得完美的ROI

现在，我们在ROI上执行透视变换，以获得图像的自顶向下视图

最后，我们将其旋转

-90

度以获得所需的结果

import cv2
import numpy as np

def rotate_image(image, angle):
    # Grab the dimensions of the image and then determine the center
    (h, w) = image.shape[:2]
    (cX, cY) = (w / 2, h / 2)

    # grab the rotation matrix (applying the negative of the
    # angle to rotate clockwise), then grab the sine and cosine
    # (i.e., the rotation components of the matrix)
    M = cv2.getRotationMatrix2D((cX, cY), -angle, 1.0)
    cos = np.abs(M[0, 0])
    sin = np.abs(M[0, 1])

    # Compute the new bounding dimensions of the image
    nW = int((h * sin) + (w * cos))
    nH = int((h * cos) + (w * sin))

    # Adjust the rotation matrix to take into account translation
    M[0, 2] += (nW / 2) - cX
    M[1, 2] += (nH / 2) - cY

    # Perform the actual rotation and return the image
    return cv2.warpAffine(image, M, (nW, nH))

def order_points_clockwise(pts):
    # sort the points based on their x-coordinates
    xSorted = pts[np.argsort(pts[:, 0]), :]

    # grab the left-most and right-most points from the sorted
    # x-roodinate points
    leftMost = xSorted[:2, :]
    rightMost = xSorted[2:, :]

    # now, sort the left-most coordinates according to their
    # y-coordinates so we can grab the top-left and bottom-left
    # points, respectively
    leftMost = leftMost[np.argsort(leftMost[:, 1]), :]
    (tl, bl) = leftMost

    # now, sort the right-most coordinates according to their
    # y-coordinates so we can grab the top-right and bottom-right
    # points, respectively
    rightMost = rightMost[np.argsort(rightMost[:, 1]), :]
    (tr, br) = rightMost

    # return the coordinates in top-left, top-right,
    # bottom-right, and bottom-left order
    return np.array([tl, tr, br, bl], dtype="int32")

def perspective_transform(image, corners):
    def order_corner_points(corners):
        # Separate corners into individual points
        # Index 0 - top-right
        #       1 - top-left
        #       2 - bottom-left
        #       3 - bottom-right
        corners = [(corner[0][0], corner[0][1]) for corner in corners]
        top_r, top_l, bottom_l, bottom_r = corners[0], corners[1], corners[2], corners[3]
        return (top_l, top_r, bottom_r, bottom_l)

    # Order points in clockwise order
    ordered_corners = order_corner_points(corners)
    top_l, top_r, bottom_r, bottom_l = ordered_corners

    # Determine width of new image which is the max distance between 
    # (bottom right and bottom left) or (top right and top left) x-coordinates
    width_A = np.sqrt(((bottom_r[0] - bottom_l[0]) ** 2) + ((bottom_r[1] - bottom_l[1]) ** 2))
    width_B = np.sqrt(((top_r[0] - top_l[0]) ** 2) + ((top_r[1] - top_l[1]) ** 2))
    width = max(int(width_A), int(width_B))

    # Determine height of new image which is the max distance between 
    # (top right and bottom right) or (top left and bottom left) y-coordinates
    height_A = np.sqrt(((top_r[0] - bottom_r[0]) ** 2) + ((top_r[1] - bottom_r[1]) ** 2))
    height_B = np.sqrt(((top_l[0] - bottom_l[0]) ** 2) + ((top_l[1] - bottom_l[1]) ** 2))
    height = max(int(height_A), int(height_B))

    # Construct new points to obtain top-down view of image in 
    # top_r, top_l, bottom_l, bottom_r order
    dimensions = np.array([[0, 0], [width - 1, 0], [width - 1, height - 1], 
                    [0, height - 1]], dtype = "float32")

    # Convert to Numpy format
    ordered_corners = np.array(ordered_corners, dtype="float32")

    # Find perspective transform matrix
    matrix = cv2.getPerspectiveTransform(ordered_corners, dimensions)

    # Return the transformed image
    return cv2.warpPerspective(image, matrix, (width, height))

image = cv2.imread('1.jpg')
original = image.copy()

mask = np.zeros(image.shape, np.uint8)
clean_mask = np.zeros(image.shape, np.uint8)
blur = cv2.medianBlur(image, 9)
gray = cv2.cvtColor(blur, cv2.COLOR_BGR2GRAY)
canny = cv2.Canny(gray, 120, 255, 1)
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))
close = cv2.morphologyEx(canny, cv2.MORPH_CLOSE, kernel, iterations=2)
dilate = cv2.dilate(close, kernel, iterations=1)

minLineLength = 150
maxLineGap = 250
lines = cv2.HoughLinesP(dilate,1,np.pi/180,100,minLineLength,maxLineGap)
for line in lines:
    for x1,y1,x2,y2 in line:
        cv2.line(mask,(x1,y1),(x2,y2),(255,255,255),2)

mask = cv2.dilate(mask, kernel, iterations=2)
mask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
cv2.imwrite('mask.png', mask)

corners = cv2.goodFeaturesToTrack(mask,4,0.5,1000)

c_list = []
for corner in corners:
    x,y = corner.ravel()
    c_list.append([int(x), int(y)])
    cv2.circle(image,(x,y),40,(36,255,12),-1)

cv2.imwrite('corner_points.png', image)
corner_points = np.array([c_list[0], c_list[1], c_list[2], c_list[3]])
ordered_corner_points = order_points_clockwise(corner_points)
ordered_corner_points = np.array(ordered_corner_points).reshape((-1,1,2)).astype(np.int32)

cv2.drawContours(clean_mask, [ordered_corner_points], -1, (255, 255, 255), 2)

cv2.imwrite('clean_mask.png', clean_mask)
clean_mask = cv2.cvtColor(clean_mask, cv2.COLOR_BGR2GRAY)
cnts = cv2.findContours(clean_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]

for c in cnts:
    # approximate the contour
    peri = cv2.arcLength(c, True)
    approx = cv2.approxPolyDP(c, 0.015 * peri, True)

    if len(approx) == 4:
        transformed = perspective_transform(original, approx)

# Rotate image
result = rotate_image(transformed, -90)

cv2.imshow('image', image)
cv2.imwrite('close.png', close)
cv2.imwrite('canny.png', canny)
cv2.imwrite('dilate.png', dilate)
cv2.imshow('clean_mask', clean_mask)
cv2.imwrite('image.png', image)
cv2.imshow('result', result)
cv2.imwrite('result.png', result)
cv2.waitKey()

编辑：另一种方法

与上述方法非常相似，但我们可以使用轮廓面积作为过滤器提取最大的内部轮廓，然后使用遮罩来获得相同的结果，而不是使用角点检测来查找ROI。获取透视变换的方法相同

---

具体来说，这是什么问题？是否要以某种方式裁剪此边框？查看是否可以反扭曲图像，即使用ImageMagick

magick input.png-deskew 40%结果.png使其垂直，如果可行，您现在正在寻找很长的水平线或垂直线，这些线很容易用一些形态学来定位。@MarkSetchell我只是用Fred的ImageMagick脚本尝试了一下，但它不起作用，您的命令起作用了，让我更近了一步，谢谢@ImtinanAzhar是的，我想裁剪像我自动发布到黑色边框内的图像。您是否尝试过使用Python/Opencv进行精明的边缘检测，然后从Hough transform获取直线交点。
@nathancy
回答得很好！我开始在Imagemagick中类似地处理它。但现在按照我建议的方法更简单了吗。你不需要只设定阈值，使用形态学使线条连续。然后提取最大内部白色区域的轮廓，并将其填充以用作遮罩，以放入图像的alpha通道？您还可以从Hough线图像中获取内部黑色区域轮廓。对不起，我对OpenCV不是很精通，我正在努力学习。是的，使用Imagemagick可能要简单得多，因为如果使用OpenCV，有很多小细节需要过滤掉。嗯，是的，那是个更好的主意。
import cv2
import numpy as np

def rotate_image(image, angle):
    # Grab the dimensions of the image and then determine the center
    (h, w) = image.shape[:2]
    (cX, cY) = (w / 2, h / 2)

    # grab the rotation matrix (applying the negative of the
    # angle to rotate clockwise), then grab the sine and cosine
    # (i.e., the rotation components of the matrix)
    M = cv2.getRotationMatrix2D((cX, cY), -angle, 1.0)
    cos = np.abs(M[0, 0])
    sin = np.abs(M[0, 1])

    # Compute the new bounding dimensions of the image
    nW = int((h * sin) + (w * cos))
    nH = int((h * cos) + (w * sin))

    # Adjust the rotation matrix to take into account translation
    M[0, 2] += (nW / 2) - cX
    M[1, 2] += (nH / 2) - cY

    # Perform the actual rotation and return the image
    return cv2.warpAffine(image, M, (nW, nH))

def perspective_transform(image, corners):
    def order_corner_points(corners):
        # Separate corners into individual points
        # Index 0 - top-right
        #       1 - top-left
        #       2 - bottom-left
        #       3 - bottom-right
        corners = [(corner[0][0], corner[0][1]) for corner in corners]
        top_r, top_l, bottom_l, bottom_r = corners[0], corners[1], corners[2], corners[3]
        return (top_l, top_r, bottom_r, bottom_l)

    # Order points in clockwise order
    ordered_corners = order_corner_points(corners)
    top_l, top_r, bottom_r, bottom_l = ordered_corners

    # Determine width of new image which is the max distance between 
    # (bottom right and bottom left) or (top right and top left) x-coordinates
    width_A = np.sqrt(((bottom_r[0] - bottom_l[0]) ** 2) + ((bottom_r[1] - bottom_l[1]) ** 2))
    width_B = np.sqrt(((top_r[0] - top_l[0]) ** 2) + ((top_r[1] - top_l[1]) ** 2))
    width = max(int(width_A), int(width_B))

    # Determine height of new image which is the max distance between 
    # (top right and bottom right) or (top left and bottom left) y-coordinates
    height_A = np.sqrt(((top_r[0] - bottom_r[0]) ** 2) + ((top_r[1] - bottom_r[1]) ** 2))
    height_B = np.sqrt(((top_l[0] - bottom_l[0]) ** 2) + ((top_l[1] - bottom_l[1]) ** 2))
    height = max(int(height_A), int(height_B))

    # Construct new points to obtain top-down view of image in 
    # top_r, top_l, bottom_l, bottom_r order
    dimensions = np.array([[0, 0], [width - 1, 0], [width - 1, height - 1], 
                    [0, height - 1]], dtype = "float32")

    # Convert to Numpy format
    ordered_corners = np.array(ordered_corners, dtype="float32")

    # Find perspective transform matrix
    matrix = cv2.getPerspectiveTransform(ordered_corners, dimensions)

    # Return the transformed image
    return cv2.warpPerspective(image, matrix, (width, height))

image = cv2.imread('1.jpg')
original = image.copy()

mask = np.zeros(image.shape, np.uint8)
clean_mask = np.zeros(image.shape, np.uint8)
blur = cv2.medianBlur(image, 9)
gray = cv2.cvtColor(blur, cv2.COLOR_BGR2GRAY)
canny = cv2.Canny(gray, 120, 255, 1)
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))
close = cv2.morphologyEx(canny, cv2.MORPH_CLOSE, kernel, iterations=2)
dilate = cv2.dilate(close, kernel, iterations=1)

minLineLength = 150
maxLineGap = 250
lines = cv2.HoughLinesP(dilate,1,np.pi/180,100,minLineLength,maxLineGap)
for line in lines:
    for x1,y1,x2,y2 in line:
        cv2.line(mask,(x1,y1),(x2,y2),(255,255,255),2)

mask = cv2.dilate(mask, kernel, iterations=2)
mask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
cv2.imwrite('mask.png', mask)

cnts = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
cnts = sorted(cnts, key = cv2.contourArea, reverse = True)[:10]

for c in cnts:
    cv2.drawContours(clean_mask, [c], -1, (255, 255, 255), -1)

clean_mask = cv2.morphologyEx(clean_mask, cv2.MORPH_OPEN, kernel, iterations=5)
result_no_transform = cv2.bitwise_and(clean_mask, image)

clean_mask = cv2.cvtColor(clean_mask, cv2.COLOR_BGR2GRAY)
cnts = cv2.findContours(clean_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]

for c in cnts:
    # approximate the contour
    peri = cv2.arcLength(c, True)
    approx = cv2.approxPolyDP(c, 0.015 * peri, True)

    if len(approx) == 4:
        transformed = perspective_transform(original, approx)

result = rotate_image(transformed, -90)
cv2.imwrite('transformed.png', transformed)
cv2.imwrite('result_no_transform.png', result_no_transform)
cv2.imwrite('result.png', result)
cv2.imwrite('clean_mask.png', clean_mask)