Error using OpenCV perspectiveTransform and cropping an image [ Python ]

Question

EDIT: updated post. Minimum reproducible example code included, with images that produce the error mentioned in this post.

I'm new and unexperienced with openCV. I don't have a deep understanding of the openCV data structures, or what data structures are returned from an OpenCV function. So far I've been able to get by piecing together different code segments found with google searches. I've run into a problem where this is biting me, because I don't understand what's going on behind the scenes with cv2.findHomography and cv2.perspectiveTransform.

Goal of the Code: take image1 and image2 as arguments if it exists, find the region in image1 that image2 matches. (using SIFT, not template matching) crop image1 to the region that matches image2

Current Status: the code works for most images, but occasionally I get an error thrown where the cropped image size dimensions are 0 in either height or width. I attempt to check for this condition, by checking the width and height from the perspectiveTransform coordinates but not catching all the invalid cropped images.

Problem: An invalid cropped image is created, which causes an error when the image is displayed with cv2.imshow()

Code:

import cv2
import numpy as np
import matplotlib.pyplot as plt
import time
from time import time
from timeit import Timer
from Stat_File import FileHelper
import os
import win32api, win32con
import win32gui
import ctypes


user32 = ctypes.windll.user32
SCREEN_WIDTH =  user32.GetSystemMetrics(0)
SCREEN_HEIGHT =  user32.GetSystemMetrics(1)

MIN_MATCH_COUNT = 10

cv2.imread("./img/verify_page/single/Captcha_Picture/button.png")

original = "./img/verify_page/single/opencv/01B.png"
match_image = cv2.imread("./img/verify_page/single/opencv/01A.png")

find_image = cv2.imread(original)

scale_width = (SCREEN_WIDTH / find_image.shape[1])*.9
scale_height = (SCREEN_HEIGHT / find_image.shape[0])*.9
scale = min(scale_width, scale_height)

#resized window width and height
window_width = int(find_image.shape[1] * scale)
window_height = int(find_image.shape[0] * scale)
cv2.namedWindow('ORIGINAL', cv2.WINDOW_NORMAL)
cv2.resizeWindow('ORIGINAL', window_width, window_height)
cv2.imshow("ORIGINAL", find_image)
cv2.waitKey(0)


sift = cv2.SIFT.create()
keypoint1, descr1 = sift.detectAndCompute(match_image, None)
keypoint2, descr2 = sift.detectAndCompute(find_image, None)
good = []
good2 = []

find_h, find_w, _ = match_image.shape
match_h, match_w, _ = find_image.shape

bf = cv2.BFMatcher()
matches = bf.knnMatch(descr1, descr2, k=2)
for m,n in matches:
    if m.distance < 0.75*n.distance:
        good.append([m])
        good2.append(m)

if len(good)>MIN_MATCH_COUNT:
    src_pts = np.float32([ keypoint1[m.queryIdx].pt for m in good2 ]).reshape(-1,1,2)
    dst_pts = np.float32([ keypoint2[m.trainIdx].pt for m in good2 ]).reshape(-1,1,2)

    M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC,5.0)
    matchesMask = mask.ravel().tolist()

    pts = np.float32([ [0,0],[0,find_h-1],[find_w-1,find_h-1],[find_w-1,0] ]).reshape(-1,1,2)
    dst = None
    try: 
        dst = cv2.perspectiveTransform(pts,M)

        isClosed = True
        color = (255, 0, 0)     # Blue color in BGR 
        thickness = 2           # Line thickness of 2 px 
        img2 = cv2.polylines(find_image,[np.int32(dst)],isClosed,color,thickness, 
        cv2.LINE_AA)

        scale_width = (SCREEN_WIDTH / img2.shape[1])*.9
        scale_height = (SCREEN_HEIGHT / img2.shape[0])*.9
        scale = min(scale_width, scale_height)

        #resized window width and height
        window_width = int(img2.shape[1] * scale)
        window_height = int(img2.shape[0] * scale)
        cv2.namedWindow('POLYLINES', cv2.WINDOW_NORMAL)
        cv2.resizeWindow('POLYLINES', window_width, window_height)
        cv2.imshow("POLYLINES", img2)
        cv2.waitKey(0)

        dst = np.int32(dst)
        upper_left,lower_left,lower_right,upper_right = dst
        x1 = upper_left[0][0]
        x2 = upper_right[0][0]
        y1 = upper_left[0][1]
        y2 = lower_left[0][1]
        print("x1: ", x1)
        print("x2: ", x2)
        print("y1: ", y1)
        print("y2: ", y2)

        width = x2-x1
        height = y2-y1
        top_left = (x1,y1)
        #print("top_left: ", top_left)
        print("calculated width: ", width)
        print("calculated height: ", height)

        orig = cv2.imread(original)
        cropped_image = orig[y1:y2, x1:x2]
        actual_height, actual_width, _ = cropped_image.shape
        print("actual width: ", actual_width)
        print("actual height: ", actual_height)
        cv2.imshow('ORIGINAL CROPPED', cropped_image)
        cv2.waitKey(0)
        cv2.destroyAllWindows()

    except cv2.error as e:
        pass

Use this as the find_image:

Use this as the match image:

Using the above two images produces the desired result. I found I get an error when the entire match image is not found in the larger image, only part of it. That is, part of the "match" is offscreen. Using below image as the find image will produce an invalid cropped image:

If I execute this code, it works for the most part. Every now and then however I'll receive an error where x1,x2 y1,y2 suggest a nonzero cropped image size. If I do the crop however with the line:

cropped_image = orig[y1:y2, x1:x2]

and check the result size, I'll get a zero dimension in either height or width. That is:

x2-x1 != cropped_image.shape[1]

If I run the above program on a problematic image, the console output I get is:

x1:  446
x2:  542
y1:  -51
y2:  130
        calculated width:  96
        calculated height:  181
        actual width:  96
        actual height:  0

Sometimes an error will be thrown, and the error is:

cv2.error: OpenCV(4.4.0) C:UsersappveyorAppDataLocalTemp1pip-req-build-h4wtvo23opencvmodulesfeatures2dsrcsift.dispatch.cpp:465: error: (-5:Bad argument) image is empty or has incorrect depth (!=CV_8U) in function 'cv::SIFT_Impl::detectAndCompute'

Any help would be appreciated. Thank you very much-

Answer 1

等待大神答复