13
\$\begingroup\$

Background:
Motivation behind writing the following code is originated in the area of computer vision. More specifically – image rectification. In order to obtain rectified images, one has to find a set of matching features/keypoints on both images beforehand. My code is supposed to operate on these points.

When calculating rectifications for a series of aerial images, I was obtaining poor results. Around 80% of rectified images were squeezed and tilted at a large angle. For example: enter image description here
To improve the results my professor suggested selecting only the top X% of the keypoints, as there are probably many bad quality keypoints (keypoints have weights or similarity scores by which we could sort them). However, that selection could introduce a strong bias, as, probably, one side of the image has much better keypoints than the other side. So, we would want them to be distributed evenly on the image.

When looking for an algorithm that could select the most scattered points, I came across two posts on Stack Exchange:

In the second post I found a link leading to the following article: Efficiently selecting spatially distributed keypoints for visual tracking. There they describe an algorithm that takes into account weights of the points. Since it's possible to sort the keypoints by a score of similarity, this looked like a way to go. I implemented their Suppression via Disk Covering (SDC) algorithm and got ~70% good quality rectified images comparing to previous ~20%. Here is an example of a good one:
enter image description here

Idea of the algorithm is pictured on the following image taken from the article:
enter image description here Relevant steps (citing the article):

1) Input image for which a set of, for example, k = 20 strong, well-distributed keypoints is sought.

2) Keypoints found by detector.

3) The first (strongest) keypoint is selected and all cells within the approximated radius r are covered.

4) Strongest uncovered point is selected, surrounding cells covered.

5) Finally, with this radius, five keypoints have been selected. This is below the desired k, so a new iteration (6)) is started with a smaller r.

7) More than k points are selected and still uncovered points left: r is too small, the iteration can be aborted and the next iteration (8)) started.

9) Finally, with this r , exactly k keypoints have been selected and are returned as result


What I want reviewed:

  • Code organization, separating logic to functions, DRY, style, choice of names, etc.
  • Performance. (It's possible that in the future this code will run in real time. Right now it's not critical, so I didn't do any profiling. But by taking a look on the current state of the code, I don't see how it could be sped up.)
  • Missed bugs, edge-cases.
  • Alternative algorithms.
  • Is this an XY problem? I am new to computer vision and I'm afraid that I could take a wrong way to tackle this problem. I asked a separate question about tilted rectified images: Skewed rectified aerial images

Code:

suppression_via_disk_covering.py

from functools import partial
from typing import Tuple

import numpy as np


def select(points: np.ndarray,
           *,
           image_shape: Tuple[int, int],
           count: int,
           count_delta: int = 1,
           radius: int = 10,
           radius_delta: int = 2,
           max_iterations_count: int = 15,
           min_cell_size: int = 2,
           max_cell_size: int = 100) -> np.ndarray:
    """
    Selects points by a Suppression via Disk Covering algorithm.
    For more details see:
    http://lucafoschini.com/papers/Efficiently_ICIP11.pdf
    :param points: original set that should be ordered by distance
    :param image_shape: shape of an image
    :param count: number of output points
    :param count_delta: let k = `count` and Δk = `count_delta`,
    if number of found points is within [k; k + Δk],
    return top-k points
    :param radius: initial radius of area where points will be removed
    :param radius_delta: determines width of cells
    :param max_iterations_count: prevents infinite loop
    :param min_cell_size:
    :param max_cell_size:
    :return: mask array with selected strong scattered keypoints
    """
    if len(points) < count:
        raise ValueError('Not enough points to select.')

    grid_resolution = radius_delta * radius / np.sqrt(2)

    max_count = count + count_delta

    points_mask = partial(selected_points_mask,
                          points,
                          image_shape=image_shape,
                          count=max_count,
                          radius=radius)

    for _ in range(max_iterations_count):
        result_mask = points_mask(grid_resolution=grid_resolution)

        selected_points_count = result_mask.sum()

        if selected_points_count == count:
            return result_mask

        if count < selected_points_count <= max_count:
            return erase_extra_points(result_mask,
                                      count=count)

        if selected_points_count < count:
            max_cell_size = grid_resolution
            grid_resolution -= (grid_resolution - min_cell_size) / 2
        else:
            min_cell_size = grid_resolution
            grid_resolution += (max_cell_size - grid_resolution) / 2

    raise ValueError('Number of iterations exceeded.')


def selected_points_mask(points: np.ndarray,
                         *,
                         grid_resolution: float,
                         image_shape: Tuple[int, int],
                         count: int,
                         radius: int) -> np.ndarray:
    """
    Calculates boolean mask corresponding to array of input points.
    True values are for those points
    that will be selected as scattered enough from each other.
    In case if there were too many points found,
    the mask still will be returned.
    :param points: input array
    :param grid_resolution: size of a cell in a grid
    :param image_shape:
    :param count: number of points to select
    :param radius: as number of cells where points won't be selected
    :return: boolean array with True values for selected points
    """
    points_grid_indices = (points // grid_resolution).astype(int)

    grid_shape = (int(image_shape[0] // grid_resolution) + 1,
                  int(image_shape[1] // grid_resolution) + 1)
    grid = np.full(shape=grid_shape,
                   fill_value=False)

    result_mask = np.full(shape=points.shape[0],
                          fill_value=False)

    for index, point_grid_index in enumerate(points_grid_indices):
        if grid[tuple(point_grid_index)]:
            continue

        result_mask[index] = True

        if result_mask.sum() > count:
            break

        mask = circular_mask(grid.shape,
                             center=point_grid_index,
                             radius=radius)
        grid[mask] = True

    return result_mask


def circular_mask(array_shape: Tuple[int, int],
                  *,
                  center: Tuple[int, int],
                  radius: int) -> np.ndarray:
    """
    Returns 2d array with applied a disc shaped mask over it.
    For more details see:
    https://stackoverflow.com/questions/8647024/how-to-apply-a-disc-shaped-mask-to-a-numpy-array
    https://stackoverflow.com/questions/44865023/circular-masking-an-image-in-python-using-numpy-arrays
    :param array_shape: shape of original image
    :param center: center of the disc
    :param radius: radius of the disc
    :return: boolean array with applied circular mask
    """
    y, x = np.ogrid[-center[0]:array_shape[0] - center[0],
                    -center[1]:array_shape[1] - center[1]]
    return x * x + y * y <= radius * radius


def erase_extra_points(array: np.ndarray,
                       *,
                       count: int) -> np.ndarray:
    """
    Let n = `count`, sets to False all elements
    after the n-th occurrence of a True element.
    :param array: input boolean array
    :param count: number of True elements to remain
    :return:
    """
    array = array.copy()
    last_true_index_to_remain = np.where(array)[0][count]
    array[last_true_index_to_remain:] = False
    return array

Examples of usage:
Simple example without , ignoring ordering of points by strength:

%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np

import suppression_via_disc_covering as sdc

image_shape = (1500, 2000)
points = np.random.uniform(low=(0, 0), 
                           high=image_shape, 
                           size=(100, 2))
mask = sdc.select(points,
                  image_shape=image_shape,
                  count=8)
plt.scatter(points[:, 0],
            points[:, 1])
plt.scatter(points[mask, 0],
            points[mask, 1],
            color='r')

Output:
enter image description here


Example with (Shi-Tomasi corner detector):

Taking the following image:
duck.jpg enter image description here

import cv2

import suppression_via_disc_covering as sdc

image = cv2.imread('duck.jpg')
image = cv2.cvtColor(image,
                     cv2.COLOR_BGR2GRAY)

corners = cv2.goodFeaturesToTrack(image,
                                  maxCorners=1000,
                                  qualityLevel=0.01,
                                  minDistance=10)
corners = corners.reshape(-1, 2)

for corner in corners:
    cv2.circle(image,
               center=tuple(corner),
               radius=3,
               color=0,
               thickness=-1)

mask = sdc.select(corners,
                  image_shape=image.shape[::-1],
                  count=25)

good_corners = corners[mask, :]

image = cv2.cvtColor(image,
                     cv2.COLOR_GRAY2RGB)

for corner in good_corners:
    cv2.circle(image,
               center=tuple(corner),
               radius=3,
               color=(0, 0, 255),
               thickness=-1)

cv2.imshow('image',
           image)
cv2.waitKey(0)

Output:
enter image description here


P.S.: I'm not including an example with pairs of aerial images, since there is much more code there that should be reviewed separately. Moreover, I still consider that 70% of good results is not good enough and hence this example would be considered as "doesn't work as intended".


1I tried this algorithm as well and quality of the results improved from ~20% of good images to ~45%. This is not good enough.

\$\endgroup\$
  • 1
    \$\begingroup\$ Well written question and clean, professional code, no suggestions re. code style. Have you looked into SIFT/SURF image features, if using them for point detection might yield better, more resilient keypoints? Or using K-means (or another kind of) clustering to select spatially distributed "centroid" keypoints? \$\endgroup\$ – scnerd May 15 '18 at 14:51
  • \$\begingroup\$ Thanks, @scnerd! I did not consider using SIFT/SURF because I was under impression that ORB yields better results after reading tutorial. But quick googling led me to the following article, and there I can see that ORB wins only by time. I will try them out. Regarding clustering, I got the same idea from a data scientist friend. Sounds like something worth investigating. \$\endgroup\$ – Georgy May 15 '18 at 15:25
  • 1
    \$\begingroup\$ I have recently published a paper that tackles the problem of homogeneous keypoint distribution on the image. C++, Python, and Matlab interfaces are provided in this repository. Moreover, comparison to SDC algorithm is in the paper. \$\endgroup\$ – Alex Bailo Oct 19 '18 at 7:12
  • \$\begingroup\$ @AlexBailo Thanks for sharing! I took a look at Python code and I see that it could use some refactoring. Would you like to post it here so me or other people review it? :) \$\endgroup\$ – Georgy Oct 19 '18 at 9:17
  • 1
    \$\begingroup\$ Your keypoints have coordinates and some quality metric, let's call those x_i, y_i and Q_i. Try to fit a function f(x, y) = ax + by + c to the keypoints' quality metric. Then create a corrected quality metric Q'_i = Q_i / f(x_i, y_i). Select the best keypoints from the correct quality metric to do your visual tracking. If it works, then it's a faster algorithm than farthest points, and possibly it selects a better set of keypoints (it doesn't constrain the selection based on position relative to nearby keypoints). \$\endgroup\$ – G. Sliepen Oct 20 '18 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.