To be clear, this code is built to be used with the Snipping Tool and Robot Framework. Use the snipping tool to snip an image of something on the screen. A strange button with weird locators, something outside of the DOM... anything as long as it consistently repeats visually during your test. Then, send that image to the keyword, and the code will locate the center of that image on the screen and execute a click command at that location.

I'm currently looking for consistency suggestions, but if you also have optimization suggestions, I'd gladly accept those as well. It's very, very fast, but I'm running into a problem where the code doesn't find the image unless it's a relatively large image with some detail (larger than a tab on Google Chrome, I haven't found the exact "x pixels by y pixels" size). I'd like to be able to tell Robot Framework to click the "X" button to close the browser window without worrying about the exact pixel-based coordinates of the browser.

EDIT: That's an example of how small I'd like the image to be, not the ultimate goal of this code.

from robot.libraries.BuiltIn import BuiltIn
import pyautogui as pag

class click_by_image(object):
    def __init__(self):
        self.selenium_lib = None

    def click_by_image(self, image_name):
        Click the center of a rectangle on the screen given the
        image_name of the image rectangle to search for
        :param image_name: Path to image
        #   If ExtendedSelenium2Library instance is undefined
        if self.selenium_lib is None:
            # Instantiate ExtendedSelenium2Library
            self.selenium_lib = BuiltIn().get_library_instance('ExtendedSelenium2Library')

Here's an example of the use of the Robot Framework keyword, assuming the image of what you're trying to click is called button.png and is located in the same folder as the resource file it's being used in. I believe it will take most common formats of picture.

Click By Image  button.png

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.