To be clear, this code is built to be used with the Snipping Tool and Robot Framework. Use the snipping tool to snip an image of something on the screen. A strange button with weird locators, something outside of the DOM... anything as long as it consistently repeats visually during your test. Then, send that image to the keyword, and the code will locate the center of that image on the screen and execute a click command at that location.
I'm currently looking for consistency suggestions, but if you also have optimization suggestions, I'd gladly accept those as well. It's very, very fast, but I'm running into a problem where the code doesn't find the image unless it's a relatively large image with some detail (larger than a tab on Google Chrome, I haven't found the exact "x pixels by y pixels" size). I'd like to be able to tell Robot Framework to click the "X" button to close the browser window without worrying about the exact pixel-based coordinates of the browser.
EDIT: That's an example of how small I'd like the image to be, not the ultimate goal of this code.
from robot.libraries.BuiltIn import BuiltIn import pyautogui as pag class click_by_image(object): def __init__(self): self.selenium_lib = None def click_by_image(self, image_name): """ Click the center of a rectangle on the screen given the image_name of the image rectangle to search for :param image_name: Path to image """ # If ExtendedSelenium2Library instance is undefined if self.selenium_lib is None: # Instantiate ExtendedSelenium2Library self.selenium_lib = BuiltIn().get_library_instance('ExtendedSelenium2Library') pag.click(pag.locateCenterOnScreen(str(image_name)))
Here's an example of the use of the Robot Framework keyword, assuming the image of what you're trying to click is called
button.png and is located in the same folder as the resource file it's being used in. I believe it will take most common formats of picture.
Click By Image button.png