Find website URLs that begin with http:// or https://

Question

This is a problem from Automate the Boring Stuff from "Pattern Matching with Regular Expression". I am a beginner in Python and the purpose of this program is to match urls which start with http:// or https://, with some urls having an optional www. after https://

How can this code be improved?

#! /usr/bin/python3
#websiteURL.py - Find website URLs that begin with http:// or https://

import pyperclip, re

#protocol Regex - checks for http:// or https://
protocolRegex = re.compile(r'''
    https?://           #match http:// or https://
    (?:w{3}\.)?         #www-dot
    [a-zA-Z0-9_-]+      #domin name
    \.                  #dot
    [a-zA-Z]{2,3}       #extension
    ''', re.VERBOSE)

text = str(pyperclip.paste()) #copying data from document to clipboard and converting it into a string
matches = [] #stores all matches in this list

for website in protocolRegex.findall(text): #finding website from the string text
    matches.append(website)

if len(matches) > 0:
    pyperclip.copy('\n'.join(map(str, matches))) #copying result to clipboard after adding newline after each match
    print('Copied to clipboard:')
    print('\n'.join(matches))
else:
    print('No website found')

Running code:

chmod +x websiteURL.py
./websiteURL.py

Answer 1

Variable Names

Variables should be lower-cased and snake-case, meaning that multiple words should look like some_variable, not someVariable or SomeVariable. I think the name you've chosen for your regex is good and expresses exactly what it is:

protocol_regex = re.compile(...)

Checking length of a list/container data structure

It is not considered pythonic to check the emptiness of a container in python by using if len(container) == 0, most* containers have a __bool__ method built-in that allows you to do if not container for an empty one and if container for non-empty:

from collections import deque

if not {}:
    print("empty dict")
empty dict

if not []:
    print("empty list")
empty list

if not '':
    print("empty string")
empty string

if not deque():
    print("empty deque")
empty deque

if ['result1']:
    print('non-empty list')
non-empty list

One of the few that does not behave this way is queue.Queue, which I've included to give context to the most comment.

With this in mind, change your match check to:

if matches:
    # rest of code
else:
    # else code here

Answer 2

The regular expression is inadequate for real domain names. Here's a few actual examples that fail:

http://hes.scot/
https://WWW.historicenvironment.scot/
https://www.bbc.co.uk
https://WWW.BBC.CO.UK

These are extracted as

http://hes.sco
https://WWW.his
https://www.bbc.co
https://WWW.BBC

Domain names are case-insensitive, can contain any number of levels, and need not end in a component having 2 or 3 letters.

(Note that the scheme name - http or https - is case-sensitive, as is the local part of a URL).

Answer 3

There is no need for reinvention of the wheel and complex pattern matching. Use urllib.parse.urlparse to check for the URL's scheme.

from urllib.parse import urlparse

def is_http_or_https(url):
    return urlparse(url).scheme in {'http', 'https'}