Batch retrieve web titles from URLs

Question

This working script reads a list of URLs from one or more text files,
and for each, retrieves the page title from the Internet
and appends the results in the format

<url/>\t<title/>\n

to the output file (for each input file it creates a corresponding output file).

I am a Python newbie and would appreciate any feedback.

How to run

1. Save the below script as "retrieveWebTitles.py" (or whatever name you like)
2. In the same folder create a folder called "url".
3. In the folder create 3 text files named "links1.txt", "links2.txt", "links3.txt" with the corresponding contents listed below.
4. Make sure you're connected to the Internet or the script won't work.
5. Load the script into Idle and press F5. It should run a couple minutes and print "Done" when finished.
6. Look in the "url" folder for files "links1.out.txt", "links2.out.txt", "links3.out.txt" which should have been created. These will contain the page titles alongside the URLs.

Contents of "retrieveWebTitles.py"

# ****************************************************************************************************************************************************************
# Batch Retrieve Web Titles From URLs
# 
# DESCRIPTION: this is a batch version of 
# Extract the title from a web page using 
# the standard lib.
# ^^^
# I would prefer to just use standard Python 
# while I am learning, so we do not use 
# any special libraries like beautiful soup.
# ****************************************************************************************************************************************************************
# BACKGROUND: I keep my bookmarks in a spreadsheet 
# which stores the URL, date visited, title, and 
# a bunch of other columns such as tags, notes, etc. 
# Somehow it got corrupted and the titles are wrong 
# for all 120,000+ URLs. I found a couple of free online 
# tools to batch retrieve Web titles but they choke on 
# this huge list, so I wrote this script to do the job. 
# You can just leave it running on a spare computer overnight 
# or for a couple of days. It takes a list of URLs 
# (actually several lists, each one corresponding to 
# a different spreadsheet tab) and goes online and 
# retrieves the titles. 
# ****************************************************************************************************************************************************************
# Input : One or more text files named like "myfile.txt", 
#         each containing a list of URLs, 
#         with one URL per line. 
#         File names are hardcoded in "main" function with "arrList.append". 
#         Files are expected to be in folder "url" in the same folder as this script.
# ****************************************************************************************************************************************************************
# Output: One or more text files, named like "myfile.out.txt", 
#         with one URL and Web page title per line delimited by tab, 
#         in the format "<url/>\t<title/>\n"
# ****************************************************************************************************************************************************************
# Current issues + questions:
# 1. Speed:         need to make it run faster
# 
# 2. Exceptions     not sure if I am handling exceptions right, 
#                   sometimes the code in except blows up
#                   so I put that inside a try/except
# 
# 3. File encoding: script was blowing up with some error that 
#                   upon googling seemed to be because it was reading text 
#                   file where it expected ascii but was utf8. 
#                   I want script to work with both so I wrote a hack function
#                   "getFileEncoding" that checks. There is probably a better 
#                   way to handle this and probably other types of encoding.
# 
# 4. GUI:           Eventually it would be cool to have this run in a GUI Window 
#                   with a file dialog to select input folder/files, 
#                   and display a progress bar while running. 
#                   I have not done any GUI in Python, any suggestions? 
#                   Maybe Kivy or PyQT or Windows Forms in IronPython 
#                   http://www.voidspace.org.uk/ironpython/winforms/index.shtml
#                   (since I am in Windows)?
# 
# 5. Unknown:       I don't really know Python so any advice on 
#                   what could be done better? 
#                   I am looking to keep the code easy to understand 
#                   and maintain, rather than advanced or complicated, 
#                   mainly I want to fix anything that is 
#                   breaking any basic rules or doing something totally wrong,
# 
# ****************************************************************************************************************************************************************

# ----------------------------------------------------------------------------------------------------------------------------------------------------------------
# The code that gets the Web page titles is based on code from:
# 
# Extract the title from a webpage using the python 3 standard lib - Code Review Stack Exchange
# https://codereview.stackexchange.com/questions/183160/extract-the-title-from-a-webpage-using-the-python-3-standard-lib

# Here is a fault tolerant HTMLParser implementation.
# You can throw pretty much anything at get_title() without it breaking, 
# If anything unexpected happens get_title() will return None.
# When Parser() downloads the page it encodes it to ASCII 
# regardless of the charset used in the page ignoring any errors. 
# It would be trivial to change to_ascii() to convert the data into UTF-8 
# or any other encoding. 
# Just add an encoding argument and rename the function to something like to_encoding().
# By default HTMLParser() will break on broken html, 
# it will even break on trivial things like mismatched tags. 
# To prevent this behavior I replaced HTMLParser()'s error method 
# with a function that will ignore the errors.

#!/usr/bin/python3

#-*-coding:utf8;-*-
#qpy:3
#qpy:console
# ^^^ NO IDEA WHAT THESE 3 LINES ARE?? 

import os
import re
import urllib
from urllib.request import urlopen # is this needed if we already imported all of urllib? 
from html.parser import HTMLParser
from pathlib import Path

from urllib.request import Request # is this needed if we already imported all of urllib? 
from urllib.error import URLError, HTTPError

# ----------------------------------------------------------------------------------------------------------------------------------------------------------------
# Time out process code from: 
# Python 101: How to timeout a subprocess | The Mouse Vs. The Python
# https://www.blog.pythonlibrary.org/2016/05/17/python-101-how-to-timeout-a-subprocess/

import subprocess

# ----------------------------------------------------------------------------------------------------------------------------------------------------------------
# Continuation of code from 
# Extract the title from a webpage using the python 3 standard lib - Code Review Stack Exchange
# https://codereview.stackexchange.com/questions/183160/extract-the-title-from-a-webpage-using-the-python-3-standard-lib

def error_callback(*_, **__):
    pass

def is_string(data):
    return isinstance(data, str)

def is_bytes(data):
    return isinstance(data, bytes)

def to_ascii(data):
    if is_string(data):
        try:
            data = data.encode('ascii', errors='ignore')
        except:
            try:
                data = str(data).encode('ascii', errors='ignore')
            except:
                try:
                    data = str(data)
                except:
                    data = "(could not encode data string)"
    elif is_bytes(data):
        try:
            data = data.decode('ascii', errors='ignore')
        except:
            try:
                data = str(data).encode('ascii', errors='ignore')
            except:
                try:
                    data = str(data)
                except:
                    data = "(could not encode data bytes)"
    else:
        try:
            data = str(data).encode('ascii', errors='ignore')
        except:
            data = "(could not encode data)"

    return data

class Parser(HTMLParser):

    def __init__(self, url):
        self.title = None
        self.rec = False

        HTMLParser.__init__(self)

        try:
            # Added urlopen Timeout parameter so script doesn't freeze up:
            #self.feed(to_ascii(urlopen(url).read()))
            self.feed(to_ascii(urlopen(url, None, 5).read()))
        except Exception as err:
            # Not sure if I am handling exception right, script sometimes dies here:
            try:
                self.feed(str(err))
            except:
                self.feed("(unknown error in urlopen)")

        self.rec = False
        self.error = error_callback

    def handle_starttag(self, tag, attrs):
        if tag == 'title':
            self.rec = True

    def handle_data(self, data):
        if self.rec:
            self.title = data

    def handle_endtag(self, tag):
        if tag == 'title':
            self.rec = False

def get_title(url):
    try:
        return Parser(url).title
    except:
        return "(unknown error in Parser)"

# ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
# Some other (untested) method of getting web title, from 
# 
# html - How can I retrieve the page title of a webpage using Python? - Stack Overflow
# https://stackoverflow.com/questions/51233/how-can-i-retrieve-the-page-title-of-a-webpage-using-python)
# 
# Rahul Chawla answered Jan 31 '17 at 12:46
# No need to import other libraries. 
# Request has this functionality in-built.
# >> hearders = {'headers':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0'}
# >>> n = requests.get('http://www.imdb.com/title/tt0108778/', headers=hearders)
# >>> al = n.text
# >>> al[al.find('<title>') + 7 : al.find('</title>')]
# u'Friends (TV Series 1994\u20132004) - IMDb' 

# ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
# Function that gets # of lines in a text file, based on code found at:

# text files - How to get line count cheaply in Python? - Stack Overflow
# https://stackoverflow.com/questions/845058/how-to-get-line-count-cheaply-in-python

# Kyle answered Jun 19 '09 at 19:07
# One line, probably pretty fast:

# ****************************************************************************************************************************************************************
# NOTE: I added an try/catch to try utf8 encoding if it failed.
#       There is probably a better way, not sure 
#       what other encoding I might want to look for,
#       right now I just have utf8 and ascii files, 
#       so script just needs to handle those.
# ****************************************************************************************************************************************************************

def fileLen(sFilePath):
    try:
        num_lines = sum(1 for line in open(sFilePath))
    except UnicodeDecodeError as ude:
        try:
            num_lines = sum(1 for line in open(sFilePath, encoding="utf8"))
        except:
            num_lines = -1
    return num_lines

# ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
# Some dumb way I came up with to check to see if file is ascii 
# or unicode or something else, based on the try/catch 
# I added to fileLen when it was blowing up.

def getFileEncoding(sFilePath):
    sType = ""
    try:
        sType = "ascii"
        num_lines = sum(1 for line in open(sFilePath))
    except UnicodeDecodeError as ude:
        try:
            sType = "utf8"
            num_lines = sum(1 for line in open(sFilePath, encoding="utf8"))
        except:
            sType = "other"
            num_lines = -1
    return sType

# ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
# Function that reads URLs from a text file sInputFile
# named like "myfile.txt"
# and gets the page title for each, 
# and writes the URL + tab + title 
# to an output file named "myfile.out.txt".
# 
# Based on code from:
# Extract the title from a webpage using the python 3 standard lib - Code Review Stack Exchange
# https://codereview.stackexchange.com/questions/183160/extract-the-title-from-a-webpage-using-the-python-3-standard-lib
#
# and whatever I could find on how to read/write text files in Python.
# 
# TODO: figure out some other method to get titles for ftp:// and other non-http URL protocols.
# TODO: just use the file name for images, PDFs (URLs ending in .jpg, .jpeg, .pdf, etc.)

def getTitles(sInputFile, sStatus):
    sResult = ""
    iLineNum = 0
    iCount = 0
    iTitle = 0
    iNull = 0
    iTimeouts = 0

    if Path(sInputFile).is_file():
        sInputFile = str(sInputFile)
        sOutputFile = sInputFile.replace(".txt", ".out.txt")

        iLineCount = fileLen(sInputFile)
        print("File \"" + sInputFile + "\" has " + str(iLineCount) + " lines.")
        #print("File \"" + sInputFile + "\":")

        sEncoding = getFileEncoding(sInputFile)
        if (sEncoding == "ascii"):
            print("File encoding = ASCII")
            #fIn = open("url.txt", "r")
            fIn = open(sInputFile, "r")
        elif (sEncoding == "utf8"):
            print("File encoding = UTF8")
            fIn = open(sInputFile, "r", encoding="utf8")
        else:
            print("*** File encoding unknown ***")

        #TODO: open output file in ascii or utf8 mode depending on sEncoding
        #fOut = open("title.txt","w+")
        #fOut = open(sOutputFile,"w+")
        fOut = open(sOutputFile,"w+", encoding="utf-8")

        fLines = fIn.readlines()
        for sLine in fLines:
            iLineNum += 1

            sLine = str(sLine)
            sLine = repr(sLine)

            #print(get_title('http://www.google.com'))

            #fOut.write("This is line %d\r\n" % (i+1))
            #fOut.write(get_title('http://www.google.com') + "\r\n")
            sLine = sLine.lstrip('\'')
            sLine = sLine.rstrip('\'')

            sLine = sLine.strip('\\n')
            sLine = sLine.strip('\\r')
            sLine = sLine.strip('\\n')

            if sLine != "":
                iCount += 1
                sTitle = get_title(sLine)
                if sTitle is None:
                    iNull += 1
                    sTitle = ''
                else:
                    iTitle += 1

                # If title is blank then just use the URL as the description for now.
                if str(sTitle)=="":
                    sTitle = sLine

                sTitle = sTitle.replace('\n', ' ').replace('\r', ' ')
                sTitle = re.sub('\s+', ' ', sTitle).strip()

                print(sStatus + "Line " + str(iLineNum) + " of " + str(iLineCount))
                #print(str(iLineNum) + " of " + str(iLineCount) + ": " + sLine + '\t' + sTitle)
                #print(sLine + '\t' + sTitle)

                ##print(sLine)
                ##print(sTitle)
                #print("")

                ##fOut.write(get_title(sLine) + "\r\n")

                #fOut.write(sLine + '\t' + sTitle + '\r\n')
                fOut.write(sLine + '\t' + sTitle + '\n')
            else:
                print (str(iLineNum) + " of " + str(iLineCount) + ": (Skipping blank line.)")
                #print("(Skipping blank line.)")
        fIn.close()
        fOut.close()

        sResult = "Retrieved " + str(iTitle) + " titles, " + str(iNull) + " empty, " + str(iTimeouts) + " timeouts, " + "from \"" + sInputFile + "\", output to \"" + sOutputFile + "\"."
    else:
        sResult = "File \"" + sInputFile + "\" not found."

    return sResult

    # END getTitles

# ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

def main():

    # TODO: save start time

    # Get full path to this current script, based on code from:
    # Open file in a relative location in Python - Stack Overflow
    # https://stackoverflow.com/questions/7165749/open-file-in-a-relative-location-in-python/51671107
    # Russ answered Aug 23 '11 at 18:59
    script_dir = os.path.dirname(__file__) # <-- absolute dir the script is in
    #print ("script_dir=" + script_dir)

    # Specifies subfolder (should be in same folder as this script)
    # that holds the input text files (and where output files are saved):
    # TODO: maybe accept a command line parameter for a different folder name or path
    sSubfolder = "url"

    # For now just add file names here hardcoded:
    # TODO: automatically process all *.txt files in "url" folder that don't end in ".out.txt"
    arrList = []
    arrList.append("links1.txt")
    arrList.append("links2.txt")
    arrList.append("links3.txt")

    # Test code method #1 to traverse array (can't remember if it worked):
    #for iLoop in range(len(arrList)):
    #    print(arrList(iLoop))

    # Traverse array and process each file:
    iCount = 0
    sTotal = str(len(arrList))
    for sInputFile in arrList:
        iCount += 1
        sStatus = "File " + str(iCount) + " of " + sTotal + ", "

        # Get filename with full path, and fix forward/back slashes in path
        # (I am on Windows so some parts have backslashes and not others):
        sInputFile = str(Path(os.path.join(script_dir, sSubfolder, sInputFile)))
        #print(str(iCount) + ". " + sInputFile)

        # Get the web titles for all the urls in the file:
        sResult = getTitles(sInputFile, sStatus)

        # Ouptut summary of results for the current file:
        print(str(iCount) + ". " + sResult)

        # Test output fileLen:
        #print("    fileLen: " + str(fileLen(sInputFile)) )

    # ALL FINISHED:
    # TODO: save end time and display run duration as days/hours/minutes/seconds
    print("Done.")

# ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
# RUNS FIRST, STARTS main SUBROUTINE:

if __name__== "__main__":
    main()

Contents of "links1.txt"

https://codereview.stackexchange.com/questions/215849/batch-retrieve-web-titles-from-urls
https://www.blog.pythonlibrary.org/2017/06/15/python-101-working-with-dates-and-time/
https://stackoverflow.com/questions/311627/how-to-print-a-date-in-a-regular-format
https://www.w3resource.com/python-exercises/python-basic-exercise-3.php
https://www.tutorialspoint.com/python3/python_date_time.htm

Contents of "links2.txt"

https://stackoverflow.com/questions/21618351/format-time-string-in-python-3-3
https://stackoverflow.com/questions/415511/how-to-get-the-current-time-in-python
https://docs.python.org/3/library/time.html#time.strftime
https://elearning.wsldp.com/python3/python-get-current-date/

Contents of "links3.txt"

https://docs.python.org/3/download.html
https://docs.python.org/3/archives/python-3.7.3rc1-docs-pdf-letter.zip
https://media.readthedocs.org/pdf/python-guide/latest/python-guide.pdf
https://www.datacamp.com/community/tutorials/python-data-science-cheat-sheet-basics
https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PythonForDataScience.pdf
https://www.python.org/community/logos/
https://www.python.org/static/community_logos/python-logo.png

Answer 1

Since there is a lot going on here I will not be able to address all of your questions, but I hope my feedback will help you nevertheless.

General Remarks

In-source Documentation

When I first had a look at your code I was stunned by the sheer amount of stuff that was happing. On the second look a lot of the content presented itself as extensive documentation on the sources of the code, its intention, and similar things.
Don't get me wrong here, well documented code is a joy to read and review. Contrary to what you will usually see in the wild, I'm inclined to say yours is a little bit to much for a source file. From my point of view a lot of the "introduction" section might actually better be suited to go into a seperate README file. This will help to reduce the amount of "noise" that presents itself to a reviewer or future you who wants to have a look at the code (only).

Seperation of Concerns / Structure

I would like remind you that keeping your code organized in digestible chunks is not something that is reserved to libraries. You can easily do this with your code as well. For example you could have a file where you collect all your functions which are used to determine the encoding, another file containing the parsers can than import this file to prepare website content. You could then import the parser in your main script file and use it without ever seeing the whole encoding detection again. Looking further, this would also give you the ability to easier test those seperate files.

The Python Code

After the general remarks I want to focus more on the code itself. The following points are in no particular order and I make them up as I go.

Header

Most Python script files will start with the following or similar lines

#!/usr/bin/python3
#-*-coding:utf8;-*-

The first line is the so called shebang used primarily on Unix systems which tells the system which interpreter to use when someone asks the system to treat the file as executable. Here it's the python3 executable.
The next line tells the interpreter how the source file is encoded. As with most of the time one can read more about this in the corresponding PEP.

In addition to these two, there is a sheer endless collection of IDE specific "magic comments" that help your IDE or their integrated tools to work better with your source files. Since this is not Python-specific, I will not go into details here.

imports

Here, I would like to address your comments on is this needed if we already imported all of urllib?. The short answer is usually no. It just saves you from writing urllib.request.urlopen whenever you want to use urlopen. The same goes for Request.

Another note on imports would be to keep your inputs grouped together, e.g. all imports from urllib should be in one block, like so:

import os
import re
import subprocess

import urllib
from urllib.request import urlopen, Request
from urllib.error import URLError, HTTPError

from html.parser import HTMLParser
from pathlib import Path

This will help you to easier find an import statement later, even without IDE support. See also the infamous PEP8 Style Guide for more details on this topic.

Style

Since we have opened the discussion on PEP8, I would like to give some hints to general style as well. Most Python code tends to use snake_case in function and variable names, following the PEP8 style guide. Also, those "type prefixes" like i or s sometimes found in C code are extremely uncommon in Python code. These aspects are no strict requirement for the code to work, just a recommendation. But however you choose to name your variables and functions be consistent, so either use snake_case or camelCase, not both. I know that, strictly speaking, not all of it is not "your" code, but you are the one who will have to work it ad it's a painfull experience not only to have to remember what function to use best, but also having to think about to how write the name.
Most IDEs have tools that can check PEP8 or other style guidelines automatically and remind you with sadistic thoroughness on were you have violated the "rules". Flake8 and Pylint are just some of the examples.

Documentation

Since we've already had a look at this topic I just want to give some further hints on this topic which are specific for Python source code.
Python has built-in support for docstrings of functions and classes, e.g. by using the help function. For this to work you have to provide your documentation in a way Python expects.

For example, you have a nice description on what getTitles is supposed to do. However, Python does not know about it, since it's freely floating in the source file. Enter docstrings, which is basically just the Python name for function and class documentation in the source file. Usually these docstrings are enclosed in """<doc here>""" and can span multiple lines. A version of getTitles which works with the built-in help system can be found below:

def getTitles(sInputFile, sStatus):
    """Function that reads URLs from a text file sInputFile

    Read URLs from a file named like "myfile.txt" and gets the page title for each, 
    and writes the URL + tab + title  to an output file named "myfile.out.txt".

    Based on code from:
    Extract the title from a webpage using the python 3 standard lib - Code Review Stack Exchange
    https://codereview.stackexchange.com/questions/183160/extract-the-title-from-a-webpage-using-the-python-3-standard-lib
        and whatever I could find on how to read/write text files in Python.

    TODO: figure out some other method to get titles for ftp:// and other non-http URL protocols.
    TODO: just use the file name for images, PDFs (URLs ending in .jpg, .jpeg, .pdf, etc.)
    """
    ...

If you were to call help(getTitles) on this version of the function, Python would be able to find your documentation. Sweet, isn't it?

String Formatting

I'm one of the persons who tends to think not everything got better with Python 3. However, one of the most convenient things to work with (and badly miss in Python 2) are the so called f-strings. An f-string is convenient way to express things like sStatus = "File " + str(iCount) + " of " + sTotal + ", " simple as sStatus = f"File {iCount} of {sTotal}, ". Wonderful.
See this blog post for a nice comparison between the different ways to format string in Python.

Opening and Closing Files

It is common knowledge that you have to close file if you open it. So far so good. But what happens if there is an exception between opening and closing the file? Not so nice. But wait, there is help! Conceptually similar to C++ RAII idiom, python offers something called context managers. A context manager in Python is a construct that aquires a resource when it is instantiated and also releases the resource no matter how the context is left.
So for example instead of

infile = open("test.txt", "r")
... # do something with infile, better not break somethig
infile.close()

one could simple write

with open("test.txt", "r") as infile:
    ... # do something with infile, now can even throw!

See also this SO post on how to open multiple files in a single with statement.

(Not so) Final Notes

There are a few topics that I have not touched, e.g. the chained handling of exceptions in the encoding detection. Maybe someone else or tomorrow me will have a go at them. Till then, happy coding and welcome to the world of Python!