Getting death row inmates' last statement

Question

Initial version of the code appeared as an answer to an SO question. I refactored the code a bit and it works pretty well, IMHO. I get a solid .csv file with all the data from Texas Department of Criminal Justice's Death Row page.

What I was especially interested in was getting all offenders' last statement, if there was any, which the codes accomplishes.

What I'd want to get here is some feedback on utilizing pandas, as I'm relatively new to it. Also, some memory efficiency suggestions would be nice too, I guess.

For example, should I save the initial version of the .csv file and then read it so I can append the last statements? Or keeping everything in memory is fine?

If you find any other holes, do point them out!

The code:

import random
import time

import pandas as pd
import requests
from lxml import html

base_url = "https://www.tdcj.texas.gov/death_row"
statement_xpath = '//*[@id="content_right"]/p[6]/text()'


def get_page() -> str:
    return requests.get(f"{base_url}/dr_executed_offenders.html").text


def clean(first_and_last_name: list) -> str:
    name = "".join(first_and_last_name).replace(" ", "").lower()
    return name.replace(", Jr.", "").replace(", Sr.", "").replace("'", "")


def get_offender_data(page: str) -> pd.DataFrame:
    df = pd.read_html(page, flavor="bs4")
    df = pd.concat(df)
    df.rename(
        columns={'Link': "Offender Information", "Link.1": "Last Statement URL"},
        inplace=True,
    )

    df["Offender Information"] = df[
        ["Last Name", 'First Name']
    ].apply(lambda x: f"{base_url}/dr_info/{clean(x)}.html", axis=1)

    df["Last Statement URL"] = df[
        ["Last Name", 'First Name']
    ].apply(lambda x: f"{base_url}/dr_info/{clean(x)}last.html", axis=1)
    return df


def get_last_statement(statement_url: str) -> str:
    page = requests.get(statement_url).text
    statement = html.fromstring(page).xpath(statement_xpath)
    text = next(iter(statement), "")
    return " ".join(text.split())


def get_last_statements(offenders_data: list) -> list:
    statements = []
    for item in offenders_data:
        *names, url = item
        print(f"Fetching statement for {' '.join(names)}...")
        statements.append(get_last_statement(statement_url=url))
        time.sleep(random.randint(1, 4))
    return statements


if __name__ == "__main__":
    offenders_df = get_offender_data(get_page())
    names_and_urls = list(
        zip(
            offenders_df["First Name"],
            offenders_df["Last Name"],
            offenders_df["Last Statement URL"],
        )
    )
    offenders_df["Last Statement"] = get_last_statements(names_and_urls)
    offenders_df.to_csv("offenders_data.csv", index=False)

The scraping part is intentionally slow, as I don't want to abuse the server, but I do want to get the job done. So, if you don't have a couple of minutes to spare, you can fetch the offenders_data.csv file from here.

Answer 1

Types

Here:

def clean(first_and_last_name: list) -> str:

list should more specifically be List[str].

Fluent syntax

Consider using fluent syntax for your clean, which - in this case - is basically just a reformat:

    return (
        "".join(first_and_last_name)
        .replace(" ", "")
        .replace(", Jr.", "")
        .replace(", Sr.", "")
        .replace("'", "")
        .lower()
    )