Collect Pokémon from a URL and store them in a dataframe

Question

if __name__ == '__main__':
    url='http://pokemondb.net/pokedex/all'
    #Create a handle, page, to handle the contents of the website
    page = requests.get(url)
    #Store the contents of the website under doc
    doc = lh.fromstring(page.content)
    #Parse data that are stored between <tr>..</tr> of HTML
    tr_elements = doc.xpath('//tr')

    # Create empty list
    col = []
    
    # For each row, store each first element (header) and an empty list
    for t in tr_elements[0]:
      
        name = t.text_content()
       
        col.append((name, []))

    # Since out first row is the header, data is stored on the second row onwards
    for j in range(1, len(tr_elements)):
        # T is our j'th row
        T = tr_elements[j]

        # If row is not of size 10, the //tr data is not from our table
        if len(T) != 10:
            break

        # i is the index of our column
        i = 0

        # Iterate through each element of the row
        for t in T.iterchildren():
            data = t.text_content()
            # Check if row is empty
            if i > 0:
                # Convert any numerical value to integers
                try:
                    data = int(data)
                except:
                    pass
            # Append the data to the empty list of the i'th column
            col[i][1].append(data)
            # Increment i for the next column
            i += 1

    Dict = {title: column for (title, column) in col}
    df = pd.DataFrame(Dict)

In the above code, I am calling an API and storing the data in tr_elements, and then by using for loop I am trying to append the headers(names of columns) and empty list to col list then I am iterating through each element of tr_elements and appending to the empty list (col list created before) after converting any numerical data to an integer type. In the end, I am creating a dictionary and then converting it to a data frame (screenshot attached). So, I want to know if I can write the above code more efficiently in a pythonic way?

Answer 1

• Not having any idea what lh is, I can't recommend using it over BeautifulSoup
• Your main guard is a good start, but you should write some actual methods
• Use https unless you have a really, really, really good reason
• Don't blindly attempt to convert every single cell to an integer - you know which ones should and should not have ints so use that knowledge to your advantage
• Never except/pass
• The table does not have jagged cells, so use of a proper selector will not need your length-ten check
• Don't use j-indexing into the rows - just iterate over search results. Also, selecting only rows in the tbody will obviate your one-row index skip
• Don't capitalize local variables like Dict
• Consider using the data's # as the dataframe index

Suggested

from typing import Iterable, Dict, Any

import pandas as pd
import requests
from bs4 import Tag, BeautifulSoup


def get_page_rows() -> Iterable[Dict[str, Tag]]:
    with requests.get('https://pokemondb.net/pokedex/all') as resp:
        resp.raise_for_status()
        doc = BeautifulSoup(resp.text, 'lxml')

    table = doc.select_one('table#pokedex')
    heads = [th.text for th in table.select('thead th')]

    for row in table.select('tbody tr'):
        yield dict(zip(heads, row.find_all('td')))


def tags_to_dict(row: Dict[str, Tag]) -> Dict[str, Any]:
    data = {
        k: int(row[k].text)
        for k in (
            # Skip Total - it's a computed column
            'HP', 'Attack', 'Defense', 'Sp. Atk', 'Sp. Def', 'Speed',
        )
    }
    data.update((k, row[k].text) for k in ('#', 'Name'))

    return data


if __name__ == '__main__':
    df = pd.DataFrame.from_records(
        (tags_to_dict(row) for row in get_page_rows()),
        index='#',
    )