3
\$\begingroup\$
if __name__ == '__main__':
    url='http://pokemondb.net/pokedex/all'
    #Create a handle, page, to handle the contents of the website
    page = requests.get(url)
    #Store the contents of the website under doc
    doc = lh.fromstring(page.content)
    #Parse data that are stored between <tr>..</tr> of HTML
    tr_elements = doc.xpath('//tr')

    # Create empty list
    col = []
    
    # For each row, store each first element (header) and an empty list
    for t in tr_elements[0]:
      
        name = t.text_content()
       
        col.append((name, []))

    # Since out first row is the header, data is stored on the second row onwards
    for j in range(1, len(tr_elements)):
        # T is our j'th row
        T = tr_elements[j]

        # If row is not of size 10, the //tr data is not from our table
        if len(T) != 10:
            break

        # i is the index of our column
        i = 0

        # Iterate through each element of the row
        for t in T.iterchildren():
            data = t.text_content()
            # Check if row is empty
            if i > 0:
                # Convert any numerical value to integers
                try:
                    data = int(data)
                except:
                    pass
            # Append the data to the empty list of the i'th column
            col[i][1].append(data)
            # Increment i for the next column
            i += 1

    Dict = {title: column for (title, column) in col}
    df = pd.DataFrame(Dict)

In the above code, I am calling an API and storing the data in tr_elements, and then by using for loop I am trying to append the headers(names of columns) and empty list to col list then I am iterating through each element of tr_elements and appending to the empty list (col list created before) after converting any numerical data to an integer type. In the end, I am creating a dictionary and then converting it to a data frame (screenshot attached). So, I want to know if I can write the above code more efficiently in a pythonic way?

Screen shot of final data frame after running my code

\$\endgroup\$
4
  • \$\begingroup\$ Welcome to Code Review! We need to know what the code is intended to achieve. To help reviewers give you better answers, please add sufficient context to your question, including a title that summarises the purpose of the code. We want to know why much more than how. The more you tell us about what your code is for, the easier it will be for reviewers to help you. The title needs an edit to simply state the task, rather than your concerns about the code. \$\endgroup\$ Jul 1, 2021 at 15:32
  • \$\begingroup\$ Please include your imports. \$\endgroup\$
    – Reinderien
    Jul 3, 2021 at 2:48
  • \$\begingroup\$ Why is it going to Pandas? \$\endgroup\$
    – Reinderien
    Jul 3, 2021 at 2:49
  • \$\begingroup\$ Finally: no, you're not calling an API; you're scraping a web page. \$\endgroup\$
    – Reinderien
    Jul 3, 2021 at 2:50

1 Answer 1

2
\$\begingroup\$
  • Not having any idea what lh is, I can't recommend using it over BeautifulSoup
  • Your main guard is a good start, but you should write some actual methods
  • Use https unless you have a really, really, really good reason
  • Don't blindly attempt to convert every single cell to an integer - you know which ones should and should not have ints so use that knowledge to your advantage
  • Never except/pass
  • The table does not have jagged cells, so use of a proper selector will not need your length-ten check
  • Don't use j-indexing into the rows - just iterate over search results. Also, selecting only rows in the tbody will obviate your one-row index skip
  • Don't capitalize local variables like Dict
  • Consider using the data's # as the dataframe index

Suggested

from typing import Iterable, Dict, Any

import pandas as pd
import requests
from bs4 import Tag, BeautifulSoup


def get_page_rows() -> Iterable[Dict[str, Tag]]:
    with requests.get('https://pokemondb.net/pokedex/all') as resp:
        resp.raise_for_status()
        doc = BeautifulSoup(resp.text, 'lxml')

    table = doc.select_one('table#pokedex')
    heads = [th.text for th in table.select('thead th')]

    for row in table.select('tbody tr'):
        yield dict(zip(heads, row.find_all('td')))


def tags_to_dict(row: Dict[str, Tag]) -> Dict[str, Any]:
    data = {
        k: int(row[k].text)
        for k in (
            # Skip Total - it's a computed column
            'HP', 'Attack', 'Defense', 'Sp. Atk', 'Sp. Def', 'Speed',
        )
    }
    data.update((k, row[k].text) for k in ('#', 'Name'))

    return data


if __name__ == '__main__':
    df = pd.DataFrame.from_records(
        (tags_to_dict(row) for row in get_page_rows()),
        index='#',
    )
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.