5
\$\begingroup\$

I am very new to Python and I am trying to learn through personnal projects and today I needed to collect a lot of baksetball player names and decided this was a good time to learn and practice.

The script is able to pull out all of the players names, height, nationality and DOB but I need to enter the link to the pages before launching it. Next step will be either to build a graphic interface where I can copy/past the links or asking in the console wether I want to enter a new team.

I am looking forward to hearing from you since this is maybe my first personnal project :)

import os
import requests
import pandas as pd
import xlsxwriter
from bs4 import BeautifulSoup

def reverse(name):
    to_reverse = name
    return ' '.join(reversed(name.split(' ')))

def conversion_df(url):
    requete = requests.get(url)
    page = requete.content
    soup = BeautifulSoup(page, features="lxml")

    header = [th.getText() for th in soup.findAll("th")]    # Why do I have to put th. before getText for it to work | Why do we use th as an updator 
    header = header[1:5] # Here we exclude the first (O) entry and we display the entries situated before 5

    row = soup.findAll("tr")[1:]
    stats = [[td.getText().strip() for td in row[i]] for i in range(len(row))]
    for j in stats: 
        del j[0]
        del j[4]
    #print(stats)

    for i in range(len(stats)):
        stats[i][0] = reverse(stats[i][0])

    table = pd.DataFrame(data = stats, columns = header)
    return table

nanterre = conversion_df("https://www.lnb.fr/fr/espa/equipe/espoirs-nanterre-61344.html")
monaco = conversion_df("https://www.lnb.fr/fr/espa/equipe/espoirs-monaco-61343.html")
boulazac = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-boulazac-61331.html")
boulogne = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-boulogne-levallois-61332.html")
bourg = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-bourg-en-bresse-61333.html")
chalon = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-chalon-saone-61334.html")
cholet = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-cholet-61336.html")
chalon_reims = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-chalons-reims-61335.html")
dijon = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-dijon-61337.html")
gravelines = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-gravelines-dunkerque-61338.html")
le_mans = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-le-mans-61339.html")
le_portel = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-le-portel-61340.html")
orleans = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-orleans-61345.html")
pau = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-pau-lacq-orthez-61346.html")
roanne = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-roanne-61347.html")
strasbourg = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-strasbourg-61348.html")
limoges = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-limoges-61341.html")
lyon = conversion_df("https://www.lnb.fr/fr/espoirs/equipe/espoirs-lyon-villeurbanne-61342.html")

clubs = [boulazac,boulogne,bourg,chalon,cholet,chalon_reims,dijon,gravelines,le_mans,le_portel,limoges,lyon,monaco,nanterre,orleans,pau,roanne,strasbourg]

# Creating Excel Writer Object from Pandas  
writer = pd.ExcelWriter('liste_espoir.xlsx',engine='xlsxwriter')   
workbook=writer.book
worksheet=workbook.add_worksheet('Espoirs')
writer.sheets['Espoirs'] = worksheet

row_count = 0
i = 0

for i in range(len(clubs)):
    clubs[i].to_excel(writer,sheet_name='Espoirs',startrow=row_count , startcol=0)   
    row_count += 20

writer.save()
\$\endgroup\$
4
\$\begingroup\$

Welcome to Code Review. Your code looks to be following some good practices from python's style guide (PEP-8). However, as a programmer, you can perhaps improve the structure/performance.

  1. No need to have separate variables for dataframes for each club. Group all the links together, and use a map for fetch everything.
  2. The reverse function defines an unused variables to_reverse.
  3. Split functionality to separate function, just like you have one to reverse names.
  4. You do not need to iterate over range(len(some_iterable)) if the index is not needed at all.
  5. Add type hinting, and some comments to functions to help understand what kind of values it might expect (and return), as well as, what it is doing.
  6. Put everything inside an if __name__ == "__main__" clause.

Rewritten code:

from typing import Tuple
import requests
import pandas as pd
import xlsxwriter
from bs4 import BeautifulSoup


def reverse(name: str) -> str:
    """Reverse name from 'Last First' to 'First Last'."""
    return " ".join(reversed(name.split(" ")))


def fetch_page(url: str) -> str:
    """Send request to given url, and return the contents on success."""
    response = requests.get(url)
    if response.ok:
        return response.content


def get_club_information(club_page) -> Tuple:
    """Read all `tr` elements on page, and extract player information.

    Each row (`tr`) consists of 6 cells. We are skipping over 1st and last cell data."""
    soup = BeautifulSoup(club_page, features="lxml")
    rows = soup.findAll("tr")
    header, *content = [[cell.getText().strip() for cell in row][1:5] for row in rows]
    for row in content:
        row[0] = reverse(row[0])
    return header, content


def conversion_df(url):
    page = fetch_page(url)
    header, content = get_club_information(page)
    print(header, content)
    table = pd.DataFrame(data=content, columns=header)
    return table


LINKS = (
    "https://www.lnb.fr/fr/espa/equipe/espoirs-nanterre-61344.html",
    "https://www.lnb.fr/fr/espa/equipe/espoirs-monaco-61343.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-boulazac-61331.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-boulogne-levallois-61332.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-bourg-en-bresse-61333.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-chalon-saone-61334.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-cholet-61336.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-chalons-reims-61335.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-dijon-61337.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-gravelines-dunkerque-61338.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-le-mans-61339.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-le-portel-61340.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-orleans-61345.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-pau-lacq-orthez-61346.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-roanne-61347.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-strasbourg-61348.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-limoges-61341.html",
    "https://www.lnb.fr/fr/espoirs/equipe/espoirs-lyon-villeurbanne-61342.html",
)


def main():
    # Creating Excel Writer Object from Pandas
    writer = pd.ExcelWriter("liste_espoir.xlsx", engine="xlsxwriter")
    workbook = writer.book
    writer.sheets["Espoirs"] = workbook.add_worksheet("Espoirs")
    row_count = 0
    for club in map(conversion_df, LINKS):
        club.to_excel(writer, sheet_name="Espoirs", startrow=row_count, startcol=0)
        row_count += 20
    writer.save()


if __name__ == "__main__":
    main()

Additionally, you can use asyncio to fetch those pages in parallel, to reduce the runtime for your program.

\$\endgroup\$
1
  • \$\begingroup\$ Thank, you gave me a lot to look into ! \$\endgroup\$
    – nouse
    Sep 29 '20 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.