-1
\$\begingroup\$

This code scrapes www.oddsportal.com for all the URLs provided in the code and appends it to a dataframe.

I am not very well versed with iterative logic hence I am finding it difficult to improvise on it.

Code:

import pandas as pd
from selenium import webdriver
from bs4 import BeautifulSoup as bs


browser = webdriver.Chrome()


class GameData:

    def __init__(self):
        self.date = []
        self.time = []
        self.game = []
        self.score = []
        self.home_odds = []
        self.draw_odds = []
        self.away_odds = []
        self.country = []
        self.league = []


def parse_data(url):
    browser.get(url)
    df = pd.read_html(browser.page_source, header=0)[0]
    html = browser.page_source
    soup = bs(html, "lxml")
    cont = soup.find('div', {'id': 'wrap'})
    content = cont.find('div', {'id': 'col-content'})
    content = content.find('table', {'class': 'table-main'}, {'id': 'tournamentTable'})
    main = content.find('th', {'class': 'first2 tl'})
    if main is None:
        return None
    count = main.findAll('a')
    country = count[1].text
    league = count[2].text
    game_data = GameData()
    game_date = None
    for row in df.itertuples():
        if not isinstance(row[1], str):
            continue
        elif ':' not in row[1]:
            game_date = row[1].split('-')[0]
            continue
        game_data.date.append(game_date)
        game_data.time.append(row[1])
        game_data.game.append(row[2])
        game_data.score.append(row[3])
        game_data.home_odds.append(row[4])
        game_data.draw_odds.append(row[5])
        game_data.away_odds.append(row[6])
        game_data.country.append(country)
        game_data.league.append(league)
    return game_data


urls = {
"https://www.oddsportal.com/soccer/england/premier-league/results/#/page/1",
"https://www.oddsportal.com/soccer/england/premier-league/results/#/page/2",
"https://www.oddsportal.com/soccer/england/premier-league/results/#/page/3",
"https://www.oddsportal.com/soccer/england/premier-league/results/#/page/4",
"https://www.oddsportal.com/soccer/england/premier-league/results/#/page/5",
"https://www.oddsportal.com/soccer/england/premier-league/results/#/page/6",
"https://www.oddsportal.com/soccer/england/premier-league/results/#/page/7",
"https://www.oddsportal.com/soccer/england/premier-league/results/#/page/8",
"https://www.oddsportal.com/soccer/england/premier-league/results/#/page/9",
"https://www.oddsportal.com/soccer/england/premier-league-2019-2020/results/#/page/1",
"https://www.oddsportal.com/soccer/england/premier-league-2019-2020/results/#/page/2",
"https://www.oddsportal.com/soccer/england/premier-league-2019-2020/results/#/page/3",
"https://www.oddsportal.com/soccer/england/premier-league-2019-2020/results/#/page/4",
"https://www.oddsportal.com/soccer/england/premier-league-2019-2020/results/#/page/5",
"https://www.oddsportal.com/soccer/england/premier-league-2019-2020/results/#/page/6",
"https://www.oddsportal.com/soccer/england/premier-league-2019-2020/results/#/page/7",
"https://www.oddsportal.com/soccer/england/premier-league-2019-2020/results/#/page/8",
"https://www.oddsportal.com/soccer/england/premier-league-2019-2020/results/#/page/9",
"https://www.oddsportal.com/soccer/england/premier-league-2018-2019/results/#/page/1",
"https://www.oddsportal.com/soccer/england/premier-league-2018-2019/results/#/page/2",
"https://www.oddsportal.com/soccer/england/premier-league-2018-2019/results/#/page/3",
"https://www.oddsportal.com/soccer/england/premier-league-2018-2019/results/#/page/4",
"https://www.oddsportal.com/soccer/england/premier-league-2018-2019/results/#/page/5",
"https://www.oddsportal.com/soccer/england/premier-league-2018-2019/results/#/page/6",
"https://www.oddsportal.com/soccer/england/premier-league-2018-2019/results/#/page/7",
"https://www.oddsportal.com/soccer/england/premier-league-2018-2019/results/#/page/8",
"https://www.oddsportal.com/soccer/england/premier-league-2018-2019/results/#/page/9",
}

if __name__ == '__main__':

    results = None

    for url in urls:
        game_data = parse_data(url)
        if game_data is None:
            continue
        result = pd.DataFrame(game_data.__dict__)
        if results is None:
            results = result
        else:
            results = results.append(result, ignore_index=True)

print(results)

|     | date        | time   | game                             | score   |   home_odds |   draw_odds |   away_odds | country   | league                   |
|-----|-------------|--------|----------------------------------|---------|-------------|-------------|-------------|-----------|--------------------------|
|   0 | 12 May 2019 | 14:00  | Brighton - Manchester City       | 1:4     |       14.95 |        7.75 |        1.2  | England   | Premier League 2018/2019 |
|   1 | 12 May 2019 | 14:00  | Burnley - Arsenal                | 1:3     |        2.54 |        3.65 |        2.75 | England   | Premier League 2018/2019 |
|   2 | 12 May 2019 | 14:00  | Crystal Palace - Bournemouth     | 5:3     |        1.77 |        4.32 |        4.22 | England   | Premier League 2018/2019 |
|   3 | 12 May 2019 | 14:00  | Fulham - Newcastle               | 0:4     |        2.45 |        3.55 |        2.92 | England   | Premier League 2018/2019 |
|   4 | 12 May 2019 | 14:00  | Leicester - Chelsea              | 0:0     |        2.41 |        3.65 |        2.91 | England   | Premier League 2018/2019 |
|   5 | 12 May 2019 | 14:00  | Liverpool - Wolves               | 2:0     |        1.31 |        5.84 |       10.08 | England   | Premier League 2018/2019 |
|   6 | 12 May 2019 | 14:00  | Manchester Utd - Cardiff         | 0:2     |        1.3  |        6.09 |        9.78 | England   | Premier League 2018/2019 |

As you can see the URLs can be optimised to run through all pages in that league/branch

Inspect element

How can I optimise this code to iteratively run for every page?

\$\endgroup\$
7
  • 2
    \$\begingroup\$ Asking for this code to be reviewed is fine, but asking for net new features to be added is off topic \$\endgroup\$
    – Reinderien
    Jun 24 at 12:55
  • \$\begingroup\$ Also this code actually just won't run. You have what appear to be string literals that are unquoted. Also you haven't shown your imports. \$\endgroup\$
    – Reinderien
    Jun 24 at 15:52
  • \$\begingroup\$ @PyNoob codereview.stackexchange.com/questions/263433/… \$\endgroup\$
    – Reinderien
    Jun 24 at 22:28
  • \$\begingroup\$ @Reinderien I am not asking for new features to be built. Apologies if it feels that way. I thought my question explained otherwise. \$\endgroup\$
    – PyNoob
    Jun 24 at 22:57
  • \$\begingroup\$ @Reinderien Yep, apologies, I agree the code did not have imports. I have edited the code. I have also posted an example output. Thanks a ton. Please help to optimise it. \$\endgroup\$
    – PyNoob
    Jun 24 at 23:56
0
\$\begingroup\$
  • It seems that GameData is meant to house multiple games, with lists for each attribute. You should re-represent this as a list of GameData instances where each instance has a scalar for each attribute and not a list.
  • Using Selenium is not out of the question, here, but Requests will be faster. However, to do this you need to perform reverse engineering to understand some of the silly things that the web page designers did, like a character substitution cipher on odds figures.
  • Why are you using Pandas? If it's only for printing data, there's no need for it.
  • First realize that urls is a set, where you want a sequence instead. You should be restructuring your code to accept a sport, country, year range and page range, and generating those URLs as part of the routine.

A suggested scraping program for these data exists here: Scraping OddsPortal with requests only

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.