1
\$\begingroup\$

I'll put my complete code below so that anyone who can help won't have any difficulty running, just copy it and it's 100% functional, just very very slow.

To register the data I need, I make several requests on different API links, a tangle of calls. This makes it very slow and takes forever to parse each of the ID's.

I need help improving my code as I only know how to work this way it currently is.

import requests
import csv

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
    }
with open("SofaScore/Goals_Mommentum.csv", "a", newline="", encoding="UTF-8") as f:
    
    datasparaurl = ["2021-01-01","2021-01-02","2021-01-03","2021-01-04","2021-01-05"]
    for dataparaurl in datasparaurl:
        url1 = f'https://api.sofascore.com/api/v1/sport/football/scheduled-events/{dataparaurl}'
        response1 = requests.get(url1, headers=headers).json()
        events = response1['events']
        for event in events:
            nometimeA = event['homeTeam']['shortName']
            nometimeB = event['awayTeam']['shortName']
            nomedojogo = str(nometimeA) + " x " + str(nometimeB)
            description = event['status']['description']
            identidade = event['id']
            jogofinalizado = event['status']['type']
            if (jogofinalizado == 'finished'):
                url2 = f'https://api.sofascore.com/api/v1/event/{identidade}/incidents'
                response2 = requests.get(url2, headers=headers).json()
                if 'incidents' in response2:
                    incidents = response2['incidents']
                    for incident in incidents:
                        if (incident['incidentType'] == 'goal'):
                            dadosdogol = []
                            dadosdogol.append(nomedojogo)
                            dadosdogol.append(identidade)
                            goalminute = incident['time']
                            if(goalminute >=0 and goalminute <=44 or goalminute >=46 and goalminute <=89):
                                dadosdogol.append(goalminute)
                                url3 = f'https://api.sofascore.com/api/v1/event/{identidade}/graph'
                                response3 = requests.get(url3, headers=headers).json()
                                if 'graphPoints' in response3:
                                    graphs = response3['graphPoints']
                                    for graph in graphs:
                                        if(graph['minute'] == goalminute-5):
                                            dadosdogol.append(abs(graph['value']))
                                        if(graph['minute'] == goalminute-4):
                                            dadosdogol.append(abs(graph['value']))
                                        if(graph['minute'] == goalminute-3):
                                            dadosdogol.append(abs(graph['value']))
                                        if(graph['minute'] == goalminute-2):
                                            dadosdogol.append(abs(graph['value']))
                                        if(graph['minute'] == goalminute-1):
                                            dadosdogol.append(abs(graph['value']))
                                    print(dadosdogol)
                                    a = csv.writer(f)
                                    a.writerow(dadosdogol)
        print(dataparaurl)
    f.close()
\$\endgroup\$
2
  • \$\begingroup\$ Why are you excluding goals whose time is exactly 45 minutes? \$\endgroup\$
    – Reinderien
    Dec 13, 2021 at 13:47
  • \$\begingroup\$ Hi @Reinderien Because if a goal happens in extra time, for example in the first half 45 minutes + 3 extra time, the data on the graph disappears in the history, so if you were to try to collect it on the graph, it would only take from 45 to the back, in 45+1 , 45+2 and so on would be missing. \$\endgroup\$ Dec 13, 2021 at 13:59

1 Answer 1

3
\$\begingroup\$

very, very slow

This is unavoidable - you're sending up to three HTTP requests for every single row of your output.

Also note that it's "at least immoral" to hammer this site; the owners have tried to protect it with CloudFlare and want you to ask them for permission:

The system identified you as a scraper and banned the IP. To use the data on the website contact the owner and request permission.

That aside, for your code itself,

  • Write some functions; don't write one giant blob of global code
  • Separate downloading, filtering and serialising concerns
  • Don't write code in Portuguese - write it in English, which is an international de-facto standard for coding
  • Mommentum is spelled Momentum
  • Refactor your goal-minutes statements into a loop
  • Don't f.close(); you've already done this in your with
  • Maintain a session object for your requests
  • Use PEP484 type hints

Suggested

from csv import DictWriter
from datetime import date, timedelta
from typing import Iterator, Any, Dict, Iterable, Tuple, List
from urllib.parse import urljoin

from requests import Session


JsonDict = Dict[str, Any]
EventsAndGoals = Tuple[JsonDict, JsonDict]
EventsGoalsGraphs = Tuple[JsonDict, JsonDict, List[JsonDict]]


def get_sofascore(session: Session, path: str) -> JsonDict:
    url = urljoin('https://api.sofascore.com', path)
    with session.get(url, headers={'Accept': 'application/json'}) as response:
        response.raise_for_status()
        return response.json()


def get_events(session: Session, start_date: date, days: int) -> Iterator[Dict[str, Any]]:
    for day in range(days):
        when = start_date + timedelta(days=day)
        response = get_sofascore(
            session=session,
            path=f'/api/v1/sport/football/scheduled-events/{when}',
        )
        for event in response['events']:
            if event['status']['type'] == 'finished':
                yield event


def get_goals(session: Session, events: Iterable[JsonDict]) -> Iterator[EventsAndGoals]:
    for event in events:
        response = get_sofascore(
            session=session,
            path=f'/api/v1/event/{event["id"]}/incidents',
        )
        for incident in response.get('incidents', ()):
            if (
                incident['incidentType'] == 'goal'
                and incident['time'] < 90
                and incident['time'] != 45  # why?
            ):
                yield event, incident


def get_graphs(session: Session, events_and_goals: Iterable[EventsAndGoals]) -> Iterator[EventsGoalsGraphs]:
    for event, goal in events_and_goals:
        response = get_sofascore(
            session=session,
            path=f'https://api.sofascore.com/api/v1/event/{event["id"]}/graph',
        )
        graph = response.get('graphPoints')
        if graph:
            yield event, goal, graph


def make_row(
    event: JsonDict,
    goal: JsonDict,
    graph: List[JsonDict],
) -> Dict[str, Any]:
    row = {
        'name': f'{event["homeTeam"]["shortName"]} x {event["awayTeam"]["shortName"]}',
        'id': event['id'],
        'minutes': goal['time'],
    }

    for point in graph:
        delta = goal['time'] - point['minute']
        if 0 < delta <= 5 and float(delta).is_integer():
            row[f'tminus{delta}'] = abs(point['value'])

    return row


def main() -> None:
    with Session() as session:
        events = get_events(
            session=session, start_date=date(2021, 1, 1), days=5,
        )
        events_and_goals = get_goals(session, events)
        all_sources = get_graphs(session, events_and_goals)
        rows = (make_row(*source) for source in all_sources)

        with open("Goals_Momentum.csv", "w", newline="", encoding="UTF-8") as f:
            writer = DictWriter(
                f=f,
                fieldnames=(
                    'name', 'id', 'minutes',
                    *(f'tminus{i}' for i in range(5, 0, -1)),
                ),
            )
            writer.writerows(rows)


if __name__ == '__main__':
    main()
\$\endgroup\$
2
  • \$\begingroup\$ Hi @Reinderien , thank you so much for everything and I would like to ask you one more question, just out of curiosity. I scraped 6 months of data with that question code using VScode and didn't get this IP ban return message. Could it be that the slowness in collecting my data didn't get me banned or was it pure luck? I really didn't know there was this banning possibility, they pointed out the API links and I was using it as if there was no problem. \$\endgroup\$ Dec 13, 2021 at 14:53
  • 1
    \$\begingroup\$ It's probably because I removed the user agent in my own code. \$\endgroup\$
    – Reinderien
    Dec 13, 2021 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.