3
\$\begingroup\$

As a fan of competitive Counter-Strike, I like to keep up with who is currently winning and who is losing. There is a website that provides me with just that. I thought it would be cool if I could display the results in a way different than visiting the website.

Here is an example result from the source of the website.

<div class="result-con" data-zonedgrouping-entry-unix="1499867767000"><a href="/matches/2312610/penta-vs-kinguin-esea-premier-season-25-europe" class="a-reset">
                        <div class="result">
                          <table>
                            <tr>
                              <td class="team-cell">
                                <div class="line-align team1">
                                  <div class="team ">PENTA</div>
<img alt="PENTA" src="https://static.hltv.org/images/team/logo/5395" class="team-logo" title="PENTA"></div>
                              </td>
                              <td class="result-score"><span class="score-lost">5</span> - <span class="score-won">16</span></td>
                              <td class="team-cell">
                                <div class="line-align team2"><img alt="Kinguin" src="https://static.hltv.org/images/team/logo/6134" class="team-logo" title="Kinguin">
                                  <div class="team bold">Kinguin</div>
                                </div>
                              </td>
                              <td class="event"><img alt="ESEA Premier Season 25 Europe" src="https://static.hltv.org/images/eventLogos/2879.png" class="event-logo smartphone-only" title="ESEA Premier Season 25 Europe"><span class="event-name">ESEA Premier Season 25 Europe</span></td>
                              <td class="star-cell">
                                <div class="map-and-stars">
                                  <div class="stars"><i class="fa fa-star star"></i></div>
                                  <div class="map map-text">trn</div>
                                </div>
                              </td>
                            </tr>
                          </table>
                        </div>
                      </a></div>

I haven't done anything relating to web scraping before, so this is my first take at it. The code does all that I want it to, however looking at it gives me headaches. It's not pretty and most likely not very efficient either. I'd appreciate constructive feedback on it.

I am aware that the code is not reusable and I should probably refactor it, but I want to learn more efficient scraping methods first.

Each match result contains a timestamp, which refers to the date the game was played. In the for loop, I compare it to today's date and if they match, the match information is assigned to appropriate variables.

#!/usr/bin/python3
from time import localtime, strftime

import requests
from bs4 import BeautifulSoup

RESULTS_PAGE_URL = 'https://www.hltv.org/results?stars=1'
current_date = strftime('%B %d %Y')

html_source = requests.get(RESULTS_PAGE_URL).text
soup = BeautifulSoup(html_source, 'lxml')

print(soup.title.text)
print('Results for {}\n'.format(current_date))

for result in soup.find_all('div', class_='result-con'):
    timestamp = int(result['data-zonedgrouping-entry-unix']) / 1000
    match_date = strftime('%B %d %Y', localtime(timestamp))
    if match_date == current_date:
        losing_team = result.find('div', class_='team ').text
        losing_team_score = result.find('span', class_='score-lost').text

        winning_team = result.find('div', class_='team bold').text
        winning_team_score = result.find('span', class_='score-won').text

        event = result.find('span', class_='event-name').text

        print('{} {} - {} {}'.format(winning_team, winning_team_score, losing_team_score, losing_team))
        print('{}\n'.format(event))
\$\endgroup\$
4
\$\begingroup\$

The code is pretty clean, great job.

Some of the things I would point out:

  • there is a way to find only the current date games directly - without filtering them out in the loop. We can incapsulate the filtering logic in a searching function:

    def current_games(tag):
        is_result_tag = tag.name == 'div' and 'result-con' in tag.get('class', [])
        if not is_result_tag:
            return False
    
        timestamp = int(tag['data-zonedgrouping-entry-unix']) / 1000
        return strftime('%B %d %Y', localtime(timestamp)) == current_date
    
    
    for result in soup(current_games):
    

    Note that soup(current_games) is a shorthand for soup.find_all(current_games).

  • I personally prefer CSS selectors to find* methods

  • .get_text() is generally more preferable than .text
  • you can use f-strings if on Python 3.6

Modified code:

#!/usr/bin/python3
from time import localtime, strftime

import requests
from bs4 import BeautifulSoup


RESULTS_PAGE_URL = 'https://www.hltv.org/results?stars=1'
current_date = strftime('%B %d %Y')

html_source = requests.get(RESULTS_PAGE_URL).text
soup = BeautifulSoup(html_source, 'lxml')

print(soup.title.get_text())
print(f'Results for {current_date}\n')


def current_games(tag):
    is_result_tag = tag.name == 'div' and 'result-con' in tag.get('class', [])
    if not is_result_tag:
        return False

    timestamp = int(tag['data-zonedgrouping-entry-unix']) / 1000
    return strftime('%B %d %Y', localtime(timestamp)) == current_date


for result in soup(current_games):
    losing_team = result.select_one('.team').get_text()
    losing_team_score = result.select_one('.score-lost').get_text()

    winning_team = result.select_one('.team.bold').get_text()
    winning_team_score = result.select_one('.score-won').get_text()

    event = result.select_one('.event-name').get_text()

    print(f'{winning_team} {winning_team_score} - {losing_team_score} {losing_team}')
    print(f'{event}\n')

# fnatic will always be the best

A more generic solution that also handles ties might be:

for result in soup(current_games):
    teams = [team.get_text() for team in result.select(".team")]
    score = [item.get_text() for item in result.select("[class^=score]")]

    event = result.select_one('.event-name').get_text()

    print(f'{teams[0]} {score[0]} - {teams[1]} {score[1]}')
    print(f'{event}\n')
\$\endgroup\$
  • \$\begingroup\$ +1 for the review, -1 for fnatic... :P I'm currently moving everything into a class, can I get ya to review it for me later? If you don't mind that is. \$\endgroup\$ – Lukasz Salitra Jul 12 '17 at 22:28
  • \$\begingroup\$ I completely forgot about best-of-two's, which can result in a tie. I get an AttributeError in the for loop, as tags differ for a match that results in a tie. I'll edit the OP accordingly. \$\endgroup\$ – Lukasz Salitra Jul 13 '17 at 12:33
  • \$\begingroup\$ @LukaszSalitra sure, no problem. Added a snippet to handle any types of scores - we are losing information which of the teams lost and won, but we can add it if needed, no problem. Thanks. \$\endgroup\$ – alecxe Jul 13 '17 at 13:33
  • \$\begingroup\$ This works okay: result.select_one('.line-align.team1').get_text().strip('\n') and allows me to preserve the winner/loser information. .line-align.team2 works for the other tied team. What's a different way of pulling the team name out that tag, rather than stripping the newlines from the string? \$\endgroup\$ – Lukasz Salitra Jul 13 '17 at 17:56
  • 1
    \$\begingroup\$ @LukaszSalitra it'll strip whitespaces, tab chars and newlines from both ends of a text..two words should not be a problem as long as they are in the same child tag..test it out, thanks. \$\endgroup\$ – alecxe Jul 13 '17 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.