I've written a script in python in combination with selenium to parse different information from a webpage and store the collected data in a csv file. The data are rightly coming through. The email portion looks weird cause I wrapped two conditions in a single line to grab all the leads (in case one missing, another will come into play) However, the way I've used csv writer doesn't look that good. I hope there is any better way of implementation of the logic I pursued below.

This is the script:

import csv
from selenium import webdriver
from bs4 import BeautifulSoup

urls = "https://www.krak.dk/byggefirmaer/p:{}/s%C3%B8g.cs?xcoord=90.3636094&ycoord=23.8150673"

def get_info(driver,urls):
    with open("output.csv","w",newline="") as infile:
        writer = csv.writer(infile)

        for url in [urls.format(page) for page in range(1,3)]:
            soup = BeautifulSoup(driver.page_source,'html.parser')
            for links in soup.select("article.vcard"):
                name = links.select_one(".hit-company-name-ellipsis a").get_text(strip=True)
                link = links.select_one(".hit-homepage-link").get("href") if links.select_one(".hit-homepage-link") else  ""
                email = links.select_one(".hit-footer-wrapper span[data-mail]").get("data-mail") or links.select_one(".hit-footer-wrapper span[data-mail-e-contact-mail]").get("data-mail-e-contact-mail")

if __name__ == '__main__':
    driver = webdriver.Chrome()

Currently the above script traverses three pages and scrapes three fields name,link,email.

  • \$\begingroup\$ There are small changes I would make (not related to messy usage of links), but in my experience has been web scraping can get very messy. \$\endgroup\$ – Dair Jun 25 '18 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.