We’re rewarding the question askers & reputations are being recalculated! Read more.
3 edited tags
| link
2 correction
| link

Scrape Scraping the full content from a lazy-loading webpage

1
source | link

Scrape the full content from a lazy-loading webpage

I've written a script in python in combination with selenium which is able to scrape 1000 links from a webpage in which lazy-loading method is applied for that reason it displays it's content 20 at a time and full content can only be seen when it is made to scroll downmost. However, my script can scroll the webpage to the end. After collecting the 1000 links from main page, it then gets to each individual link to scrape the Name of CEO and Web address of that organization. It is working great now. I tried to make the whole thing accordingly. Here is the full code:

from selenium import webdriver
import time

def get_links(driver):

    driver.get('http://fortune.com/fortune500/list/')

    while True:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(3)

        links = [posts.get_attribute("href") for posts in driver.find_elements_by_xpath("//li[contains(concat(' ', @class, ' '), ' small-12 ')]//a")]
        if (len(links) == 1000):
                break

    for link in links:
        process_links(driver, link)

def process_links(driver, sub_links):

    driver.get(sub_links)

    for items in driver.find_elements_by_xpath('//div[contains(@class,"company-info-card-table")]'):
        Name = items.find_element_by_xpath('.//div[contains(@class,"small-7")]/p[@class="remove-bottom-margin"]')
        Web = items.find_element_by_xpath('.//div[contains(@class,"small-9")]/a')

        print(Name.text, Web.get_attribute("href"))

if __name__ == '__main__':

    driver = webdriver.Chrome()
    try:
        get_links(driver)
    finally:
        driver.quit()