# Extracting necessary records from LinkedIn

I wanted to create a scraper in python which can fetch required data from LinkedIn. I tried with python in many different ways but I could not make it until I used selenium in combination with. However, I have created it and got it working as I wanted.

The most difficult part I had to face while making this crawler is that there are hundreds of profile pages which can be located with mostly three different XPath patterns. I somehow managed to cement the three different XPath patterns into one. Now it is working great.

This scraper firstly clicks on the view all recommendation tab in home page then parse 200 profiles [customized in this case] by going to the main page of each profile. I've tried to make it error-free. Here is what I've done:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait.until(EC.visibility_of_element_located((By.XPATH, "//a[contains(@class,'feed-s-follows-module__view-all')]")))
driver.find_element_by_xpath("//a[contains(@class,'feed-s-follows-module__view-all')]").click()

while True:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

break

try:
for item in driver.find_elements_by_xpath("//div[contains(@class,'pv-top-card-section__information') or contains(@class,'org-top-card-module__details') or (@class='org-top-card-module__main-column')]"):
name = item.find_element_by_xpath(".//h1[@title]|.//h1[contains(@class,'pv-top-card-section__name')]").text
except Exception as e:
print(e)

finally:
try:
print(name, title)
except Exception as ex:
print(ex)

if __name__ == '__main__':

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

try:
finally:
driver.quit()

• Is this your own code? This question appears to have been posted before, under a different account. – 200_success Jul 17 '17 at 22:13

I would recommend a more modular design - having a LinkedInScraper class, initialized with a login and password and with separate methods for logging in and getting profile links.

Also, I think you are overusing the XPaths overall - whenever possible, first explore if you can use "by id", "by name" or "by css selector" locators and fall back to XPath only if you cannot get to the element with other locators.

Also note that wait.until combined with built-in expected conditions returns a WebElement instance - if you are waiting for a specific element and then clicking it - you can do it in one go without re-finding the element.

Unfortunately, cannot test the below code (for some reason, I don't see the recommendation link on the main page when logging in with my credentials), but hope this is still useful:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

self.driver = webdriver.Chrome()
self.wait = WebDriverWait(self.driver, 10)

def __del__(self):
self.driver.close()

while True:
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

break

def profiles(self):

for item in self.driver.find_elements_by_xpath("//div[contains(@class,'pv-top-card-section__information') or contains(@class,'org-top-card-module__details') or (@class='org-top-card-module__main-column')]"):
name = item.find_element_by_xpath(".//h1[@title]|.//h1[contains(@class,'pv-top-card-section__name')]").text

yield (name, title)

if __name__ == '__main__':

I am pretty sure we can also refactor the profile() method, but I cannot get to that page in order to see if locators can be simplified.
• @ShahinIqbal ah, sure, needed the yield from, fixed, please try again. – alecxe Jul 17 '17 at 14:13