Web Scraping with Python

Question

My target website is Indeed. I tried to implement the scraper such that it is convenient for the end user. I am introducing my code in the readme file on my github repo.

I am somewhat a beginner in programming so I am looking for guidance on things such as if the libraries that I used are approriate, the code itself and on generally making the script better.

Link to the README file.

import requests
from bs4 import BeautifulSoup

jobName = input('Enter your desired position: ').replace(' ', '-')
place = input("Enter the location for your desired work(City, state or zip): ")

URL = 'https://www.indeed.com/q-'+jobName+'-l-'+place.replace(' ', '-')+'-jobs.html'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')
pages = soup.find(id='searchCountPages')
noPag = pages.text.split('of')[1].strip(' jobs').replace(',', '')

nmoPag = input(f"There are {noPag} number of pages. If you want to scrape all of them write 'Max' else write number of pages you wish to scrape: ")
if nmoPag == 'Max':
    nmoPag = noPag

for i in range(0, int(nmoPag)*10, 10):
    URL = 'https://www.indeed.com/jobs?q='+jobName+'&l='+place.replace(' ', '+')+'&start='+str(i)
    page = requests.get(URL)
    soup = BeautifulSoup(page.content, 'html.parser')
    results = soup.find(id='resultsCol')
    listings = results.find_all('div', class_='result')

    for job in listings:
        jobT = job.find('a', class_='jobtitle')
        jobL = job.find('span', class_='location').text.strip()
        jobS = job.find('div', class_='summary').text.strip()
        link = jobT['href']
        if any(any(subs in s for s in (jobT.text.strip().lower(), jobS.lower())) for subs in (jobName.split('+')[0], jobName[1])):
            print('Your job in '+jobL+' as a '+ jobT.text.strip()+
                    '.\nHere is a quick summary of your job here: '+
                        jobS+'\nLink for more information and application for the job - https://indeed.com'+link, end='\n\n\n')

Answer 1

What you are doing sounds reasonable overall.

It's good that you are using F-strings:

nmoPag = input(f"There are {noPag} number of pages. If you want to scrape all of them write 'Max' else write number of pages you wish to scrape: ")

But there are many other places where you don't eg:

print('Your job in '+jobL+' as a '+ jobT.text.strip()+
                    '.\nHere is a quick summary of your job here: '+
                        jobS+'\nLink for more information and application for the job - https://indeed.com'+link,

So I would upgrade the rest of the code for more consistency and better readability :)

Consistency: you are mixing lower case and upper case in some variable names eg jobName vs place. Remember that variable names are case-sensitive in Python. The practice could be dangerous and confusing. Imagine that you have jobName and jobname, it's two variables that may be assigned different values.

There is redundancy in the use of functions, for example this bit of code is repeated twice:

jobT.text.strip()

Don't repeat yourself, just assign the result of jobT.text.strip() to a variable once, and reuse it.

More repetition: www.indeed.com is hardcoded 3 times. Define a global variable for the root URL, then add the query string parameters as required.

With the urllib library you could take advantage of the URI-building functions that are available to you. See Creating URL query strings in Python. Tip: if you want to build URIs but don't fire them immediately, you can also use prepared requests.

Although in the present case, the site does not use classic query string parameters separated with a &. So you can instead use an F-string, again, with sanitized variable values:

url = f"https://www.indeed.com/q-{jobName}-l-{place}-jobs.html"

Note regarding user input: the most obvious is to check that the input is not empty. Always trim the input too, because some people might copy-paste text with extra tabs or spaces (think Excel). Maybe you could use a regex to replace multiple occurrences of whitespace with a single hyphen.

I would also add more checks to make sure that all the DOM elements you are expecting were found - because a website is subject to change at any time. When that happens, the code must alert you.

Finally, a quality script should have exception handling (10 lines of code would suffice). Catch exceptions always and log the full details to a file. This will help you a lot with debugging and troubleshooting.

As your code is on Github, some people might want to use it. If they have problems with it, it would be good if they could attach a log in their bug report, so that you get better insight into the error.

Since you are scraping a website, all sorts of errors will happen often: DNS resolution errors, timeouts, requests denied etc. Your script should handle those errors gracefully.