# Extract name, address and phone number from some web pages using multiprocessing

I've written a script in Python using the multiprocessing module to scrape values from web pages (one page per subprocess). As I'm very new to multiprocessing, I'm not sure whether I did everything in the right way. It works without error, though.

Here goes the full script:

import requests
from lxml.html import fromstring
from multiprocessing import Pool

response = requests.get(url).text
tree = fromstring(response)
for title in tree.cssselect("div.info"):
name = title.cssselect("a.business-name span")[0].text
try:
except IndexError: street = ""
try:
phone = title.cssselect("div[class^=phones]")[0].text
except IndexError: phone = ""
print(name, street, phone)

if __name__ == '__main__':
links = [link.format(page) for page in range(1,4)]
with Pool(4) as p:


Any idea to make it more robust will be highly appreciated.

• Were you intending to include page 4? If so, you need to change the range's (exclusive) stop value to 5. – Solomon Ucko Nov 16 '18 at 12:35

Proper use of a Pool p should include p.close() and p.join() (in this order).

Cases of the websites not responding should be handled. Requests should have a timeout, the timeout exception is caught, and non-200 should be handled as well.

Other than that, the script is correct for a one time extract of a few pages, but will need to be extended if you intend to produce a high volume daemon. If that is the case, usage of Pool can be questionnable and has multiple alternatives, such as using Scrapy framework scheduler or using a celery broker for high-level handling of workers (this will avoid your workers to crash entirely on exceptions, among a few other benefits).

You've (manually?) escaped your query params in your URL string. This is OK, and technically a few nanoseconds faster, but less legible than the alternative:

requests.get('https://www.yellowpages.com/search',
params={'search_terms': 'coffee',
'geo_location_terms': 'Los Angeles, CA',
'page': page})


Then, rather than calling format, you simply pass in the page parameter.