I've written some code in python to scrape item names, category and the variety of images connected to each item from four different links out of a webpage. The main problem with which i had to hurdle is that each item may have one image or around five different images. Parsing those image links are no big deal but problem appears when it comes to save them in a local folder. Coder like me find it difficult cause few links contain single image and the rest contain several. It was hard for me to twitch the scraper in such a way so that all the images can be parsed and saved properly. I used four different links: two of which contain one image for each item and the rest contain five images for each item . However, my parser is running smoothly now. Here is what I did:
import requests from lxml import html import os urls = [ "http://www.floralart.com.au/products/39-melbourne-online-floristflorist-melbourne-flowers-delivery-funeral-flowers-free-online.aspx", "http://www.floralart.com.au/products/316-amore-mio-vase-arrangement.aspx", "http://www.floralart.com.au/products/90-wedding-florist-wedding-flowers-wedding-bouquet-bridal-flowers-flowers-for-weddings-artificial.aspx", "http://www.floralart.com.au/products/182-flowers-on-line-flower-bouquets-flower-deliveries-flowers-melbourne-cheap-flower-deliveries-mel.aspx", ] def opening_link(link): response = requests.get(link) tree = html.fromstring(response.text) cat = tree.cssselect(".breadcrumb a").text.strip() #category name of the item for item in tree.cssselect(".product-details-page"): name = item.cssselect("h1.productname").text.strip() if item.cssselect("h1.productname") else "" image_link = [img.attrib['src'] for img in item.cssselect(".picture a img")] #for single image image_links = [img_link.attrib['src'] for img_link in item.cssselect(".overview a img") if '.jpeg' in img_link.attrib['src']] #for list of images if image_links: saving_images(cat,name,image_links) else: saving_images(cat,name,image_link) def saving_images(cat, item_name, item_links): for link in item_links: response = requests.get(link, stream = True) image_name = link.split('/')[-1] if response.status_code == 200: os.chdir(r"C:\Users\ar\Desktop\mth") with open(image_name, 'wb') as f: for chunk in response.iter_content(1024): f.write(chunk) print(item_name, cat) if __name__ == '__main__': for url in urls: opening_link(url)
Btw, if there are more than one image in a link then it is not necessary to make use of single image cause the list containing several images also contain the main image.