There's a lot to improve about your code, and I'll start from the top. I'll also rewrite some of the logic you had there because IMO it's a bit clumsy and hard to understand.
Imports
Don't use urllib and requests at the same time. Use either one or another. They serve the same purpose.
from bs4 import BeautifulSoup
import os
import random
import requests
As you can see, I've alphabetically ordered them so that it becomes easier to locate each of them.
Constants
You have one url which might be moved at the top of your program. That way, you'll allow your users to change it without searching through all the code.
Moving on, I realised that you didn't give the user a chance to save the pictures wherever he wants. Let's add another constant, say IMAGE_LOCATION.
URL = 'https://pixabay.com'
IMAGE_LOCATION = '/home/images/'
Functions
You can split your code into separate functions so that each of them can serve its own purpose. Read more about Single Responsability Principle:
def get_html_source():
"""Docstring here."""
html_source = requests.get(URL).text
return BeautifulSoup(html_source, 'html.parser')
The above function will return a BeautifulSoup object which contains the html source code of the desired URL.
We can now build another function which will generate a path containing a random name for each picture we're downloading:
def image_name():
"""Docstring here."""
return '{}.jpg'.format(os.path.join(IMAGE_LOCATION, str(random.randrange(1, 100000))))
I've also increased the maximum number just in case there's a huge number of files.
Another helper function would be one that checks if a src URL is a valid one or not. I've modified a bit your condition and made it more strict:
def validate(image_url):
"""Docstring here."""
return image_url.endswith('.jpg') and image_url.startswith('http')
The above, verifies if the url starts with http (implicitly https will pass) and also if it ends in .jpg.
Only now, we can finally create the download_images function, and wrap our logic into it:
def download_images(soup):
"""Docstring here."""
for link in soup.find_all('img'):
image_url = link.get('src')
if validate(image_url):
image = requests.get(image_url)
if image.status_code == 200:
with open(image_name(), 'wb') as f:
f.write(image.content)
As you can see, I've removed most of the logic you had, and simply did the following steps:
- Go through all the
img tags and save their src attribute (which is an URL
- Check the URL if it matches our condition and build our image object
- Double check to see if the status code was successful (200) and save the image to our desired location.
Next, wrapping all our code within a main function is as simple as:
def main():
"""Docstring here."""
soup = get_html_source()
download_images(soup)
All the code
from bs4 import BeautifulSoup
import os
import random
import requests
URL = 'https://pixabay.com'
IMAGE_LOCATION = '/home/images/'
def get_html_source():
"""Docstring here."""
html_source = requests.get(URL).text
return BeautifulSoup(html_source, 'html.parser')
def image_name():
"""Docstring here."""
return '{}.jpg'.format(os.path.join(IMAGE_LOCATION, str(random.randrange(1, 100000))))
def validate(image_url):
"""Docstring here."""
return image_url.endswith('.jpg') and image_url.startswith('http')
def download_images(soup):
"""Docstring here."""
for link in soup.findAll('img'):
image_url = link.get('src')
if validate(image_url):
image = requests.get(image_url)
if image.status_code == 200:
with open(image_name(), 'wb') as f:
f.write(image.content)
def main():
"""Docstring here."""
soup = get_html_source()
download_images(soup)
if __name__ == '__main__':
main()
If you don't know what's with if __name__ == '__main__' you can read more about it here
More, note that all the functions have a docstring which contain a placeholder like "Docstring here.". You can replace that placeholder with a summary that describes what each function does.
wget --accept jpg,png http://example.com. Additional flags like recursive as you need. Note that neitherwgetnor your work-in-progress will get anything inserted into the page with javascript, and that's very common these days. \$\endgroup\$