Here's what I did:
First, I went to the site in the browser, started capturing network traffic with developer tools, and scrolled all the way down the page as far as possible. Then I exported the log to a .csv file. This produced a bunch of URLs like:
Luckily, the full resolution version (for almost every image) is located in the same place, except with "1920x1200" substituted at the end:
I ran the following script to retrieve the full resolution images:
import csv import os import re import urllib.request target_dir = 'C:/Users/[Username]/Desktop/Bing' def main(): os.chdir(target_dir) with open('NetworkData.csv', 'r', encoding='utf-8') as csvfile: for line in csv.reader(csvfile): url_request = repr(line) if r'vo.msecnd.net/files' in url_request: pic_url = re.sub('_\d+x\d+\.jpg', '_1920x1200.jpg', url_request) pic_url = pic_url.strip('\'"') try: urllib.request.urlretrieve(pic_url, pic_url.split('/')[-1]) except: print("Image not found.") if __name__ == '__main__': main()
I thought this solution was actually okay, but I was wondering if there was a way to automate or avoid the manual step of capturing the network traffic. I was hoping to avoid selenium if possible, because it looks kind of slow and cumbersome. Also, if you have any advice on improving the script, that would be good. I had some issues with string encoding, and I feel like in the end the script isn't very readable.