Scraping with asyncio/aiohttp and retrying the request on error

Question

I am doing some practicing with back-connect proxies and asyncio/aiohttp. The IP changes on every request to the proxy and I am allowed up to 1500 threads. The big issue with back-connect proxies, is you do not always get a proxy that works. This forces me to keep requesting the URL until there are no errors so I can return the HTML.

I am mainly asking for a code review to see if using a while loop is the best option to keep retrying the request. And how I can speed up the script overall? I have been looking into different HTML parsers, using multiprocessing, etc... but just not sure what direction to go in. The code below does work fine, and it processes 12,000 URLs in about 30 minutes. I really appreciate you looking at my code and let me know if you have any questions! Thank you for your time.

import asyncio
import aiohttp

from bs4 import BeautifulSoup
from datetime import datetime
from aiohttp.resolver import AsyncResolver

class Proxy:
    def __init__(self, headers, proxy):
        self.headers = headers
        self.proxy = proxy

    async def build(self, urls):
        tasks = []
        resolver = AsyncResolver(nameservers=["8.8.8.8", "8.8.4.4"])
        connector = aiohttp.TCPConnector(limit=1500, limit_per_host=0, resolver=resolver, use_dns_cache=False)
        async with aiohttp.ClientSession(connector=connector) as session:
            for url in urls:
                task = self.fetch(session, url)
                tasks.append(task)
            r = await asyncio.gather(*tasks)
            return r

    async def fetch(self, session, url):
        while True:
            try:
                async with session.get(url, headers=self.headers, proxy=self.proxy) as resp:
                    assert resp.status == 200
                    r = await resp.read()
                    soup = BeautifulSoup(r, 'lxml')
                    html_title = soup.title.string
                    print (html_title)
                    return html_title
            except Exception as e:
                print (e)

if __name__ == '__main__':
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36'}
    proxy = 'XXX.XX.X.XXX:XXXXX'
    proxy = "http://{}".format(proxy)

    px = Proxy(headers, proxy)

    urls= []
    with open('urls.txt') as f:
        for line in f:
            urls.append(line.strip())

    startTime = datetime.now()
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    future = asyncio.ensure_future(px.build(urls))
    html_title = loop.run_until_complete(future)
    loop.close()
    print (datetime.now() - startTime)

Answer 1

with back-connect proxies, is you do not always get a proxy that works.

There are several possibilities here. The origin webserver could be responsive, slow (slower than your timeout), or dead, and the proxy could be responsive, slow, or dead. Call those "health" details. It doesn't look like you are tracking health details of either one, so you don't adapt to such details. For faster processing, use adaptive timeouts and don't re-submit requests to dead hosts (don't contact known-dead proxies, and don't contact known-dead webservers).

EDIT: An example of an adaptive timeout would be triple the observed median response time of a host, which attempts to capture a large fraction of the pdf (probability distribution function) of its observed response times, conditioned on actually getting a response.

class Proxy:
    def __init__(self, headers, proxy):

I'm not crazy about that last identifier. Call it proxy_url, please. It clearly isn't an instance of the Proxy class.