I am doing some practicing with back-connect proxies and asyncio/aiohttp. The IP changes on every request to the proxy and I am allowed up to 1500 threads. The big issue with back-connect proxies, is you do not always get a proxy that works. This forces me to keep requesting the URL until there are no errors so I can return the HTML.

I am mainly asking for a code review to see if using a while loop is the best option to keep retrying the request. And how I can speed up the script overall? I have been looking into different HTML parsers, using multiprocessing, etc... but just not sure what direction to go in. The code below does work fine, and it processes 12,000 URLs in about 30 minutes. I really appreciate you looking at my code and let me know if you have any questions! Thank you for your time.

import asyncio
import aiohttp

from bs4 import BeautifulSoup
from datetime import datetime
from aiohttp.resolver import AsyncResolver

class Proxy:
    def __init__(self, headers, proxy):
        self.headers = headers
        self.proxy = proxy

    async def build(self, urls):
        tasks = []
        resolver = AsyncResolver(nameservers=["", ""])
        connector = aiohttp.TCPConnector(limit=1500, limit_per_host=0, resolver=resolver, use_dns_cache=False)
        async with aiohttp.ClientSession(connector=connector) as session:
            for url in urls:
                task = self.fetch(session, url)
            r = await asyncio.gather(*tasks)
            return r

    async def fetch(self, session, url):
        while True:
                async with session.get(url, headers=self.headers, proxy=self.proxy) as resp:
                    assert resp.status == 200
                    r = await resp.read()
                    soup = BeautifulSoup(r, 'lxml')
                    html_title = soup.title.string
                    print (html_title)
                    return html_title
            except Exception as e:
                print (e)

if __name__ == '__main__':
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36'}
    proxy = 'XXX.XX.X.XXX:XXXXX'
    proxy = "http://{}".format(proxy)

    px = Proxy(headers, proxy)

    urls= []
    with open('urls.txt') as f:
        for line in f:

    startTime = datetime.now()
    loop = asyncio.new_event_loop()
    future = asyncio.ensure_future(px.build(urls))
    html_title = loop.run_until_complete(future)
    print (datetime.now() - startTime)
  • \$\begingroup\$ Please do not update the code in your question to incorporate feedback from answers, doing so goes against the Question + Answer style of Code Review. This is not a forum where you should keep the most updated version in your question. Please see what you may and may not do after receiving answers. \$\endgroup\$
    – Mast
    Apr 10, 2018 at 17:24
  • \$\begingroup\$ Sorry about that and noted for next time. \$\endgroup\$ Apr 10, 2018 at 18:41

1 Answer 1


with back-connect proxies, is you do not always get a proxy that works.

There are several possibilities here. The origin webserver could be responsive, slow (slower than your timeout), or dead, and the proxy could be responsive, slow, or dead. Call those "health" details. It doesn't look like you are tracking health details of either one, so you don't adapt to such details. For faster processing, use adaptive timeouts and don't re-submit requests to dead hosts (don't contact known-dead proxies, and don't contact known-dead webservers).

EDIT: An example of an adaptive timeout would be triple the observed median response time of a host, which attempts to capture a large fraction of the pdf (probability distribution function) of its observed response times, conditioned on actually getting a response.

class Proxy:
    def __init__(self, headers, proxy):

I'm not crazy about that last identifier. Call it proxy_url, please. It clearly isn't an instance of the Proxy class.

  • \$\begingroup\$ I changed proxy to proxy_url. Do you have an example of tracking these health details? Also, what do you mean adaptive timeouts? Timeouts that change based on the health details? When you say dead hosts, are you referring to the server I am requesting or the proxy I am requesting? An example of what you mention would be great. Regardless, thank you for your input and taking your time reading my code. \$\endgroup\$ Apr 10, 2018 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.