1
\$\begingroup\$

Currently I need to download thousand of images, I can use 2 functions to download the images: LoadUrl or LoadUrlProxy.

Each function has different parameters:

LoadUrl(url, timeout)
LoadUrlProxy(url, proxy_client, proxy_params)

As I don't want to duplicate the code, I'm looking to simplify the way I'm passing a function reference and its parameters, not sure If below is the best way or I can improve this code?

     # func(Function): Function reference which defines how to download images
      func = LoadUrl
      if use_proxy:
        proxy_client = CreateProxyClient()
        proxy_params = GetProxyParams()        
        func = LoadUrlProxy
      images = []

      with futures.ThreadPoolExecutor(max_workers=_MAX_WORKERS) as executor:
        # Start the load operations and mark each future with its URL
        if use_proxy:
          future_to_url = {
              executor.submit(func, image, proxy_client, proxy_params): image
              for image in image_list
          }
        else:
          future_to_url = {
              executor.submit(func, image, _EXECUTOR_TIMEOUT): image
              for image in image_list
          }
        for future in futures.as_completed(future_to_url):
          image = future_to_url[future]
          try:
            images.append((image.master_id, future.result()))
          except Exception as exc: 
            logging.exception('%r Generated an exception: %s', image.url, exc)
\$\endgroup\$
1
\$\begingroup\$

Why not use functools.partial to simplify further and avoid the if use_proxy a second time?

Something along the lines of

if use_proxy:
    func = functools.partial(
            LoadUrlProxy,
            proxy_client=CreateProxyClient(),
            proxy_params=GetProxyParams())
else:
    func = functools.partial(LoadUrl, timeout=_EXECUTOR_TIMEOUT)

with futures.ThreadPoolExecutor(max_workers=_MAX_WORKERS) as executor:
    future_to_url = {
            executor.submit(func, image): image
            for image in image_list
    }

    for future in futures.as_completed(future_to_url):
        image = future_to_url[future]
        try:
            images.append((image.master_id, future.result()))
        except Exception as exc: 
            logging.exception('%r Generated an exception: %s', image.url, exc)
| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.