Python3's async
/await
syntax is great, but it does create a divide between libraries which are async-based and those which are not. For example, boto3 (AWS API library) currently doesn't work with async. There is a separate project, aiobotocore, attempts to recreate some of this functionality in an async context.
I have been thinking for a while about how to create HTTP API clients which can be run async and sync contexts. I have come up with a strategy that involves creating a separation between the logic (preparation of requests, interpretation of responses), and the sending. The logic is implemented as a generator function, which yields out an object which represents the request, and receives back an object representing the response. This generator is "run" by a runner function, which does the actual sending, and may be sync or async.
You can imagine that if, for example, boto3
had been written this way, it would be easy to re-use the bulk of the code to make an async version, rather than having to do an async rewrite.
I have included a toy example below. I would welcome comments.
from typing import Iterable, NamedTuple, Dict, Any, Optional
import requests
from aiohttp import ClientSession
URL_TEMPLATE = "https://api.icndb.com/jokes/{id}/"
class Request(NamedTuple):
method: str
url: str
json: Optional[Dict[str, Any]] = None
class Response(NamedTuple):
status: int
json: Optional[Dict[str, Any]] = None
def get_joke(id: int) -> Iterable[Request]:
response = yield Request("GET", URL_TEMPLATE.format(id=id))
try:
if response.status != 200:
raise JokeApiError("API request failed")
data = response.json
if data.get("type") == 'NoSuchQuoteException':
raise NoSuchJoke(data.get("value", ""))
if data.get("type") != "success":
raise JokeApiError("API request failed")
raise Return(data["value"]["joke"])
except (JokeApiError, Return, StopIteration):
raise
except Exception as e:
raise JokeApiError() from e
def call_api_sync(it):
try:
for req in it:
response = requests.request(req.method, req.url, json=req.json)
it.send(
Response(status=response.status_code, json=response.json())
)
except Return as e:
return e.value
async def call_api_async(it):
async with ClientSession() as session:
try:
for req in it:
async with session.request(
method=req.method,
url=req.url,
json=req.json) as res:
json = await res.json()
response = Response(status=res.status, json=json)
it.send(response)
except Return as e:
return e.value
class JokeApiError(Exception):
pass
class NoSuchJoke(JokeApiError):
pass
class Return(Exception):
def __init__(self, value):
self.value = value
if name...
section as well \$\endgroup\$ – hjpotter92 Jun 28 '18 at 17:23