I am writing a benchmarking tool from scratch in Python.
However I can't get the performance of other benchmarking tools like
wrk2. Using wr2 I can make 42k requests/s while my code can only create up to 2200 reqs/s. I have tried multiple ways to parallelize the code execution. I have tried using multiprocessing and parallel computing libraries like Dask. But I can't get better performance. I understand
wrk2 are written in C which can be one reason but still 42k vs 2200 seems like a very large difference.
I have tried with different number of workers and
number_of_request, but the performance does not change much.
I am trying to understand if I am really hitting the upper limit or I am doing something wrong. The server is running on localhost and written in Java Spring.
This is my code using multiprocessing:
import time import multiprocessing from collections import Counter, defaultdict import requests # import multiprocessing as mp num_workers = multiprocessing.cpu_count() output = multiprocessing.Queue() def runner(number_of_request): output="" for i in range(number_of_request): try: output+=str(requests.get("http://127.0.0.1:8000/").text) except: pass # print(output) return output if __name__ == '__main__': number_of_request = 1000 start = time.time() pool = multiprocessing.Pool(processes=num_workers) outputs = [pool.apply_async(runner, args = (number_of_request,)) for x in range(num_workers)] pool.close() pool.join() duration = time.time() - start req_s = (number_of_request*num_workers)/duration print("duration =", time.time() - start) print(req_s)