1
\$\begingroup\$

I need to make multiple http requests very fast. I've currently use this method.

import queue as Queue
import threading
from threading import Thread
import os
import requests

Lock = threading.Lock()
Queue = Queue.Queue(maxsize=0)
for i in range(1000):
    Queue.put((i,i+1)) # These Are Examples

Task_Done = 0
Task_Total = 1000
os.system("title Task Done : {}".format(Task_Done))

class MultiThreading(threading.Thread):
    def __init__(self,Queue):
        threading.Thread.__init__(self)
        self.Queue = Queue
    def run(self):
        global Lock
        global Task_Done
        TaskLeft = self.Queue.get()
        Part1,Part2 = TaskLeft
        try:
            A = requests.get("https://twitter.com/{}{}".format(Part1,Part2))
            B = A.text
            Task_Done += 1
            with Lock:
                os.system("title Task Done : {}".format(Task_Done))
        except:
            self.Queue.put((Part1,Part2))
        self.Queue.task_done()

while not (Task_Total == Task_Done):
    for i in range(100):
        try:
            t = MultiThreading(Queue)
            t.setDaemon(True)
            t.start()
        except:
            time.sleep(5)

I'm currently new to this so excuse my messy code. I want to know what's the most efficient, best, and fastest way of sending HTTP requests.

\$\endgroup\$
  • 1
    \$\begingroup\$ You're importing threads and queues, but still hand-maintained global locks? Why did you write it this way? Were you specifically working around a specific bug or problem? \$\endgroup\$ – Mast Aug 27 '18 at 13:07
  • \$\begingroup\$ What do you want to do with the results? I suggest that you show us that code too, so that we can give you better advice. \$\endgroup\$ – 200_success Aug 27 '18 at 16:24
  • \$\begingroup\$ If you want more speed, look into the MultiProcessing module instead, as Threading can only ever execute a single thread on the GIL at a time. \$\endgroup\$ – C. Harley Aug 28 '18 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.