0
\$\begingroup\$

Can someone help me how to speed up this scraping process getting all the usernames from the # 'cats' on instagram. My goal is to make this as fast as possible because currently the process is kinda slow. Maybe using multithreading?

from instaloader import Instaloader

HASHTAG = 'cats'

loader = Instaloader(sleep=False)

users = []
for post in loader.get_hashtag_posts(HASHTAG):
    if post.owner_username not in users:
        users.append(post.owner_username) 
    print(post.owner_username)

The following is a profiling run, ordered by: cumulative time.

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    331/1    0.002    0.000   18.736   18.736 {built-in method builtins.exec}
        1    0.001    0.001   18.736   18.736 GetUsernamesFromHashtags.py:1(<module>)
       36    0.001    0.000   18.468    0.513 instaloadercontext.py:354(get_json)
       36    0.000    0.000   18.453    0.513 sessions.py:528(get)
       36    0.001    0.000   18.452    0.513 sessions.py:457(request)
       36    0.001    0.000   18.368    0.510 sessions.py:608(send)
       36    0.001    0.000   18.126    0.504 adapters.py:394(send)
       36    0.001    0.000   18.009    0.500 connectionpool.py:446(urlopen)
      492    0.001    0.000   18.002    0.037 socket.py:575(readinto)
      492    0.001    0.000   18.000    0.037 pyopenssl.py:295(recv_into)
      492    0.002    0.000   17.999    0.037 SSL.py:1795(recv_into)
      492   17.994    0.037   17.994    0.037 {built-in method _openssl.SSL_read}
       36    0.001    0.000   17.993    0.500 connectionpool.py:319(_make_request)
       36    0.000    0.000   17.912    0.498 client.py:1292(getresponse)
       36    0.001    0.000   17.911    0.498 client.py:299(begin)
     2521    0.002    0.000   17.872    0.007 {method 'readline' of '_io.BufferedReader' objects}
       36    0.001    0.000   17.860    0.496 client.py:266(_read_status)
       35    0.000    0.000   17.134    0.490 structures.py:180(owner_username)
       35    0.000    0.000   17.134    0.490 structures.py:165(owner_profile)
       35    0.000    0.000   17.133    0.490 structures.py:141(_full_metadata)
       35    0.001    0.000   17.133    0.490 structures.py:132(_obtain_metadata)
       36    0.000    0.000    1.336    0.037 instaloader.py:835(get_hashtag_posts)
\$\endgroup\$
0

1 Answer 1

3
\$\begingroup\$

You probably won't be able to do much to speed up get_hashtag_posts since that's an API call; if you try to hammer it by running multiple queries in parallel, a well-designed API will interpret that as a DDOS attack and rate-limit you.

As far as your code goes, though, you should be using a set instead of a list, since sets are optimized for the exact thing you're doing:

from typing import Set


users: Set[str] = set()
for post in loader.get_hashtag_posts(HASHTAG):
    users.add(post.owner_username)

This should be a constant-time operation, whereas a list will take longer to search the longer it is (linear-time).

\$\endgroup\$
2
  • 1
    \$\begingroup\$ NameError: name 'Set' is not defined. how can I fix that? (sry I'm new to python) \$\endgroup\$
    – kk_Chiron
    Feb 9, 2020 at 21:32
  • \$\begingroup\$ from typing import Set (at the top of the file) \$\endgroup\$
    – Samwise
    Feb 9, 2020 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.