5
\$\begingroup\$

I am working on a personal project where I analyse Big Data by using twitter statuses. Therefore I am trying to query the Twitter API as many times as I can.

The API has a limit of 180 queries per 15 minutes, so I figured I can do 1 query every 5 seconds (at the minimum) if I want to not exceed that limit.

In the code below I do this the old-fashioned way with System.currentTimeMillis():

while (true) {
    newStatuses.clear();

    long startTime = System.currentTimeMillis();
    // Querying the API 
    newStatuses.addAll(DbTools.TWITTER_FACTORY.search(_query).getTweets());
    long totalTime = System.currentTimeMillis() - startTime;

    allStatuses.addAll(newStatuses);
    Thread.sleep((totalTime > 5000) ? 0 : 5000 - totalTime);
    System.out.println(System.currentTimeMillis());
}

Pretty much if the total querying time took more than 5 seconds then sleep() for 0 seconds, if it took less than or equal 5 seconds then sleep() for the time that is remaining.

What I basically want is the time period from between hitting the line newStatuses.addAll(DbTools.TWITTER_FACTORY.search(_query).getTweets()); each time to be at least 5 seconds.

Is there any better way to do this or do you have any suggestions of how I can improve my solution?

\$\endgroup\$
8
\$\begingroup\$

Rather than waiting for 5 second between calls, consider that Twitter API returns three HTTP headers that explicitly tells you your quota:

  • X-Rate-Limit-Limit: the rate limit ceiling for that given request
  • X-Rate-Limit-Remaining: the number of requests left for the 15 minute window
  • X-Rate-Limit-Reset: the remaining window before the rate limit resets in UTC epoch seconds

You can just query as much as you like, until the X-Rate-Limit-Remaining reaches zero or until you receive a 429 Too Many Request status code, then sleep for X-Rate-Limit-Reset.

This gives you several advantages:

  1. You're utilising your limit to the fullest
  2. If twitter changes their rate limit (e.g. someone is DDoSing them and they need to stricken it up or they make a significant upgrade that allows them to increase their quota), then your application will automatically adapt.
  3. sleeping can't take into account of the per-user quota when you share access tokens for multiple applications/multiple purposes

Refer to Twitter documentation: https://dev.twitter.com/docs/rate-limiting/1.1

\$\endgroup\$
3
  • \$\begingroup\$ I like this solution but it seems like that running getRateLimitStatus().get("/application/rate_limit_status").getRemaining() counts as a query on itself somehow because if I have exceeded the limit and I try to run that then I get the limit exception again. So I guess I will inevitably have to wait for the exception to happen and then run a worker for X-Rate-Limit-Reset that keeps a flag false and when it's done the flag turns true. \$\endgroup\$
    – Aki K
    Aug 21 '14 at 14:43
  • \$\begingroup\$ If I understands Twitter docs correctly, you should get the HTTP response headers mentioned above on every API responses, not just responses to "GET /application/rate_limit_status". You do not need to make any additional queries if you find out your remaining limit from the response headers. IOW, you don't need to make requests to /application/rate_limit_status. \$\endgroup\$
    – Lie Ryan
    Aug 21 '14 at 15:45
  • \$\begingroup\$ I now found the correct way of doing it, instead of explicitly making an API call for the rate limit I just need to store DbTools.TWITTER_FACTORY.search(_query) in a QueryResult variable and that object contains that information as well. \$\endgroup\$
    – Aki K
    Aug 21 '14 at 16:00
3
\$\begingroup\$

I recommend using Java timers or schedulers. I personally like ScheduledThreadPoolExecutor. They are more elegant way to code. Sleep utilizes resources when waiting.

ScheduledThreadPoolExecutor

import java.util.concurrent.ScheduledThreadPoolExecutor;
ScheduledThreadPoolExecutor executer = new ScheduledThreadPoolExecutor(10);
executer.scheduleWithFixedDelay(new StartWork(), 0, 5, TimeUnit.SECONDS);

Sample Class

class StartWork implements Runnable {

    @Override
    public void run() {
     // Your logic
    }
}

Please note that your logic this way will execute every 5 seconds irrespective of network delays in API calls. Since we initialized with 10 threads, a new worker thread will start incase previous result is still being fetched.

\$\endgroup\$
2
  • 3
    \$\begingroup\$ I don't think that sleep is busy waiting. The documentation is not too good, but I found this from oracle and these two questions on stackoverflow. See also this in-depth post \$\endgroup\$
    – tim
    Aug 21 '14 at 10:48
  • \$\begingroup\$ Please fix your example, it should always contain compilable code. It is allowed to give excerpts, as long as they can compile. In your case you cannot mix up an import-statement with other code in this way. \$\endgroup\$
    – skiwi
    Aug 21 '14 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.