Build a local cache of all streams from Twitch.TV API

Question

I am trying to cache the entirety of streams from Twitch.TV public API with the following code. It is successful but takes almost a full minute to execute so I am wondering if it's my code or just the speed of the API delivery. In any case, because it is my first PHP script (and really, I'm new in general), I am certain there are logical missteps and performance crushing flows and I would appreciate some review!

The function buildPages() is mainly what is in question for this post.

The data is always returned as json and I dump it all to disk for local testing. Suggestions on how to implement an accelerator for this collection are welcomed since I've never played with one before. APC or Memcache(d)? I have a lot of reading to do..

Also, I am not certain why the lines

$length = count($page->streams);
$length = count($page['streams']);

had to be written as such. stdClass Object versus ArrayAccess? I don't really understand why I would get errors when trying to use the same accessor in the case of my code. Is $page not always an associative array created from the JSON data?

<?php

$base = 'https://api.twitch.tv/kraken/';

$thatlist = array(
    'games'=> array(
        'url'=>'games/top',
        'params'=> array(
            'limit'=>100,
            'offset'=>0
        )
    ),
    'featured'=>array(
        'url'=>'streams/featured',
        'params'=>array(
            'limit'=>100,
            'offset'=>0
        )
    ),
    'summary'=>array(
        'url'=>'streams/summary',
        'params'=>array(
            'limit'=>100,
            'offset'=>0
        )
    ),
    'teams'=>array(
        'url'=>'teams',
        'params'=>array(
            'limit'=>100,
            'offset'=>0
        )
    )
);

foreach($thatlist as $key => $val) {

    $fullurl = $base . $val['url'];

    foreach($val['params'] as $foo => $bar){

        $fullurl .= '?' . $foo . '=' . $bar;

    }

    $quicklist[$key] = json_decode(file_get_contents($fullurl), true);

}

function buildPages() {

    $streams = array();
    $offset = 0;

    $page = json_decode(file_get_contents("https://api.twitch.tv/kraken/streams?offset=".$offset."&limit=100", true));

    $length = count($page->streams);

    array_push($streams, $page);

    // request always returns 100 items unless at the end of live data
    while($length = 100) {

        $index = count($streams);
        $offset = $index * 100;

        $page = json_decode(file_get_contents("https://api.twitch.tv/kraken/streams?offset=".$offset.'&limit=100'), true);

        $length = count($page['streams']);

        echo "Index: ".$index."<br/>";
        echo "Length: ".$length."<br/>";

        array_push($streams, $page);

    }

    var_dump($streams);
    $sd = fopen('streams-dump.json', 'w');
    fwrite($sd, json_encode($streams));
    fclose($sd);

}

$qld= fopen('quicklist-dump.json', 'w');
fwrite($qld, json_encode($quicklist));
fclose($qld);

buildPages();

?>

Answer 1

At a glance I'd be almost certain that the network requests are your bottleneck here.

If you think of the difference between some code interpreted and run locally vs instantiating a network connection, sending the request, waiting for their servers to run their code and the network latency to respond I think it's safe to say the network time is going to be the bottleneck. Particularly when you consider you're doing hundreds of network connections.

To speed this up I'd say Memcached is your best option. APC is also pretty useful, but not specifically for this. The general idea would be to check if you have cached results and if not generate the data you need and stick it in memcached and return the data. Then, in separate code, take that data and format it as required.

Roughly speaking:

if (Cache->check('key-to-your-data'))
{
    $twitchTvData = Cache->get('key-to-your-data');
}
else
{
    $twitchTvScraper = new TwitchTvScraper();
    $twitchTvData = $twitchTvScraper->scrape();
    Cache->set('key-to-your-data', $twitchTvData);
}

echo PageBuilder->buildFoo($twitchTvData);

In this example I've split out the caching, scraping of content and building of pages into separate classes. Having these all in the one place is a recipe for disaster/embarrassingly silly bugs.

Also, it's worth noting that hitting someone's API repeatedly like this is usually considered bad form and if they're monitoring for such activity you could get blocked. It might be worth looking into a package that can scrape pages with a rate limit. I'm sure there are a few out there. If you don't need the data refreshed often you could just use wget and read the files from a directory locally.

Finally; it's fine to stick some html together when you're learning a new language but in practice it isn't the way to go. Typically you use an off the shelf framework that can build HTML from templates. Something like, Zend, Laravel etc. (http://www.phpframeworks.com/ for more).

Having said all that, if you're just learning to code, don't take this all on at once! Take it piece by piece and save early, save often.

Answer 2

I got in touch with a friend who helped me smash out some of the bad parts (read: all of it). Here is the modified function which is much cleaner and more logical. Works like a charm:

function buildPages() {

    $streams = array();
    $offset = 0;
    $url = 'https://api.twitch.tv/kraken/streams?limit=100&offset=';

    do {
        $page = file_get_contents($url . $offset);

        if(empty($page))
            throw new Exception('Error loading URL: ' . $url . $offset);

        $pageData = json_decode($page, true);
        $numStreams = count($pageData['streams']);
        $offset += $numStreams;
        $streams[] = $pageData;

    } while($numStreams >= 100);

    var_dump($streams);

    $sd = fopen('streams-dump.json', 'w');
    fwrite($sd, json_encode($streams));
    fclose($sd);

}

Now I have to think of a way to compare the contents of the current page and previous page to eliminate any duplicate entries being caused by the behavior of the API delivery regarding offset shifting as I crawl the pages.

Then I'm ready for memcache and live testing, I think. :D