Scraping links from provided URLs

Question

This script takes the list of provided URLs and scrapes the present links in each URL. For each scraped link, Facebook share, tweet, Google Plus is found out.

For example, if the provided URL is www.example.com, the URL is scraped and, let us say, 2 links are found:

www.example.com/1.php
www.example.com/2.php

Then Facebook likes and others are found for both the links and average (i.e. total Facebook likes / 2 (two links scraped)) is calculated. The problem is that the script takes too much time. Is there any method to optimize it?

<?php
/* Enable Error Reporting in php */
ini_set('display_errors',1);
ini_set('display_startup_errors',1);
error_reporting(-1);

function get_tweets($url) {

    $url =trim(rtrim($url,"\n"));
    $json_string = file_get_contents('http://urls.api.twitter.com/1/urls/count.json?url='.$url);
    $json = json_decode($json_string, true);

    return @intval( $json['count'] );
}

function get_likes($url) {

    $url =trim(rtrim($url,"\n"));
    $json_string = file_get_contents('http://graph.facebook.com/?ids='.$url);
    $json = json_decode($json_string, true);

   return @intval( $json[$url]['shares'] );
}

function get_plusones($url) {

    $url =trim(rtrim($url,"\n"));
    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL, "https://clients6.google.com/rpc");
    curl_setopt($curl, CURLOPT_POST, 1);
    curl_setopt($curl, CURLOPT_POSTFIELDS, '[{"method":"pos.plusones.get","id":"p","params":{"nolog":true,"id":"' . $url . '","source":"widget","userId":"@viewer","groupId":"@self"},"jsonrpc":"2.0","key":"p","apiVersion":"v1"}]');
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl, CURLOPT_HTTPHEADER, array('Content-type: application/json'));
    $curl_results = curl_exec ($curl);
    curl_close ($curl);

    $json = json_decode($curl_results, true);

    return @intval( $json[0]['result']['metadata']['globalCounts']['count'] );
}

function scrapper($Url){
    $resultant = array();
    $details="URL,Likes,Tweets,Plus ones\n";

    $fb_sum = 0;
    $tweets = 0;
    $g_plus = 0;


    $userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)'; //Change it to some other

    $master_curl = curl_multi_init();
    $curl_arr = array();

    for($i=0;$i<count($Url);$i++)
    {
        $url = $Url[$i];
        $curl_arr[$i] = curl_init($url);
        curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curl_arr[$i], CURLOPT_USERAGENT, $userAgent);
        curl_setopt($curl_arr[$i], CURLOPT_FAILONERROR, true);
        curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER,true);
        curl_multi_add_handle($master_curl, $curl_arr[$i]); 
    }

    $running = 0;
    do {
        curl_multi_exec($master_curl, $running);
    } while ($running > 0);

    for($j=0; $j<count($Url); $j++)
    {
        $html = curl_multi_getcontent($curl_arr[$j]); 
        $dom = new DOMDocument();
        @$dom->loadHTML($html);

        //Grab the entire Page 
        $xpath = new DOMXPath($dom);
        $hrefs = $xpath->evaluate("/html/body//a");

        for ($i = 0; $i < $hrefs->length; $i++) {

            $href = $hrefs->item($i);
            $url2 = $href->getAttribute('href');

            // Url's are obtained 
            //Obtain Tweets, fb likes and google plus ones of the url 

            $fb_sum += get_likes($url2);
            $tweets += get_tweets($url2);
            $g_plus += get_plusones($url2);

        }
        array_push($resultant,array($Url[$j],$fb_sum,$tweets,$g_plus));

    }

    return $resultant;
}



if(isset($_POST['submit']) && (trim($_POST['urls'])!=""))
{

    $getUrls = explode("\n",rtrim($_POST['urls'],"\n"));

    $time_start = microtime(true); 

    $final_display = scrapper($getUrls);

    print_r($final_display);

    $time_end = microtime(true);

    $execution_time = ($time_end - $time_start)/60;

    echo '<br><b>Total Execution Time:</b> '.$execution_time.' Mins';

}
else
{

    echo "<center><h3>Now Go <a href='index.php'>Back</a> and Type something. </h3></center>";

}


?>

Answer 1

I can't comment on performance, as I'm not very familiar with the cURL library and web scraping techniques.

However, I'd be happy to comment on other points! I'll point things out in the order the script is ran.

• && (trim($_POST['urls'])!="")) is redundant, consider just using && !empty($_POST['urls'])).

• Thorughout the code, I see rtrim($_POST['urls'],"\n") a lot. Factor this into a single function:

  function remove_last_newline($input) {
      return rtrim($input, "\n");
  }
  

• scrapper is a poor function name. It has no meaning and confuses the reader. Also, the parameter $Url should be array $urls. We type hint an array, and make it plural.

• Avoid variables with numbers concatenated: $url2.
• I notice you check every single link on the page for likes and such. This is my guess as to why it's so slow. Only query the links you think would have likes.
• array_push is actually slower than $resultant[] = array($Url[$j],$fb_sum,$tweets,$g_plus);. If you can, use the brackets for single pushes.
• $resultant isn't a word... Just use $results.

• This part is repeated a lot, consider using a function and passing the URL as a parameter:

  $url =trim(rtrim($url,"\n"));
  $json_string = file_get_contents('http://urls.api.twitter.com/1/urls/count.json?url='.$url);
  $json = json_decode($json_string, true);
  

• Why are you suppressing @intval?

• You spacing is very inconsistent, give unity to the spaces around operators and assignments.

My advice to making this faster:

• use as little code as possible. The more clutter, the less likely it is to be fast (unless it's for a good reason).
• Use any APIs you can. Much faster than scraping.
• Use microtime to figure out where