0
\$\begingroup\$

I have a table in my database with 35000 unique URLs. I use them to create my XML sitemaps for the site. I have setup crontab jobs to create automatically new URLs, delete non existing URLs and determine if a URL is valid.

For the last part I use the following function:

function get_url_status($url)
{
    $httpCode = 0;
    $headers = get_headers($url);
    $http_header_info = $headers[0];
    $httpCode = substr($http_header_info, 9, 3);

    if ($httpCode=='200')
    {
        // check in page for noindex and dissable that url
        $metas = get_meta_tags($url);
        if ( isset($metas['robots']) && strpos( strtolower($metas['robots']),  'noindex')!==false  )
        {
            $httpCode = 410;
        }
    }

return $httpCode;
}

This will return to me the HTTP status code - 200 for legal URLs that will be included in sitemap, 400 for not found, 410 for URLs with noindex in them, 301 for redirections, etc.

In my XML sitemaps I include only URLs with status code 200.

My problem is that this function takes around 40 minutes to check the URLs in database. Is there a way to speed things up?

\$\endgroup\$
  • \$\begingroup\$ What part takes 40 minutes? If you're fetching all the records from the database I wouldn't be surprised about that runtime (you do have an index for the URL I hope). \$\endgroup\$ – ferada Nov 14 '14 at 14:50
  • \$\begingroup\$ @ferada - yes db table has indexes, this function is used in a loop where each url from db is checkeck. So, what other way is there? \$\endgroup\$ – andrew Nov 14 '14 at 14:54
1
\$\begingroup\$

So you're fetching 70k pages. You could half that if you only fetch the page once and then do what get_headers and get_meta_tags do manually. That will presumably take now 20 minutes.

Then you can try doing this in parallel, so run two instances of the script with one half of the URLs each.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.