# Detecting headers status takes too long

I have a table in my database with 35000 unique URLs. I use them to create my XML sitemaps for the site. I have setup crontab jobs to create automatically new URLs, delete non existing URLs and determine if a URL is valid.

For the last part I use the following function:

function get_url_status($url) {$httpCode = 0;
$headers = get_headers($url);
$http_header_info =$headers[0];
$httpCode = substr($http_header_info, 9, 3);

if ($httpCode=='200') { // check in page for noindex and dissable that url$metas = get_meta_tags($url); if ( isset($metas['robots']) && strpos( strtolower($metas['robots']), 'noindex')!==false ) {$httpCode = 410;
}
}

return \$httpCode;
}


This will return to me the HTTP status code - 200 for legal URLs that will be included in sitemap, 400 for not found, 410 for URLs with noindex in them, 301 for redirections, etc.

In my XML sitemaps I include only URLs with status code 200.

My problem is that this function takes around 40 minutes to check the URLs in database. Is there a way to speed things up?

• What part takes 40 minutes? If you're fetching all the records from the database I wouldn't be surprised about that runtime (you do have an index for the URL I hope). – ferada Nov 14 '14 at 14:50
• @ferada - yes db table has indexes, this function is used in a loop where each url from db is checkeck. So, what other way is there? – andrew Nov 14 '14 at 14:54

So you're fetching 70k pages. You could half that if you only fetch the page once and then do what get_headers and get_meta_tags do manually. That will presumably take now 20 minutes.