1
\$\begingroup\$

I have +-645 IRL links I need to validate. The code below is my attempt, but my problem is that it takes forever to check all links and sometimes times out. Is there a better way of validating so many links?

index.php

<?php 
error_reporting(E_ALL);
ini_set('display_errors', 1);

$notValid = [];

// This array contains +-645 url in this array, http://www.example.com is just for example purposes.
$companyData = ['http://www.example.com','http://www.example.com', 'http://www.example.com', 'http://www.example.com'];
// Basic structure of $companyData array

Array
(
    [0] => Array
        (
            [Name] => Name-Ltd
            [Code] => XYZ
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd
        )

    [1] => Array
        (
            [[Name] => Name-Ltd
            [Code] => ZXY
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd
        )

    [2] => Array
        (
            [Name] => Name-Ltd
            [Code] => ZYX
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd
        )

    [3] => Array
        (
            [Name] => Name-Ltd
            [Code] => YZX
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd
        )

    [4] => Array
        (
            [Name] => Name-Ltd
            [Code] => XZY
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd

)

// This is to loop through all url's
for ($i=0; $i < count($companyData); $i++) 
{ 
    $url = $companyData[$i]['Url'];

    $headers = get_headers($url, 1);

    if(@$headers['Content-Length'] == 1153) 
    {
        $exists = 'not valid';

        $notValid[$i] = $companyData[$i]['Url'];
    }
    else 
    {
        $exists = 'valid';
    }
}

foreach ($notValid as $not) 
{
    echo "The following URL's need your attention: <br>";
    echo $not;
}

?>
\$\endgroup\$
  • \$\begingroup\$ you make a request for each url every-time you want to verify? no wonder it is slow, cache the result \$\endgroup\$ – ratchet freak Feb 11 '15 at 15:00
  • \$\begingroup\$ @ratchetfreak hi, do you know of any website or link i can go lookup on caching \$\endgroup\$ – Wayne Links Feb 11 '15 at 15:01
  • \$\begingroup\$ @WayneLinks caching is no magic. Just save the output from get_headers into a file or database so you dont have to do a request the second time you validate this url. If this makes sense depends on how often you validate the same url \$\endgroup\$ – braunbaer Feb 11 '15 at 15:15
  • \$\begingroup\$ @WayneLinks: How up-to-date do you need this information to be? Wouldn't it be better for you to have a cron running once or twice a day, that checks these urls? Store/cache that data somewhere, notify the admin (you, sys admin, whoever is supposed to be notified) by sending a mail, a push message, an AMQP message, smoke signals, whatever... PS: @$headers -> just use isset, and not the @ operator of death \$\endgroup\$ – Elias Van Ootegem Feb 11 '15 at 15:19
  • \$\begingroup\$ @braunbaer I hear what you say, it does seem as if the initial run to check the url's the first time might take long, after that with the practice of caching then it will go faster. My main problem is the initial attemp that takes long, is there not a way i can set time interval so not to overwhelm the sever. \$\endgroup\$ – Wayne Links Feb 11 '15 at 15:21
1
\$\begingroup\$

Your script is slow because you are doing a sequence of HTTP REQUESTS. Most of the time your server / script is just waiting for the RESPONSE of the requested URL.

small things you can do

  • as mentioned yourself in the comments you can change your max execution time to prevent reaching it. (which does not make your script any faster ; -)
  • move away from get headers and use curl or similar to have the ability to change the connection timeout. You could set it to 1 sec and flag every url which fails to manual review (assuming all healthy urls will respond in time , and some slow false positives). Small benefit: your script should be finished in max url*sec time

big things

so the problem is: in practice is your script waiting for responses, in theory it does not have to , as the checking of an url does not depend on each other. Classic case for parallelization. Downside: php is not the language you want to do parallelization in.

upside: you are not the first with this problem

or use golang or something with build in parallelization

\$\endgroup\$
  • \$\begingroup\$ Thanks for the links, definitely made me think about my script. This will definitely help. \$\endgroup\$ – Wayne Links Feb 11 '15 at 16:19
  • \$\begingroup\$ Though it's hardly easier than multiple curl requests, OP could use non-blocking socket streams. If only because it allows for a bit more debugging options in case an url goes down, and because getting to grips with sockets is helpful if you're looking into other (more low-level) languages \$\endgroup\$ – Elias Van Ootegem Feb 11 '15 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.