Take the 2-minute tour ×
Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.

My script, which is a web scraping script, is very slow, I even needed to put set_time_limit(0);

This is the whole http://phpfiddle.org/main/code/9qt-78n

I think the problem is here:

foreach ($array_with_links as $url_job) {

$info=Array(getTitle($url_job, getID($url_job)), getTitle_Short($url_job, getID($url_job)), getCity($url_job), 
 getDepartment($url_job), getSalary($url_job), getJobNumber($url_job), getPositionStartDate($url_job), getFullTimeEquivalent($url_job),
  getPermTermCasual($url_job), getLocation($url_job), getQualifications($url_job), getDuties($url_job), 
  getClosingDate($url_job), getContact($url_job), getEmail($url_job), getCreated_On($url_job), getID($url_job) );

array_push($data, $info);}

Example of some of those functions:

function getCity($url)
    {
    $url = curl_get_contents($url);
    $html_object = str_get_html($url);
    return $ret = $html_object->find('td', 86)->plaintext;
    }

function getDepartment($url)
    {
    $url = curl_get_contents($url);
    $html_object = str_get_html($url);
    return $ret = $html_object->find('td', 90)->plaintext;
    }

And this is my cURL funtion:

function curl_get_contents($url)
{
  $curl_moteur = curl_init();
  curl_setopt($curl_moteur, CURLOPT_URL, $url);
  curl_setopt($curl_moteur, CURLOPT_RETURNTRANSFER, 1);

  curl_setopt($curl_moteur,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');

  curl_setopt($curl_moteur, CURLOPT_FOLLOWLOCATION, 1);
  $web = curl_exec($curl_moteur);
  curl_close($curl_moteur);
  return $web;
}

Those getX come from an URL one by one. Maybe is there any method to make multiple simultaneous insertions in that array?

I really don't know what to do and what's my mistake.

share|improve this question

1 Answer 1

up vote 10 down vote accepted

It seems that every one of your getCity(), getDepartment() etc functions loads the same web page over and over.

You should load each URL once with your curl_get_contents(), then pass its result into each get*() function to parse it.

share|improve this answer
    
I believe you just hit the bullseye! Spot on! Nailed it! –  Simon André Forsberg Jan 31 at 12:23
    
str_get_html is also called repeatedly. –  ChrisW Jan 31 at 12:24
    
that's it!!! so, how can I implement a cache system in order to extract all the information fields requesting each url just one time –  user204415 Jan 31 at 12:32
1  
@user204415 Invoke curl_get_contents and str_get_html only once at the start of your foreach loop, and pass $html_object instead of $url as the parameter to each of your getXXX functions. –  ChrisW Jan 31 at 14:17
    
@user204415 you are right sir. –  user204415 Jan 31 at 15:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.