Code Review Stack Exchange is a question and answer site for peer programmer code reviews. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My script, which is a web scraping script, is very slow, I even needed to put set_time_limit(0);

This is the whole

I think the problem is here:

foreach ($array_with_links as $url_job) {

$info=Array(getTitle($url_job, getID($url_job)), getTitle_Short($url_job, getID($url_job)), getCity($url_job), 
 getDepartment($url_job), getSalary($url_job), getJobNumber($url_job), getPositionStartDate($url_job), getFullTimeEquivalent($url_job),
  getPermTermCasual($url_job), getLocation($url_job), getQualifications($url_job), getDuties($url_job), 
  getClosingDate($url_job), getContact($url_job), getEmail($url_job), getCreated_On($url_job), getID($url_job) );

array_push($data, $info);}

Example of some of those functions:

function getCity($url)
    $url = curl_get_contents($url);
    $html_object = str_get_html($url);
    return $ret = $html_object->find('td', 86)->plaintext;

function getDepartment($url)
    $url = curl_get_contents($url);
    $html_object = str_get_html($url);
    return $ret = $html_object->find('td', 90)->plaintext;

And this is my cURL funtion:

function curl_get_contents($url)
  $curl_moteur = curl_init();
  curl_setopt($curl_moteur, CURLOPT_URL, $url);
  curl_setopt($curl_moteur, CURLOPT_RETURNTRANSFER, 1);

  curl_setopt($curl_moteur,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20080311 Firefox/');

  curl_setopt($curl_moteur, CURLOPT_FOLLOWLOCATION, 1);
  $web = curl_exec($curl_moteur);
  return $web;

Those getX come from an URL one by one. Maybe is there any method to make multiple simultaneous insertions in that array?

I really don't know what to do and what's my mistake.

share|improve this question
up vote 11 down vote accepted

It seems that every one of your getCity(), getDepartment() etc functions loads the same web page over and over.

You should load each URL once with your curl_get_contents(), then pass its result into each get*() function to parse it.

share|improve this answer
I believe you just hit the bullseye! Spot on! Nailed it! – Simon Forsberg Jan 31 '14 at 12:23
str_get_html is also called repeatedly. – ChrisW Jan 31 '14 at 12:24
that's it!!! so, how can I implement a cache system in order to extract all the information fields requesting each url just one time – user204415 Jan 31 '14 at 12:32
@user204415 Invoke curl_get_contents and str_get_html only once at the start of your foreach loop, and pass $html_object instead of $url as the parameter to each of your getXXX functions. – ChrisW Jan 31 '14 at 14:17
@user204415 you are right sir. – user204415 Jan 31 '14 at 15:41

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.