# Web scraper for job listings

Is there any room for improvement on this code?

I use mechanize to get the links of a job listing web site. There are pages with pagination (when jobs > 25) and pages without.

If there is, then the link number 5 is labelled after "Next >" and link number 6 is "Last >>". In order to identify them, I make assertions on their name and act accordingly. I then check how many pages the pagination occurs, and then, taking the numbers, I scrape the pages one by one.

  def scrape_duty_station
agent = Mechanize.new

scrape_single_page
(pagestart..pageend).each do |i|
scrape_single_page
end
else
scrape_single_page
end
end

def scrape_single_page
@page.each do |i|
p=Position.new
p.title = i.text
p.job_id = i.href.match(/\d+/).to_s
p.save
end
end


Here are some comments. - If you are only @page as a way to communicate with scarpe_single_page, why not make it a parameter?

def scrape_duty_station
agent = Mechanize.new


Avoid extra indent when one of the cases is simpler and can be dealt with easily.

    return scrape_single_page(page.links[5...30]) unless page.links[5].text =~ /^next/i
end


If you are using any statement more than once, it may be worthwhile to facter it out to a function.

  def l_i(page, link)

  def scrape_single_page(pages)