Skipping Google search results that point to certain sites

Question

I have the following code that will skip certain URLs if needed:

SEARCH = "inurl:index.php?id="

def get_urls
  puts "Searching for possible SQL vulnerable sites, using search query #{SEARCH}"
  agent = Mechanize.new
  agent.user_agent = @user_agent
  page = agent.get('http://www.google.com/')
  google_form = page.form('f')
  google_form.q = "#{SEARCH}"
  url = agent.submit(google_form, google_form.buttons.first)
      url.links.each do |link|
        if link.href.to_s =~ /url.q/
          str = link.href.to_s
          str_list = str.split(%r{=|&})
          urls = str_list[1]
          next if str_list[1].split('/')[2] == 'webcache.googleusercontent.com'
          next if str_list[1].split('/')[2] == 'search.clearch.org'
          next if str_list[1].split('/')[2] == 'duckfm.net'
          next if str_list[1].split('/')[2] == 'search1.speedbit.com'
          next if str_list[1].split('/')[2] == 'yoursearch.me'
          next if str_list[1].split('/')[2] == 'search.speedbit.com'
          next if str_list[1].split('/')[1] == 'ads/preferences?hl=en'
          next if str_list[1].split('/')[2] == 'www.sa-k.net'
          next if str_list[1].split('/')[2] == 'github.com'
          next if str_list[1].split('/')[2] == 'stackoverflow.com'
          urls_to_log = URI.decode(urls)
          puts "Site found: #{urls_to_log}"
          sleep(1)
          File.open("#{PATH}/SQL_sites_to_check.txt", 'a+') { |s| s.puts("#{urls_to_log}'") }
        end
      end
  puts "Possible vulnerable sites dumped into #{PATH}/SQL_sites_to_check.txt"
end

Is there a better more readable, less smelly, way to write the next if's?

Answer 1

You seem to be referring to the same information str_list[1].split('/')[2] several times. I highly suggest you simply do this one type and assign it to a variable so that your list can go

next if url == "github.com"
next if url == "google.com"

Not only is this clearer, but it should save time because splits are probably not the thing you wanna do 16 times in a loop.

Also right at the top you have urls = str_list[1] but then you don't use that variable??