I've built a website that scrapes a guitar forum's pages and populates Rails model. I'm using rake task along with heroku scheduler to run background scrapes every hour. On the homepage, the forum ads (in the form of a Post Model) are displayed using a dataTables (jQuery plugin), so user can search from the front-end.


I had a few questions to see if there's a better way to do things. If the questions require too much effort, then any general feedback or pointer to online resources would be appreciated.

1. Best Caching Strategy?

I'm currently implementing fragment caching, but I'm not sure if this is best practice.

First I cache all posts:

<% cache(cache_key_for_posts) do %>
<% @posts.each do |p| %>
    <td class="replies"><%= p.replies %></td>
    <td class="views"><%= p.views %></td>
<% end %>
<% end %>

...using a helper method to set cache key:

def cache_key_for_posts
  last_updated_at = Post.last.updated_at.to_i

My understanding is that @posts = Post.all is gets cached. And when I re-scrape, memcache is sets new key-value pair, which is then being used to read current data.

The caching seems to be working at the moment, but I'm curious if there's a better way to implement caching.

2. Scraping / Exception Handling

I put all the scrape logic inside the Post Model:

  class Post < ActiveRecord::Base
  default_scope { order(:id) }

  URL = "http://www.acousticguitarforum.com/forums/forumdisplay.php"\

  # scrape guitar forum pages and create records in posts table
  def self.scrape_create
    require 'open-uri' 
      data = Nokogiri::HTML(open(URL))
      numPages = data.css('td.vbmenu_control').text.split(' ').last.to_i  
    rescue StandardError=>e
      puts "Error: #{e}"
      puts "Test scrape failed."
      destroy_all             # delete old records
      reset_pk_sequence       # reset primary key to 0

And then run the scrape using rake task:

namespace :scrape do
  desc "Scrape AGF For Sale forum"
  task agf: :environment do
      puts "scraping AGF pages..."
      # Post.delay.scrape_create
      # no moneny for heroku workers :(
      puts "scraping complete."

Again, I'm not sure if this is best practice, or if I'm missing anything. Would appreciate a second pair of eyes to skim over and let me know if something looks off.


I received a-okay from forum owner.

Source Code


Browse other questions tagged or ask your own question.