I've built a website that scrapes a guitar forum's pages and populates Rails model. I'm using rake task along with heroku scheduler to run background scrapes every hour. On the homepage, the forum ads (in the form of a Post Model) are displayed using a dataTables (jQuery plugin), so user can search from the front-end.
Questions
I had a few questions to see if there's a better way to do things. If the questions require too much effort, then any general feedback or pointer to online resources would be appreciated.
1. Best Caching Strategy?
I'm currently implementing fragment caching, but I'm not sure if this is best practice.
First I cache all posts:
//app/views/posts/index.html
<% cache(cache_key_for_posts) do %>
<% @posts.each do |p| %>
....
<td class="replies"><%= p.replies %></td>
<td class="views"><%= p.views %></td>
</tr>
<% end %>
<% end %>
...using a helper method to set cache key:
def cache_key_for_posts
last_updated_at = Post.last.updated_at.to_i
"/all-#{last_updated_at}"
end
My understanding is that @posts = Post.all
is gets cached. And when I re-scrape, memcache is sets new key-value pair, which is then being used to read current data.
The caching seems to be working at the moment, but I'm curious if there's a better way to implement caching.
2. Scraping / Exception Handling
I put all the scrape logic inside the Post Model:
//app/models/post.rb
class Post < ActiveRecord::Base
default_scope { order(:id) }
URL = "http://www.acousticguitarforum.com/forums/forumdisplay.php"\
"?f=17&pp=200&sort=lastpost&order=desc&daysprune=200"
# scrape guitar forum pages and create records in posts table
def self.scrape_create
require 'open-uri'
begin
data = Nokogiri::HTML(open(URL))
test_page_scrape(data)
numPages = data.css('td.vbmenu_control').text.split(' ').last.to_i
rescue StandardError=>e
puts "Error: #{e}"
puts "Test scrape failed."
else
destroy_all # delete old records
reset_pk_sequence # reset primary key to 0
scrape_pages(numPages)
end
end
And then run the scrape using rake task:
//app/lib/tasks/scrape.rake
namespace :scrape do
desc "Scrape AGF For Sale forum"
task agf: :environment do
puts "scraping AGF pages..."
# Post.delay.scrape_create
# no moneny for heroku workers :(
Post.scrape_create
puts "scraping complete."
end
end
Again, I'm not sure if this is best practice, or if I'm missing anything. Would appreciate a second pair of eyes to skim over and let me know if something looks off.
Notes
I received a-okay from forum owner.