1
\$\begingroup\$

I have a fairly simple app set up with couchrest_model to capture data pulled in from vendors to couchdb for later searching and archiving. Each document has a show (the name of the vendor), a datatype (which is limited to seven or eight unique strings), and a pulled_at timestamp field.

The core method on this model needs to return a Hash where each key is a show and the value is an array of the most recent document for each datatype. Below, I have my (simplified) model code which defines this method. It works, but my code in self.most_recent_by_type_and_show() which loops over the list of _ids and calls get() on each one (thus generating n HTTP requests on each method call) will not scale.

class Snapshot < CouchRest::Model::Base
  property :show, String
  property :sources, [String]
  property :pulled_at, DateTime
  property :datatype, String
  property :data, [Hash]

  design do
    view :_most_recent_by_type_and_show, :map => <<-MAP, :reduce => <<-REDUCE
      function(doc) {
        if (doc.type == "Snapshot") {
          emit(doc.show, [doc.datatype, new Date(doc.pulled_at).valueOf()]);
        }
      }
    MAP
      function(keys, values, rereduce) {
        var maxes = {},
            out = [];

        if (!rereduce) {
          for (var i in keys) {
            var show = keys[i][0];
            var doc_id = keys[i][1];
            var datatype = values[i][0];
            var timestamp = values[i][1];

            if (typeof maxes[show] == 'undefined') { maxes[show] = {}; }
            if (typeof maxes[show][datatype] == 'undefined') { maxes[show][datatype] = [0, null]; }

            if (timestamp > maxes[show][datatype][0]) {
              maxes[show][datatype] = [timestamp, doc_id];
            }
          }
        }

        if (Object.keys(maxes).length == 0) { maxes = values; }
        for (var show in maxes) {
          for (var datatype in maxes[show]) {
            out.push(maxes[show][datatype][1]);
          }
        }

        return out;
      }
    REDUCE
  end

  class << self
    def most_recent_by_type_and_show opts = {}
      options = opts.merge({ reduce: true,  })
      raw = _most_recent_by_type_and_show(options).group.rows

      raw.reduce({}) do |hash, row|
        hash[row['key']] = row['value'].map { |id| get id }.sort_by(&:pulled_at).reverse
        hash
      end
    end
  end
end

This feels like I'm overcomplicating the view code, and I'm also not sure that my reduce function is written correctly for rereduce. I initially tried emitting [show, datatype, pulled_at] as the key and the entire doc itself as the value in the map function, but I've read that this is Bad Practice as it will cause the size of my view index to balloon over time. And for some reason, couchrest_model will not allow me to include_docs=true if I'm reducing a view, which seems to be just a limitation of the gem as I can't find anything saying this should be the case in the CouchDB docs.

Would this be better written as a different data structure altogether? Should I maybe be using a _list function? I'm not sure where to go from here.

\$\endgroup\$
2
\$\begingroup\$

You can use CouchRest::Database.get_bulk API:

hash[row['key']] = database.get_bulk(row['value']).sort_by(&:pulled_at).reverse
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.