Generating sample data for a Rails application

Question

I am working on generating sample data for my Rails application. For now we will just be using the sample data while we are designing the website. I'd like it to be as easy as possible to generate sample data each time we introduce a new model.

In our tests, we use FactoryGirl. The factories we have for testing are "bare-bones" factories - they include only what is required for a record to be valid. In general, the data that would be created as users interact with the site is much richer (more attributes and associations would exist).

Approaches considered but not taken

I could approach this using FactoryGirl features like traits and nested factories, and just add to the test factories. One thing I don't like about this approach is that it would clutter my test factories with stuff that is really only intended to be used in generating sample data.

Alternatively, I could have a separate set of factories that are only for generating sample data. To avoid duplication with what is already defined for the tests, I could use factory modification, but you are a little limited on what you can modify. And this isn't really true to what I want to do - I really want to create children of the base-case factories.

I could open up the test factories and add children via modification in a separate set of files that are only loaded when the sample data rake task is running.

What we're currently doing

The approach I've taken so far is to create "generator" classes that typically shadow/observe a factory (that is a factory defined in our test factories set). Each generator inherits from BaseGenerator:

class BaseGenerator
  attr_accessor :min, :max

  def initialize(args = {})
    @min = args[:min]
    @max = args[:max]
  end

  def generate_one
    FactoryGirl.create(factory_name, *options)
  end

  def generate_set
    Faker::Number.between(min, max).times do
      generate_one
    end
  end

  def factory_name
    define_factory
  end

  def options
  end

end

min and max refer to how many records should be created - if two different numbers are provided, then a random number is selected between them. If the same number is given for both, then that is how many records are created. Children of BaseGenerator should typically override the initialize method and provide default values for min and max. Child classes must implement the define_factory method; it can simply return the name of a factory that is already defined and should be used, or it can be more complicated (I'll explain after the following example). The options method can be used to add traits to a factory and to override the factory's attributes (for a while I had forgotten that I created this feature, and there is a good chance I'm missing cases where I should be using it).

Here is an example of a class that inherits from BaseGenerator:

class UserGenerator < BaseGenerator
  attr_accessor :include_name

  def initialize(args = {})
    @min = args[:min] || 10
    @max = args[:max] || 15
    @include_name = args[:include_name]
  end

  private

  def define_factory
    :user
  end

  def options
    # with_name is a trait that has already been defined in the test factories
    [(:with_name if include_name?)]
  end

  def include_name?
    case @include_name
    when :never
      false
    when :always
      true
    else
      [true, false].sample
    end
  end
end

UserGenerator is an example of one of the simple cases, where define_factory just returns the symbol for the existing factory to be used, and also uses the options method.

LocationGenerator is an example of a more complicated one:

class LocationGenerator < BaseGenerator
  attr_accessor :rand_lat_long
  alias_method :rand_lat_long?, :rand_lat_long

  def initialize(args = {})
    @rand_lat_long = true unless args[:rand_lat_long] == false
  end

  private
  def define_factory
    if parent_factory
      include_address = include_address?
      FactoryManager.define_child_of parent_factory do
        # with_address is a trait that has already been defined in the test factories
        with_address if include_address
      end
    else
      :generic_location
    end
  end
  def parent_factory
    :location unless rand_lat_long?
  end
  def include_address?
    # a method that I wrote that returns true 4/5 of the time
    four_fifths_of_the_time
  end
end

A generated location can either have a randomly generated address (via the :location factory), and then use the default geocoding stub (which returns a fixed latitude and longitude), or it can have a fixed address (via the :generic_location factory), and then use the geocoding stub for that specific address (which returns a random latitude and longitude). This is probably one of those cases where I should be using the options method and I'm not...

The FactoryManager is a small module who's purpose is to define child factories (of existing factories, e.g. test factories). It looks like this:

class FactoryManager
  def self.define_child_of(parent_factory, &block)
    unless FactoryGirl.factories.registered? parent_factory
      raise ArgumentError, "parent factory '#{parent_factory}' is not defined... are you sure you passed in a symbol?"
    end

    locals_fingerprint = block.binding.local_variables_with_values.hash

    child_factory_name = "#{parent_factory}_#{locals_fingerprint}"
    unless FactoryGirl.factories.registered? child_factory_name
      FactoryGirl.define do
        factory(child_factory_name, parent: parent_factory, &block)
      end
    end

    return child_factory_name
  end
end

class Binding
  def local_variables_with_values
    local_variables.map do |name|
      [name, local_variable_get(name)]
    end
  end
end

So it looks at the local variable defined when the block was created, and uses them to create a hash to identify the child factory. It uses this hash to check if the child has been defined, and then defines it if it needs to.

So far, this might seem like overkill (and maybe it is - that is one of my questions/concerns), but we also have things like this:

class ClimbGenerator < BaseGenerator

  def initialize(args = {})
    @gym = args[:gym]
  end

  private
  def define_factory
    include_grade = include_grade?
    include_moves_count = include_moves_count?
    include_name = include_name?
    _gym_section = gym_section

    FactoryManager.define_child_of [:boulder, :route].sample do
      grade { type.constantize.grades.keys.sample if include_grade }
      moves_count { Faker::Number.between(1, 30) if include_moves_count }
      name { Faker::Hipster.words(Faker::Number.between(1,5)).join(' ').titlecase if include_name }
      before(:create) do |climb|
        climb.gym_section = _gym_section
      end
    end
  end

  def include_grade?
    four_fifths_of_the_time
  end

  def include_moves_count?
    four_fifths_of_the_time
  end

  def include_name?
    a_fifth_of_the_time
  end

  def gym_section
    @gym.try(:sections).try(:sample)
  end
end

And finally, here is an example of where I use all of this:

namespace :db do
  desc "Fill database with sample data"
  task :populate => :non_prod do
    require "#{Rails.root}/lib/data_generators"

    Rake::Task['db:reset'].invoke

    # Create some "new" users (users that don't have a role yet).
    UserGenerator.new.generate_set

    GymGenerator.new.generate_set

    Gym.all.each { |gym| generate_members_for gym }

  end

  def generate_members_for(gym)
    climb_generator = ClimbGenerator.new(gym: gym)
    alog_generator = AthleteClimbLogGenerator.new(climb_generator: climb_generator)
    athlete_generator = AthleteGenerator.new(min: 1, max: 15, alog_generator: alog_generator)

    athlete_generator.generate_set
  end
end

Questions / things that don't feel right

I'd appreciate feedback on anything that catches your eye, but these are the questions I have in mind:

• It wouldn't really make sense for the class BaseGenerator to be instantiated on its own... does this make it weird that it is a class and not a module? It seems kind of like ActiveRecord::Base - I've never seen that class instantiated on its own yet it is still a class, it's mainly for inheritance (or maybe it does get instantiated on its own and I just haven't seen it yet). If it is fine leaving it as a class, should I do something to make it obvious that it can't/shouldn't be instantiated? E.g. I could implement define_factory in BaseGenerator and have it raise an exception (I just thought of this while writing this question).
• Having to define the local variables like that before calling FactorManager.define_child_of feels kind of messy and in some ways it feels wrong. But then it provides a really good way of knowing when a new factory needs to be defined. I don't have a particular question here, but this is one of the main things I want feedback on / alternative ideas for.
• Sometimes I think that this all overcomplicated and I would be better off using just plain FactoryGirl. Other times I feel certain that this way makes it easier to control things (e.g. the creation of associated records).

Answer 1

I've reworked this and it is a lot better (simpler and easier to use) to just use plain FactoryGirl :) To avoid bloating my factories that I have for testing, I've just created a separate folder that has "data generation" factories. For example, this is a factory for testing:

# spec/factories/climbs.rb
FactoryGirl.define do
  factory :climb do
    type { ['Boulder', 'Route'].sample }

    transient do
      gym_factory :gym
      gym do
        if loggable_type == 'AthleteClimbLog'
          athlete_climb_log.athlete_story.athlete_climb_logs.sample.try(:climb).try(:gym) or
          Gym.random or
          create(gym_factory)
        else
          Gym.random or create(gym_factory)
        end
      end
    end

    after :build do |climb, evaluator|
      climb.gym_section = evaluator.gym.sections.sample unless climb.gym_section
    end

    factory :boulder do
      type  'Boulder'
    end
    factory :route do
      type  'Route'
    end
  end

end

and this is the corresponding data generation factory:

# lib/factories/climbs.rb
FactoryGirl.define do
  factory :_climb_, parent: :climb do
    grade do
      four_fifths_of_the_time { type.constantize.grades.keys.sample }
    end
    moves_count do
      four_fifths_of_the_time { Faker::Number.between(1, 30) }
    end
    name do
      a_fifth_of_the_time { Faker::Hipster.words(Faker::Number.between(1,5)).join(' ').titlecase }
    end

    transient do
      gym_factory :_gym_
    end
  end
end

I use the data generation factories in a rake task like this:

# lib/tasks/db_populate.rake
namespace :db do
  desc "Fill database with sample data"
  task :populate => :non_prod do
    require "#{Rails.root}/lib/factories/factories"

    Rake::Task['db:reset'].invoke

    FactoryGirl.create_list(:_user_, Faker::Number.between(10, 15))

    FactoryGirl.create_list(:_gym_, Faker::Number.between(5, 10))

    FactoryGirl.create_list(:_setter_, Faker::Number.between(15, 20))

    FactoryGirl.create_list(:_athlete_, Faker::Number.between(35, 40))
  end
end

(the particular factory _climb_ is only used in the definition of the other factories, it isn't used directly in the rake task above).