Skip to main content
1 of 2
Anthony
  • 439
  • 3
  • 13

Deduplicate data

I currently have an implementation that helps us deduplicate data but I know it's slow.

Here's an example, note that I'm using OpenStruct here when really these are Mongoid objects pulled from the database.

require 'ostruct'

vendor_data = [
  OpenStruct.new(activity_id: "abc123"),
  OpenStruct.new(activity_id: "def456"),
  OpenStruct.new(activity_id: "ghi789"),
  OpenStruct.new(activity_id: "jkl012"),
]

existing_ids = ["abc123", "def456"]

def slow_method(vendor_data, existing_ids)
  vendor_data.each do |vendor|
    next if existing_ids.include? vendor.activity_id
    vendor.name = "A company"
    # in real life we'll call vendor.save or update here
  end
end

slow_method(vendor_data, existing_ids)

I know this is \$O(n^2)\$ but I'm struggling to figure out a faster way to do this.

I'm not sure on CodeReview if it's best practice to list out the 4 other variations I've tried but none of them seem to be any better or readable than my first pass.

Anthony
  • 439
  • 3
  • 13