I have to run a script against 40 million objects in my database: I wrote the script and I want 10 instance of it so I process my objects faster.
The goal is to find all the neighborhood for each property.
- A little bit of context: Multiple properties can have the same lat/lon.
- A Location is a city.
- I stored all the properties IDs i
need to process in a separate table because I have a lot more than 40
millions property and querying that table with
whereconditions if really slow even with indexes. - The reason why I use this
Object.const_set("neighborhoods_bp_ids".singularize.classify, Class.new(ActiveRecord::Base))is because I do a lot of processing of millions of ids and I don't want to create empty model all over. But if that's one big issue with the memory then I can make a proper AR class.
But somehow in the code I wrote there is a memory leak or something like this because running all the instances consume all the memory on my server.
property.rb
class Property < ActiveRecord::Base
extend GoogleApi
@proxies = Proxy.all.map(&:ip).shuffle
self.table_name = 'propertys'
self.primary_key = 'id'
self.inheritance_column = :ruby_type #needed to allow 'type' column in db table
has_and_belongs_to_many :neighborhoods, :limit=>1
attr_protected
def self.process_neighborhood(min_id, max_id)
range = min_id..max_id
klass = Object.const_set("neighborhoods_bp_ids".singularize.classify, Class.new(ActiveRecord::Base))
klass.where(id: range).find_in_batches do |group|
group.each do |bp|
add_neighborhood(bp.property_id, klass)
bp.delete
end
end
end
def self.add_neighborhood(property_id, klass)
property = Property.find_by_id(property_id)
if property.present? && property.neighborhoods.blank?
address = property.full_address.gsub(",", "").squish.present? ? Property.parse_google_geocode(property.full_address) : nil
if address.present? && address[:neighborhood].present?
neighborhood = Neighborhood.joins(:location).where("neighborhoods.name =? and locations.city = ? and locations.state = ?", address[:neighborhood], address[:city], address[:state]).limit(1)
if neighborhood.blank?
neighborhood = Neighborhood.find_by_name_and_location_id(address[:neighborhood], nil)
if neighborhood.blank?
location = Location.get_closest_known_city(address[:lat], address[:lng], address[:city], address[:state])
if location.present?
neighborhood = Neighborhood.create(name: address[:neighborhood], location_id: location.id)
else
neighborhood = Neighborhood.create(name: address[:neighborhood])
end
end
else
neighborhood = neighborhood.first if neighborhood.present?
end
begin
property.neighborhoods << neighborhood if property.neighborhoods.blank?
all_bps = Property.includes(:neighborhoods).where(latitude: property.latitude, longitude: property.longitude)
all_bps.each do |u|
u.neighborhoods << neighborhood if u.neighborhoods.blank?
end
klass.delete_all(property_id: all_bps.map(&:id))
rescue Exception => e
logger.debug "#{e.class}"
end
end
end
end
def full_address
"#{street}, #{city}, #{state} #{zipcode}"
end
end
google_api.rb
module GoogleApi
require 'open-uri'
PROXY_LOGIN = 'buildzoom'
PROXY_PASSWORD = 'jonxlord'
USER_AGENT = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"
def parse_google_geocode(full_address)
full_address = Rack::Utils.escape(full_address)
url_full = "http://maps.googleapis.com/maps/api/geocode/json?address=#{full_address}"
proxy = @proxies.first
begin
geocode = JSON.parse(open(url_full, :proxy_http_basic_authentication => ["http://#{proxy}", PROXY_LOGIN, PROXY_PASSWORD], "User-Agent" => "HARVESTOR NIGHTMARE V0.1").read)
if geocode['status'] == "OK"
if check_geocode(geocode)
geo = parse_geo(geocode['results'][0]['geometry']['location'])
address = parse_address(geocode['results'][0]['address_components'])
full_geolocation = address.merge(geo)
return full_geolocation
else
return false
end
elsif geocode['status'] == "OVER_QUERY_LIMIT"
puts "OVER_QUERY_LIMIT"
@proxies.rotate!
parse_google_geocode(full_address)
elsif geocode['status'] == "ZERO_RESULTS"
puts "ZERO_RESULTS"
return {:lat =>nil, :lng => nil, :postal_code => nil}
else
return false
end
rescue => e
puts "#{proxy}:#{e}"
@proxies.rotate!
parse_google_geocode(full_address)
end
end
def check_geocode(geocode)
geocode['results'].present? && geocode['results'][0].present? && geocode['results'][0]['geometry'].present? && geocode['results'][0]['geometry']['location'].present? && geocode['results'][0]['geometry']['location']['lat'].present? && geocode['results'][0]['geometry']['location']['lng'].present?
end
def parse_geo(geo)
return {:lat =>geo['lat'], :lng => geo['lng']}
end
def parse_address(address_components)
address = {street_number: "", route: "", city: "", state: "", country: "", postal_code: "", neighborhood: ""}
address_components.each do |ac|
case ac["types"].first
when "street_number"
address[:street_number] = ac["short_name"]
when "route"
address[:route] = ac["short_name"]
when "locality" || "sublocality_level_1"
address[:city] = ac["long_name"]
when "sublocality_level_1"
address[:city] = ac["long_name"]
when "administrative_area_level_1"
address[:state] = ac["short_name"]
when "country"
address[:country] = ac["short_name"]
when "postal_code"
address[:postal_code] = ac["short_name"]
when "neighborhood"
address[:neighborhood] = ac["short_name"]
end
end
return address
end
end
To start an instance of the script I run Property.add_neighborhood(1, 1000000) for example.
How can I track which parts are consuming all the memory and what can I improve?
I'm using "ruby-2.1.4-railsexpress". I am under the impression that the array rotate for the proxies is the issue. I'm not sure what's the best solution for this.
