Revisions to Reduce memory use for a property geolocator

edited body

Source Link

edited Dec 8, 2014 at 15:03

111
3

You are repeatedly rotating the entire Array in place-place and then calling #first, this is \$\Theta(N^2)\$ behavior for a single traversal through the proxy list.

Instead you could create a cyclical Enumerator (think Generators in Python) to hold your shuffled proxies and advance it whenever a proxy is over-used. This \$\Theta(N)\$ access pattern should reduce memory usage somewhat:

 @proxies = Enumerator.new do |fresh_proxies|
   # Using #pluck avoids building intermediate objects before projection
   Proxy.pluck(:ip).shuffle.cycle.each do |proxy|
     fresh_proxies.yield proxy
   end
 end

Now you can replace accessing #first with:

def current_proxy
  @proxies.peek
end

and your #rotate! with:

def advance_to_next_proxy!
  @proxies.next
end

Docs:

You are repeatedly rotating the entire Array in place and then calling #first, this is \$\Theta(N^2)\$ behavior for a single traversal through the proxy list.

Instead you could create a cyclical Enumerator (think Generators in Python) to hold your shuffled proxies and advance it whenever a proxy is over-used. This \$\Theta(N)\$ access pattern should reduce memory usage somewhat:

 @proxies = Enumerator.new do |fresh_proxies|
   # Using #pluck avoids building intermediate objects before projection
   Proxy.pluck(:ip).shuffle.cycle.each do |proxy|
     fresh_proxies.yield proxy
   end
 end

Now you can replace accessing #first with:

def current_proxy
  @proxies.peek
end

and your #rotate! with:

def advance_to_next_proxy!
  @proxies.next
end

Docs:

You are repeatedly rotating the entire Array in-place and then calling #first, this is \$\Theta(N^2)\$ behavior for a single traversal through the proxy list.

Instead you could create a cyclical Enumerator (think Generators in Python) to hold your shuffled proxies and advance it whenever a proxy is over-used. This \$\Theta(N)\$ access pattern should reduce memory usage somewhat:

 @proxies = Enumerator.new do |fresh_proxies|
   # Using #pluck avoids building intermediate objects before projection
   Proxy.pluck(:ip).shuffle.cycle.each do |proxy|
     fresh_proxies.yield proxy
   end
 end

Now you can replace accessing #first with:

def current_proxy
  @proxies.peek
end

and your #rotate! with:

def advance_to_next_proxy!
  @proxies.next
end

Docs:

added 9 characters in body

Source Link

edited Dec 8, 2014 at 14:11

Aaron Cronin

111
3

You are repeatedly rotating the entire Array in place and then calling #first, this is O(N^2)\$\Theta(N^2)\$ behavior for a single traversal through the proxy list.

Instead you could create a cyclical Enumerator (think Generators in Python) to hold your shuffled proxies and advance it whenever a proxy is over-used. This \$\Theta(N)\$ access pattern should reduce memory usage somewhat:

 @proxies = Enumerator.new do |fresh_proxies|
   # Using #pluck avoids building intermediate objects before projection
   Proxy.pluck(:ip).shuffle.cycle.each do |proxy|
     fresh_proxies.yield proxy
   end
 end

Now you can replace accessing #first with:

def current_proxy
  @proxies.peek
end

and your #rotate! with:

def advance_to_next_proxy!
  @proxies.next
end

Docs:

You are repeatedly rotating the entire Array in place and then calling #first, this is O(N^2) behavior for a single traversal through the proxy list.

Instead you could create a cyclical Enumerator (think Generators in Python) to hold your shuffled proxies and advance it whenever a proxy is over-used.

 @proxies = Enumerator.new do |fresh_proxies|
   # Using #pluck avoids building intermediate objects before projection
   Proxy.pluck(:ip).shuffle.cycle.each do |proxy|
     fresh_proxies.yield proxy
   end
 end

Now you can replace accessing #first with:

def current_proxy
  @proxies.peek
end

and your #rotate! with:

def advance_to_next_proxy!
  @proxies.next
end

Docs:

You are repeatedly rotating the entire Array in place and then calling #first, this is \$\Theta(N^2)\$ behavior for a single traversal through the proxy list.

Instead you could create a cyclical Enumerator (think Generators in Python) to hold your shuffled proxies and advance it whenever a proxy is over-used. This \$\Theta(N)\$ access pattern should reduce memory usage somewhat:

 @proxies = Enumerator.new do |fresh_proxies|
   # Using #pluck avoids building intermediate objects before projection
   Proxy.pluck(:ip).shuffle.cycle.each do |proxy|
     fresh_proxies.yield proxy
   end
 end

Now you can replace accessing #first with:

def current_proxy
  @proxies.peek
end

and your #rotate! with:

def advance_to_next_proxy!
  @proxies.next
end

Docs:

added 1 character in body

Source Link

edited Dec 8, 2014 at 12:22

Aaron Cronin

111
3

You are repeatedly rotating the entire Array in place and then calling #first, this is O(N^2) behavior for a single traversal through the proxy list.

Instead you could create a cyclical Enumerator (think Generators in Python) to hold your shuffled proxies and advance it whenever a proxy is over-used.

 @proxies = Enumerator.new do |fresh_proxies|
   # Using #pluck avoids building intermediate objects before projection
   Proxy.pluck(:ip).shuffle.cycle.each do |proxy|
     fresh_proxies.yield proxy
   end
 end

Now you can replace accessing #first with:

def get_current_proxycurrent_proxy
  @proxies.peek
end

and your #rotate! with:

def temporarily_retire_proxyadvance_to_next_proxy!
  @proxies.next
end

Docs:

You are repeatedly rotating the entire Array in place and then calling #first, this is O(N^2) behavior for a single traversal through the proxy list.

Instead you could create a cyclical Enumerator (think Generators in Python) to hold your shuffled proxies and advance it whenever a proxy is over-used.

 @proxies = Enumerator.new do |fresh_proxies|
   # Using #pluck avoids building intermediate objects before projection
   Proxy.pluck(:ip).shuffle.cycle.each do |proxy|
     fresh_proxies.yield proxy
   end
 end

Now you can replace accessing #first with:

def get_current_proxy
  @proxies.peek
end

and your #rotate! with:

def temporarily_retire_proxy
  @proxies.next
end

Docs:

You are repeatedly rotating the entire Array in place and then calling #first, this is O(N^2) behavior for a single traversal through the proxy list.

Instead you could create a cyclical Enumerator (think Generators in Python) to hold your shuffled proxies and advance it whenever a proxy is over-used.

 @proxies = Enumerator.new do |fresh_proxies|
   # Using #pluck avoids building intermediate objects before projection
   Proxy.pluck(:ip).shuffle.cycle.each do |proxy|
     fresh_proxies.yield proxy
   end
 end

Now you can replace accessing #first with:

def current_proxy
  @proxies.peek
end

and your #rotate! with:

def advance_to_next_proxy!
  @proxies.next
end

Docs:

added 75 characters in body

Source Link

edited Dec 8, 2014 at 12:16

Aaron Cronin

111
3

Loading

added 8 characters in body

Source Link

edited Dec 8, 2014 at 12:09

Aaron Cronin

111
3

Loading

Source Link

created Dec 8, 2014 at 12:03

Aaron Cronin

111
3

Loading

Stack Exchange Network

Return to Answer