# Optimize the CSV Upload function in Ruby On Rails

The code is working fine however, I would like to fix code after unless and can I store thread ID in the database too? Is there a way to improve performance?

require 'csv'
class CsvDb
class << self
def convert_save(model_name, csv_data, field_name=nil)
target_model = model_name.classify.constantize
counter=0;
CSV.parse(csv_file) do |row|
if counter==0
temp=row
counter +=1
next
end
temp={}
end
end
ActiveRecord::Base.connection.close
end
#end
end
end

• I would like to fix code after unless and can I store thread ID in the database too? Could you re-phrase that? I have no idea what you're trying to convey. – Mast Apr 12 '16 at 11:27
• What do you mean by fix after unless, are you talking about formatting ? You know fixing code is off-topic. – Marc-Andre Apr 12 '16 at 14:43
• By "fix code" do you mean "improve"? – the Tin Man Apr 13 '16 at 3:53
• I am not asking about formatting, I am asking about using map filter that will make 14 line to two or three – Dinesh Saini Apr 13 '16 at 6:20
• Fix means to improve not formatting, Second concern is that if thread start a task how could I know that thread complete the task or not. – Dinesh Saini Apr 13 '16 at 6:21

The following code is not tested, maybe some minor changes will be necessary: I'm not 100% sure it will work as is, don't hesitate to tell me if there is anything wrong with the code

## Refactoring the existing code

The first thing I did was to set the headers option to true it will enable the parsing of the first line of the CSV and instead of giving you an array for each row it will return an hashmap with keys extracted from the header. It replace the code you wrote to parse the header of the file.

Then I replace the imbricated ternary operations to use a case when expression for the business_type_id which is far more readable.

Next I made two small function to extract all the needed values, I use reduce to automatically fill a hashmap from an array of symboles.

At least I merge all the hashmap in the constructor of the target_model before saving.

CSV.parse(csv_file, {headers: true, header_converters: :symbol}) do |row|
when 'Online' then 1
when 'Local' then 2
else 3
end

.save()

end

def extract_required_fields(row)
[:user_id, :name, :country_id, :free_shipping, :category_ids, :style_ids, :shipping_country_ids]
.reduce({}) do |carry, item|
carry[item] = row[item]
end
end

def extract_optionals_fiels(row)
.reduce({}) do |carry, item|
carry[item] = row[item] ||= ''
end
end


## Bulk update

But this code, even if it has a better separation of concern and less duplication will not really speed your import process because your performance bottleneck is in the insert operation performed for each item by the save method.

So the solution is to make a bulk update instead of saving each row separately: you'll group the insert !

To do so, you can use the https://github.com/zdennis/activerecord-import gem which will allow you to do something like this :

entities = []
when 'Online' then 1
when 'Local' then 2
else 3
end

target_model.import entities if entities.size == 100
end

#don't forget to save the remaining entities:
target_model.import entities unless entities.size == 0


In this case we proceed the save operation each 100 row, which will make the import process way faster !

To collect the thread's Id, I suggest to use the inspect method of the current_thread, you'll get more information than only the thread's id but you'll be able to have all the needed informations about which thread processed the insert operation.

thread_identifier = Thread.current.inspect

• Should I have to write code in model now or I can use same code as it is placed now. – Dinesh Saini Apr 18 '16 at 5:18
• You should be able to use the code as it is placed now – lilobase Apr 18 '16 at 7:59
• But there is issue with bulk update, I have to skip few columns which is not possible by gem. – Dinesh Saini Apr 18 '16 at 8:59
• You have to skip columns or row ? For the columns, just remove them from the constructor's hash, for the rows you can make the append (<<) conditional. – lilobase Apr 18 '16 at 9:28
• Need to remove columns because the column name is same as of nested model. That model is just for main user help who create csv: CSV has fields [:user_id, :shipping_countries, shipping_country_ids] here shipping_countries is useless. The first case cover this case but gem one not. – Dinesh Saini Apr 18 '16 at 9:40

No idea what "thread ID" means.

I'll just focus on the the translation of the CSV rows into an object. There's just to much repetition in the unless body. All those rows should be mentioned once, then handled in a loop instead of hard coding them like this. E.g.

unless row[row_headers["name"]].nil?
temp={}
for name in [:user_id, :name, :country_id, :latitude, :longitude, :free_shipping, :category_ids, :style_ids, :shipping_country_ids]