Compare two CSV files In Ruby

Question

I wrote my first program in Ruby that compares two CSV files, but I'm sure there are more efficient ways to do it. I tried using the Ruby CSV library at first, but it was unproductive. Please let me know if you see a better way of doing it. Specifically, I am asking if anyone knows a simpler way to compare two CSV lists other than loading each first and last name into an array and comparing them, then joining them.

Here is a sample name list that I am comparing to:

Kanye,West
Jay,Z
Chance,TheRapper
Two,Chains

---

# A program to compare two CSV lists 
# and return data that is on both lists

# Files must be present in the directory the
# program is in and the two files must
# match the naming conventions in the program

# open an array for each list of names
people = Array.new
people1 = Array.new

# load each list into a variable
first_list = File.open("namelist.csv", "r")
second_list = File.open("namecomplist.csv", "r")

# split the file up by line and push
# the contents of the line to an array
# each element is delineated by the new line
# character "\n"
first_list.each_line { |line|
    fields = line.split('\n')
    people.push(fields)
}

# split the second file up by line and push
# the contents of the line to an array
second_list.each_line { |line|
    fields = line.split('\n')
    people1.push(fields)
}

# There has to be a cleaner way to do the folowing
# Put the data that is in both files in a variable
final_list = people & people1

# convert the array to a string by joing it
final_list1 = final_list.join

# print the matching data and substitute a comma
# for a space for formatting purposes
# the result should be the names of the people
# who are on both lists
puts final_list1.gsub(',', ' ')

Answer 1

Parsing CSV data

First, let me point out that what you are doing is not proper CSV parsing. Recommended reading: How to parse CSV data with Ruby. Carving up CSV files manually is not often ideal, as it can lead to mistakes, such as what I will explain below.

Using a well established library is most often a better choice. This also holds true of other types of data files, such as XML and JSON. Those libraries help improve consistency and reduce bugs.

Once you've chosen a library to use, read further into how to use that library effectively. You mentioned that you are pretty new, so I will leave that here as something to study when you feel ready to tackle it. =)

---

Bugs

Your code "works" for your simple use case, but only because you are comparing identical CSV files, rather than because the CSV files contain the same data. What your code is doing is splitting up a string (the characters from your CSV files) into lines, and comparing those. If the lines are identical, but are not in the same order, then you start getting incorrect results. For demonstration, you can try this:

sample_data1 = \
'Kanye,West
Jay,Z
Chance,TheRapper
Two,Chains'

sample_data2 = \
'Jay,Z
Kanye,West
Two,Chains
Chance,TheRapper'

# open an array for each list of names
people = Array.new
people1 = Array.new

# load each list into a variable
first_list = sample_data1
second_list = sample_data2

# rest of the code stays the same

I would expect the result to be the same either way (if it was working correctly), however this is what I get back:

Kanye West
Jay Z

Interesting. Let's see why this is happening, by adding some print and puts instructions in your each_line statements. (note that I converted from curly brackets to do...end syntax since it makes it simpler for statements which aren't one-liners)

puts '- first_list'
first_list.each_line do |line|
    fields = line.split('\n')
    print fields
    people.push(fields)
    puts
end

Did the same thing to second_list. Here is what the console shows now. I added the comments to show you what is happening.

- first_list
["Kanye,West\n"]
["Jay,Z\n"]
["Chance,TheRapper\n"] # has newline
["Two,Chains"] # no newline
- second_list
["Jay,Z\n"]
["Kanye,West\n"]
["Two,Chains\n"] # has newline
["Chance,TheRapper"] # no newline

The issue that is causing those errors is as follows. In Ruby, single-quoted strings and double-quoted strings do not work the same way.

When you use single-quoted strings, they are string literals, so '\n' is not the newline escape character; rather it is a backslash character and the lowercase letter n. If you want to be able to use backslash escaping, you need to use double-quoted string "\n" instead. (this also holds true of things like string interpolation)

You will also notice that you have an array of arrays (or 2D array), which is fine if that's what you need, but I suspect that's possibly not what you expected.

The reason each value is an array is because you are splitting them in a loop with each_line, and then "splitting" (incorrectly due to single quote) each line again with .split('\n') and pushing that into the array.

Implementing the above fixes:

• Splitting on "\n" after the file is read, not while iterating;
• Using double-quotes to look for newline characters rather than string literals of '\n';
• Pushing split strings into the destination array, rather than into sub-arrays/

---

# open an array for each list of names
people = Array.new
people1 = Array.new

# load each list into a variable
first_list = File.open("namelist.csv", "r").split("\n")
second_list = File.open("namecomplist.csv", "r").split("\n")

# split the file up by line and push
# the contents of the line to an array
# each element is delineated by the new line
# character "\n"
first_list.each do |line|
    people.push(line)
end

# split the second file up by line and push
# the contents of the line to an array
second_list.each do |line|
    people1.push(line)
end

This now functions "correctly", at least to the extent you want to compare for the same lines of comma-separated characters. Note that this is still a naive solution, because if there were spaces or other unexpected characters between the commas and words, it would also not work correctly.

---

Review

With the nasty bug out of the way, here are some things I would do to improve your code.

First, I would suggest to make a method to read the CSV file. Flambino's answer goes into more detail on that matter, so I will just post a naive implementation that works OK. I recommend to heed Flambino's advice on this file-reading matter.

# Converts a CSV raw character string into
# an array of newline-separated strings.
def load_list_from_csv(csv_file_raw_str)
  list = csv_file_raw_str.split("\n")
end

Then we can just update this:

# load each list into a variable
first_list = load_list_from_csv(sample_data1)
second_list = load_list_from_csv(sample_data2)

Or in your case...

# load each list into a variable
first_list = load_list_from_csv("namelist.csv", "r")
second_list = load_list_from_csv("namecomplist.csv", "r")

While we're at it, let's make a method to compare the 2 files. Basically just combining a few bits of code into one method. By the way, the operation you are making on those data sets is called an intersection.

Here is a working code example on repl.it showing the changes:

sample_data1 = \
'Kanye,West
Jay,Z
Chance,TheRapper
Two,Chains'

sample_data2 = \
'Jay,Z
Kanye,West
Two,Chains
Chance,TheRapper'

# Converts a CSV raw character string into
# an array of newline-separated strings.
def load_list_from_csv(csv_file_raw_str)
  list = csv_file_raw_str.split("\n")
end

# Intersect two arrays to return identical strings/lines
def intersect_csv_names_lists(file1, file2)
  people1 = load_list_from_csv(file1)
  people2 = load_list_from_csv(file2)
  [people1 & people2].join("\n")
end

# Replace commas with spaces in a string
def format_replace_commas_with_spaces(arr)
  arr.gsub(',', ' ')
end

# Print the matching data and substitute a comma
# for a space for formatting purposes.
# The result should be the names of the people
# who are on both lists.
puts format_replace_commas_with_spaces(intersect_csv_names_lists(sample_data1, sample_data2))

Output:

ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]

 Kanye West
 Jay Z
 Chance TheRapper
 Two Chains
=> nil

Answer 2

• As others have pointed out, the line.split('\n') calls are odd. Each line can contain a \n, but it's at the end, so at best you'd just get ["name,name", ""] when splitting (Ruby actually skips the empty string). However, as Phrancis explains, since you're using single quotes, you're splitting on the string backslash+n which probably doesn't exist in any line - you're not splitting on the linefeed character.
  If anything, use chomp to remove the trailing linebreak, or strip to remove all leading/trailing whitespace.

• File.open should be followed up by a call to close the file too

• But instead of File.open, you can use File.readlines to get an array right away, without having to remember to close.

I'd do this:

# Simple method to keep things DRY
def load_names(filename)
  File.readlines(filename)
    .map(&:strip) # remove leading/trailing whitespace
    .reject(&:empty?) # skip blank lines (e.g. if file ends with a blank line)
end

# Get common names
common_names = load_names("file1.csv") & load_names("file2.csv")

# Do something with the list
common_names.each do |name|
  puts name.gsub(/,/, ' ')
end