I'm a beginner and wrote a program that takes a text file and writes a downcased vocabulary list to another text file. I intend to use it mainly to work with text in Spanish, so I added a line to downcase capitalized-accented words. I'm wondering if there is a more efficient way of reading from the original file, as well as removing non-letters and sorting for unique items.

f = File.open("/.../quijote.txt")
words = f.read.split.map(&:downcase)

#remove numbers and non-letters
words = words.map {|item| item.tr('0-9.,;:¿¡?!«»\‘\“\”\–\]\[\-\(\)\'\"', '')}

#downcase capitalized accented words
words = words.map {|item| item.tr('ÁÉÍÓÚÑ', 'áéíóúñ')}

words = words.uniq.sort

# write each word on a separate line in the file...
File.open("/.../quijotewords.txt", "w+") do |f|
  words.each { |element| f.puts(element) }

Some notes:

  • open + read + close: Better to use the block form: contents = File.open(path) { |fd| fd.read } or simply contents = File.read(path)
  • words = words.something: Don't re-use variable names. New values, new names. For example: sorted_words = words.sort.
  • Use File.write
  • Instead of removing chars that you don't want, I'd remove the chars that you do want.
  • You can apply the processing to the whole file or line and then split.
  • string.tr(something, '') -> string.delete(something).

I'd write:

words = File.read("quijote.txt").downcase.
  tr("ÁÉÍÓÚÑ", "áéíóúñ").delete("^[a-z]áéíóúüñ \n").
File.write("quijote-words.txt", words.join("\n"))

If your input file is not UTF8-encoded but, let's say, ISO8859-15, you'd write: File.read("quijote.txt", encoding: "iso8859-15").encode("utf-8").

  • \$\begingroup\$ Thank you. This worked for some of my files, but for others I'm getting rb:2:in downcase': invalid byte sequence in UTF-8 (ArgumentError) I read that ruby does sometimes. Is there a way around it given your suggestion? \$\endgroup\$ – gonzalo2000 Mar 7 '16 at 2:23

Based on @tokland code.
No need to join array with newline symbol -- #puts does that for you silently:

File.open("quijote-words.txt", "w") do |file|
  file.puts File.read("quijote.txt").
    downcase.tr("ÁÉÍÓÚÑ", "áéíóúñ").
    delete("^[a-z]áéíóúüñ \n").split.uniq.sort
  • 2
    \$\begingroup\$ This is nice and compact. However, note a subtle change in behaviour: if it fails to open quijote.txt, it will still have clobbered quijote-words.txt anyway. \$\endgroup\$ – 200_success Mar 7 '16 at 12:15

Here's a bug: any numbers in the text would initially be classified as a word, then its digits and decimal point would be discarded, leaving an empty word to be written to the result. I recommend converting the garbage characters into spaces before word-splitting. (You can even coalesce the two tr calls into one.)

f.close is rarely desirable. A less error-prone approach is to call File.open with a block, like you did at the end. Here, you wouldn't even need a block to read the file: just call File.read.

You can avoid a lot of calls to words.map if you postpone the split. If you do need to use map or each, avoid using |item| or |element| as block parameter names; prefer a meaningful name like |word| instead.

words = File.read('/.../quijote.txt')
  .tr('ÁÉÍÓÚÑ0-9.,;:¿¡?!«»\‘\“\”\–\]\[\-\(\)\'\"', 'áéíóúñ ')

File.open('/.../quijotewords.txt', 'w+') do |f|

Listing 'ÁÉÍÓÚÑ' as special cases is a hack, though. You should probably use a proper library to perform internationalization-sensitive downcase. Similarly, stripping punctuation would be better done using gsub(/[[:punct:]]/, ' ').


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.