# Generate word list based on Spanish text file

I'm a beginner and wrote a program that takes a text file and writes a downcased vocabulary list to another text file. I intend to use it mainly to work with text in Spanish, so I added a line to downcase capitalized-accented words. I'm wondering if there is a more efficient way of reading from the original file, as well as removing non-letters and sorting for unique items.

f = File.open("/.../quijote.txt")
f.close

#remove numbers and non-letters
words = words.map {|item| item.tr('0-9.,;:¿¡?!«»\‘\“\”\–\]$\-\'\"', '')} #downcase capitalized accented words words = words.map {|item| item.tr('ÁÉÍÓÚÑ', 'áéíóúñ')} words = words.uniq.sort # write each word on a separate line in the file... File.open("/.../quijotewords.txt", "w+") do |f| words.each { |element| f.puts(element) } end  ## 3 Answers Some notes: • open + read + close: Better to use the block form: contents = File.open(path) { |fd| fd.read } or simply contents = File.read(path) • words = words.something: Don't re-use variable names. New values, new names. For example: sorted_words = words.sort. • Use File.write • Instead of removing chars that you don't want, I'd remove the chars that you do want. • You can apply the processing to the whole file or line and then split. • string.tr(something, '') -> string.delete(something). I'd write: words = File.read("quijote.txt").downcase. tr("ÁÉÍÓÚÑ", "áéíóúñ").delete("^[a-z]áéíóúüñ \n"). split.uniq.sort File.write("quijote-words.txt", words.join("\n"))  If your input file is not UTF8-encoded but, let's say, ISO8859-15, you'd write: File.read("quijote.txt", encoding: "iso8859-15").encode("utf-8"). • Thank you. This worked for some of my files, but for others I'm getting rb:2:in downcase': invalid byte sequence in UTF-8 (ArgumentError) I read that ruby does sometimes. Is there a way around it given your suggestion? – gonzalo2000 Mar 7 '16 at 2:23 Based on @tokland code. No need to join array with newline symbol -- #puts does that for you silently: File.open("quijote-words.txt", "w") do |file| file.puts File.read("quijote.txt"). downcase.tr("ÁÉÍÓÚÑ", "áéíóúñ"). delete("^[a-z]áéíóúüñ \n").split.uniq.sort end  • This is nice and compact. However, note a subtle change in behaviour: if it fails to open quijote.txt, it will still have clobbered quijote-words.txt anyway. – 200_success Mar 7 '16 at 12:15 Here's a bug: any numbers in the text would initially be classified as a word, then its digits and decimal point would be discarded, leaving an empty word to be written to the result. I recommend converting the garbage characters into spaces before word-splitting. (You can even coalesce the two tr calls into one.) f.close is rarely desirable. A less error-prone approach is to call File.open with a block, like you did at the end. Here, you wouldn't even need a block to read the file: just call File.read. You can avoid a lot of calls to words.map if you postpone the split. If you do need to use map or each, avoid using |item| or |element| as block parameter names; prefer a meaningful name like |word| instead. words = File.read('/.../quijote.txt') .tr('ÁÉÍÓÚÑ0-9.,;:¿¡?!«»\‘\“\”\–$\[\-\'\"', 'áéíóúñ ')
.downcase
.split
.uniq
.sort

File.open('/.../quijotewords.txt', 'w+') do |f|
f.puts(words)
end


Listing 'ÁÉÍÓÚÑ' as special cases is a hack, though. You should probably use a proper library to perform internationalization-sensitive downcase. Similarly, stripping punctuation would be better done using gsub(/[[:punct:]]/, ' ').