Splitting a text file into paragraphs and words

Question

I'm iterating through a number of text files, trying to locate all carriage-returns, and individually save the text between carriage-returns. I get the index numbers of all carriage-returns, but I haven't got the slightest about saving the text.

Basically I want to save every string between two carriage-returns into an individual variable. The next step is to save all the words within a string as an individual hash.

Here is my code so far (edited based on the help by Tin Man and Screenmutt) to get every single paragraph of a file into an array:

# script start

# outputfile
output_text = File.open("output.txt", 'w')

# directory with files
Dir.chdir("nkamp")

#count lines
lines = File.readlines("first.txt")
line_count = lines.size
text = lines.join
paragraph_count = text.split("\.\r").length
puts "#{paragraph_count} paragraphs."

#array of paragraphs
paragraphs = Array.new
contents = []
File.foreach("first.txt", "\.\r") do |paragraph|
  puts paragraph.chomp
  puts '-' * 40
  contents << paragraph.chomp
  paragraphs << paragraph.chomp
end

puts paragraphs[10]

This code gives me an array with all the paragraphs. I am using ".\r" instead of "\n\n" because the texts are copied from PDF files, and have lost the normal page layout structures.

The next step is to save an array of the words in a paragraph into the array instead of simply a string of text:

words_in_each_paragraph = Array.new

File.foreach("Ann Reg Sci (2).txt", "\.\r") do |paragraph|
    word_hash = {}
    paragraph.split(/\W+/).each_with_object(word_hash) { |w, h|
        h[w] = []
    }
    words_in_each_paragraph << word_hash
end

puts words_in_each_paragraph[8]

Which gives the following output:

{""=>[], "The"=>[], "above"=>[], "contributions"=>[], "highlight"=>[], "the"=>[], "importance"=>[], "of"=>[], "sophisticated"=>[], "modeling"=>[], "work"=>[], "for"=>[], "a"=>[], "better"=>[], "understanding"=>[], "complexity"=>[], "entrepreneurial"=>[], "space"=>[], "economy"=>[]}

Now the next step is loop through every file, and create a dynamic hash that gives me

a. a number for the article. b. a number for the paragraph. c. the list of words as seen above.

For this I need to learn how to dynamically create hashes. This is where it goes wrong:

lines = File.readlines("test.txt")
line_count = lines.size
text = lines.join
paragraph_count = text.split("\.\r").length
puts "#{paragraph_count} paragraphs."

testArray = Array.new(paragraph_count.to_i, Hash.new)
for i in 0...paragraph_count.to_i do
    testArray[i] = Hash.new 
    puts "testArray #{i} has been made"
end
words_in_each_paragraph = Array.new

File.foreach("test.txt", "\.\r") do |paragraph|
    word_hash = {}
    paragraph.split(/\W+/).each_with_object(word_hash) { |w, h|
        h[w] = []
    }
    words_in_each_paragraph << word_hash
    testArray[i][:value] = word_hash
    puts testArray[i] # IT WORKS HERE #
end

puts testArray[1] # AND IT DOESN'T WORK HERE #

Answer 1

So there's a lot going on here - a lot more than there needs to be. You're working way too hard for what is actually very simple.

A few notes: I can't review code that hasn't been written. You're saying you want the "article number", but I don't see code for anything like that, so I'll ignore that bit for now. Similarly, you talk about "a number of files" but all your current code has hard-coded file names (that are different from code block to code block), so I'll leave that alone too. That leaves finding paragraphs and the words in those paragraphs.

First, though, here are some of the things I noticed in your code:

• Your code keeps doing the same things over and over. You open, read, and split the target file into paragraphs again and again. And in different ways.
• A lot of code is never used for anything, or made entirely redundant by later code. E.g. you read all the lines in the file... and then don't use that for anything before joining all the lines into a single string again. And then you split it again, only differently. And then you read everything again a moment later...
• You open the "output.txt" file for writing, but you never use it for anything. Nor do you close it again. I'd suggest just printing to stdout (i.e. puts) and then you can use I/O redirection in the shell to dump that into a file.

In all, there are problems, a lot of which, quite frankly, seem to suggest you don't understand your current code.

Here's a detailed look at your last code block

# Why use readlines when you don't need the actual lines?
lines = File.readlines("test.txt")

# Never used for anything
line_count = lines.size

# File.readlines + join is the same as just using File.read
text = lines.join

# Now you're re-splitting the text you just joined
paragraph_count = text.split("\.\r").length

# ... just so you can print this, which the next bit
# of code can do too, since it also splits the text
# into paragraphs
puts "#{paragraph_count} paragraphs."

# So all of the code above can safely be replaced with:
# Nothing (just delete it)

# When you use Array.new with a length and an object,
# that object is *repeated* - not cloned - across the
# array. So each index in the array points to the
# exact same object.
# This would likely lead to a bug (except that your
# next bit of code makes all of this redundant)
# Also, Ruby convention is to use underscores, so
# "test_array", not "testArray". Of course, that name
# is bad to begin with because it doesn't give you any
# hints about what it contains. 
testArray = Array.new(paragraph_count.to_i, Hash.new)

# You never need to use a basic for loop in Ruby, ever.
# Want you want is Enumerable#map, or - better yet -
# use a block initializer in the first place to populate
# the array
for i in 0...paragraph_count.to_i do
    # So here, you overwrite each index, making the
    # the above array initialization pointless
    testArray[i] = Hash.new
end

# Elsewhere in your post, you just used `[]` to create
# a new, empty array. Why not do that here? It's the
# most common way of doing it.
words_in_each_paragraph = Array.new

# And now you open the file yet again, and split it
# into paragraphs. Didn't we just do all this?
File.foreach("test.txt", "\.\r") do |paragraph|

    # So you make a hash, and make each word a key in
    # that hash. Presumably because you only want unique
    # words. But that's what Array#uniq is for
    word_hash = {}
    paragraph.split(/\W+/).each_with_object(word_hash) { |w, h|
        h[w] = []
    }

    # Why do this? You can transform the structure for
    # testArray into words_in_each_paragraph, and back
    # again, as needed, rather than build two pretty
    # similar arrays.
    words_in_each_paragraph << word_hash
    testArray[i][:value] = word_hash
end

So yeah, there are issues.

Here's my take on practically all of your code: How to get an array, where each item is itself an array of unique words in the paragraph

# read file, split into paragraphs, and map each paragraph
# into its unique, constituent words
paragraphs = File.read("test.txt").split(/\s*?\r\s*/).map do |paragraph|
  paragraph.scan(/[[:alnum:]]+/).uniq
end

Done. That's all of it in 3 lines.

And it's more careful with its regexp patterns: It doesn't split at anything plus \r, and it'll include non-"word" characters in words, so it won't split non-English names for instance.

Answer 2

text = File.read "test.txt"
puts "#{text.split("\n").size} lines."
paragraphs = text.split ".\n"
puts "#{paragraphs.length} paragraphs."

words_in_each_paragraph = paragraphs.map.with_index do |paragraph, i|
    puts "working on paragraph ##{i}"
    paragraph.split("\n").map.with_index do |line, j|
        puts "working on line ##{j}"
        line.scan(/\w+/).tap &method(:p)
    end
end

p words_in_each_paragraph[1]

1. i isn't inside File.foreach loop. It was used in previous looping, but not here.
2. The best practice to create a new array with the same length as another one is to use .map. Anyway try to avoid usage of i in Ruby. You shouldn't bother with indexes.
3. I assume you wanted to get a three-dimensional array, not the array and some hashes.
4. Method .tap can be handy for debug printing while processing/converting data. It allows do some post-processing while returning the object itself to keep your .map blocks safe.
5. Note, I used \n -- you may want to change it back to \r, but usually you don't need to think about line-endings -- \n should be ok everywhere since OS should take care about it.