Process a line in chunks via Regular Expressions with Ruby

I'm writing a Ruby script to reverse engineering message log files. They come from an external system with the following characteristics:

• Each line of the log file has at least one message.
• Each line of the log file can have multiple messages.
• Each message consists of a set of numbers separated by spaces (e.g. 30 0 -1 1 2 1).
• Each message can have one of many different templates (e.g. some contain five numbers, others contains six).

The approach I'm using is to process each line, one at a time, via a method that takes a string to work on as an argument. It saves a copy of the initial input (for later comparison) then tries to match known patterns. When a pattern is matched, the string that made it up is removed. If there is nothing left, or if no more matches are found, the method exits. Otherwise, it calls itself with the remainder of the string to process. Here's the code I camp up with along with an example.

#!/usr/bin/evn ruby

def parse_line remainder_of_line

puts "Processing: #{remainder_of_line}"

# Save a copy of the initial input for later comparison
initial_snapshot = remainder_of_line.dup

# Look for known pattern matches, removing them if found
if remainder_of_line.gsub!(/^(\d+) 0 -1 1 (\d+) \d+\s*/, '')
puts " - Matched format 1 - found: #{$1} - #{$2}\n\n"
elsif remainder_of_line.gsub!(/^\d+ 0 -1 2 (\d+) \d+\s*/, '')
puts " - Matched format 2 - found: #{$1}\n\n" ### More patterns here. end # If noting changed, then no matches were found. if initial_snapshot.eql? remainder_of_line puts " - Line still has data but no matches found. (Left with: #{remainder_of_line}\n\n" # Keep going if there is anything left. elsif !remainder_of_line.empty? parse_line remainder_of_line end end line = "11 0 -1 2 13560 2 11 0 -1 2 13564 2 11 0 -1 1 36880 106 91 0 -1 1 36881 106 36881 106 91 1 13556 2 36880 106 36880 106 11 1 734 11 0 -1 1 36884 106 91 0 -1 1 36885 106 36885 106 91 1 13556 2 36884 106 36884 106 11 1 735 13556 2 31 18 799 13556 2 31 25 799 " parse_line line This works but I'm wondering if there is a better way. • Two questions: 1. Suppose the first part of a line has been matched by zero or more patterns. Could zero, two or more than two patterns match the beginning of the remainder of the line, or will it always be matched by exactly one pattern (i.e., can there be ambiguity or no match)? (cont.) – Cary Swoveland Aug 25 '14 at 2:21 • (..cont.) 2. Must you use patterns or are there other ways to identify the beginning or end of each message? In line, for example, I see many positive numbers with three or fewer digits, some -1's and some numerous 5-digit numbers. Could one identify messages merely by numbers of digits (e.g., each message begins with a number with 3 or fewer digits and ends with a 5-digit number)? – Cary Swoveland Aug 25 '14 at 2:23 1 Answer • Because you're using the "bang" version of gsub, parse_line modifies the string you pass to it, which is generally a not a good idea. I wouldn't expect a parsing method to "eat" my input. • Since there's only one line and your regexes are anchored to the start of it, there's little point in using gsub (i.e. global substitution), since you'll only ever match 1 occurrence of the pattern. • Don't bother with all the newline literals. puts will automatically add one, and if you want an extra one, you should be able to just say puts with no argument in a strategic location (i.e. after having tried all the patterns). This seems like a good fit for Ruby's case statement (aka switch) since you can match against regexes directly. And Ruby also sets other magic variables besides$1 and $2 whenever you match a regex. There's no reason to make the method recursive, though. A simple loop would do nicely too. For instance: def parse_line(line) puts "Processing: #{line}" # Loop until the string's empty (or we hit the return below) until line.empty? # Try matching the line case line when /^(\d+) 0 -1 1 (\d+) \d+\s*/ puts " - Matched format 1 - found: #{$1} - #{$2}" when /^\d+ 0 -1 2 (\d+) \d+\s*/ puts " - Matched format 2 - found: #{$1}"

# more patterns...

else # no match
puts " - Line still has data but no matches found. (Left with: #{line})"
return # stop here
end
line = \$' # set line to the *unmatched* part, i.e. the remainder
puts "" # output an extra blank line
end

puts "Entire line matched, yay"
end
• I really like this but found a gotcha. In my original, simplified example I didn't show a case where further processing is done to the message string. For example, if the second match is triggered, additional items have to be shifted off the message stack in one of a number of ways. This is done by a different method which returns the updated line. (This is the reason I'm removing items from the message line when I'm done with them.) I've updated my example to demonstrate that case. – Alan W. Smith Aug 21 '14 at 0:56
• @AlanW.Smith Please don't alter the code in your question like that. I reviewed the code as presented, because that's all I could do - I don't know more than what you tell me. When you change your question, my answer becomes invalid, though it wasn't before. This is why questions must contain real code to begin with. You're more than welcome to post a new question with new code (really, post as many as you want!), but please roll back your edit here. Otherwise the Q&A style gets broken, as the As stop matching the Qs and vice-versa. – Flambino Aug 21 '14 at 7:40
• @AlanW.Smith Oh, and if you want to remove more numbers, then just match those in your regexes, too. E.g. if format 1 always necessitates removing an extra number, simply put that extra number in the pattern to begin with - you know it's going to be there. – Flambino Aug 21 '14 at 12:00
• Yeah, I went a little to far pruning the code example. It's real but there is a alot of other processing that happens with each match. I tried to make a representative sample that would actually fit, but should have stubbed the other method calls. I thought adding a new section (instead of altering the original) would make sense since it is a gotcha that cropped up. Since that changes the question a little, I'll roll this one back and open a new one. – Alan W. Smith Aug 21 '14 at 12:17
• @AlanW.Smith Great - I appreciate that. And the checkmark :) And yes, it does often make sense to prune your code a bit before posting, but in this case, the overall function became just "match" rather than "match and process". I assumed you were mostly interested in seeing how much of a line you could match (since you said you were reverse engineering the formats), not so much the matches themselves. That'd be a concern for future code maybe. Since your code wasn't using the matches for anything (besides printing), it seemed like a reasonable assumption at the time – Flambino Aug 21 '14 at 14:00