Arrow test Ruby gem

I wrote my first Ruby gem, which is for testing code in the shortest way possible.

Here's a usage example:

require 'arrow_test'

1 + 1 # -> 3

arrow_test


The code (including RDoc documentation):

 # Performs the logic of the arrow  test.
#
# Example:
#   >> helper_arrow_test(["", "1+1 #-> 3"])
#   => ["ERROR:\n        EXECUTING: 1  + 1\n        EXPECTED: 3\n        INSTEAD GOT: 2\n\n\n"]
#
# Arguments:
#   lines: (String) The lines of code to be arrow tested.
#   verbose: (Boolean) If true returns also correct tests.
def helper_arrow_test(lines, verbose=false)
messages = []
arrow_regex = Regexp.new('\ *\#\ *\-\ *\>\ *')
thick_arrow_regex = Regexp.new('\ *\#\ *\=\ *\>\ *')
lines
.select{|line| arrow_regex.match(line) || thick_arrow_regex.match(line)}
.each do |line|
expression = line.split('#').first.strip
result = line.split('#').last.sub("\n", '').sub("-",'').sub(">",'').sub('=','').strip
got = eval(expression)
actual = eval(result)
if got != actual
messages.push <<END
ERROR:
EXECUTING: #{expression}
EXPECTED: #{result}
END
elsif verbose
messages.push <<END
CORRECT:
EXECUTING: #{expression}
REALLY_IS: #{got}\n\n
END
end
end
messages
end

# Performs Input/Output wrapping helper_arrow_test
# Call this in your code.
#
# Example:
#   require 'arrow_testing'
#   1 + 1 # -> 3
#   arrow_test
#
# Arguments:
#   filename: (String) The filename to be tested. Default is $0: your current file. # verbose: (Boolean) If true outputs also correct tests. def arrow_test(filename=$0, verbose=false)
puts helper_arrow_test(IO.foreach(filename).to_a, verbose)
end


I also have some tests using the UnitTestModule (testing my testing framework with itself seemed meaningless):

require 'test/unit'
require_relative 'lib/arrow_test'

class ArrowTestTest < Test::Unit::TestCase
def test_empty
assert(helper_arrow_test(['1 # -> 1']) == [], 'Empty is returned for no fails.')
end

def test_wrong_math
result = ["ERROR:\n        EXECUTING: 1  + 1\n        EXPECTED: 3\n        INSTEAD GOT: 2\n\n\n"]
assert(helper_arrow_test(['1  + 1 # -> 3']) == result, 'Wrong math is reported.')
end

def test_verbose_output
result = ["CORRECT:\n        EXECUTING: 1\n        REALLY_IS: 1\n\n\n"]
assert(helper_arrow_test(['1 # -> 1'], verbose=true) == result, "Verbose returns also correct expressions")
end

def test_thick_line_with_spaces
assert(helper_arrow_test(['1 #   => 1']) == [], 'Thick line also works.')
end
end


Fancily formatted documentation.

• Why would testing with arrow_test be meaningless? I think it would be very meaningful: it shows that you trust it enough to test a non-trivial program. As it stands now, the only thing someone looking at your code will see is that you don't even trust your program enough to use it yourself. Jul 31 '15 at 11:15

I like your idea for the syntax, but I fail to see its usefulness. You're really only able to test "static" code. Yes, 1 + 1 is 2, but what should 1 + x be? Well, that depends on what x is at runtime. But comments aren't written at runtime.

There's also the issue of scope. E.g. something simple like this won't work:

foo = "hello"
foo.reverse # => "oleh"


foo doesn't exist in eval's scope, so you don't even get a failure message, just a NoMethodError exception caused by calling reverse on nil.

So you're limited to testing only the most simple things. Sure, you can add an extra line to e.g. call a method with hardcoded arguments (provided said method can even be called from eval) and verify the result against a hardcoded value, but then your code would be full of junk code when not running tests. Especially as you probably want to check multiple arguments and return values. And if the expression you run changes the program's state in any way, you can't make that call without altering things.

E.g. take file like this:

require 'arrow_test'

$count = 0 def increment$count += 1
end

increment #-> 1

arrow_test($0, true)  First off, the expectation fails because the code's run once before testing, so increment has been called once already. When the testing system then calls it a second time, $count becomes 2. So the test fails.

If you "fix" the expectation to be 2, it instead stops making any sense as a test, because at that point in the code, it should definitely be 1. A comment saying 2 would only confuse everyone. Regardless of what you do the program's state is altered simply by running the tests. And passing tests may mean breaking the program. That's pretty bad.

Lastly, there's the matter of syntax. I can't run your first code block as a file, despite the if __FILE__ == $0 at the end implying I should be able to (edit: that refers to a previous version of question; it was edited while I was writing my answer), because the documentation contains an example of the -> syntax. So it tries to verify that this:  eval( ' # >> helper_arrow_test(["", "1+1' )  or, rather (due to some bugs I'll get to later) this:  eval( '' )  is equal to this: eval( '3"])' )  Obviously, that doesn't work out. Similarly, weird things will happen for anything spanning more than one line, e.g. a hash declaration, or a chained expression like the multiline lines.select {...}.each do .. end in your own code. It does make sense that you picked expr # => result as your syntax since it's commonly used in documentation. But the problem is that... well, that it's commonly used in documentation. So you can't document your code without the test system misinterpreting it. Your code isn't a full lexer/tokenizer, so it'll easily trip over all kinds of syntax. For instance: str = "here's a string containing # -> but it's not even a comment"  In the end, you can only really test that, say, a constant is what it is (provided, of course, that it's a global constant) but that's not useful. Or you can test expressions that have no effect like 1 + 1. Expressions with no effect is something compilers/linters for other languages actually complain about, because they don't really make sense. It's considered a probable bug. Sooo... yeah, I'm not too keen on it. Sorry. Still, there's code to review. Your regexes can be combined and simplified: pattern = /^(.*)#\s*[=-]>(.*)$/


That'll match both -> and =>, and capture what comes before and after. Since captures are stored in the global $1, $2, $3 etc. variables, you can do stuff like: if line =~ pattern expression =$1.strip
expected = $2.strip # ... end  Incidentally, expected is a better name than result. I'd think "result" would be the, well, the result of evaluating the expression. Using captures also skips the whole split.last.sub.sub.etc. line. That line is very buggy by the way: 1. You split at every #, so a line like "#hashtag" # -> "#hashtag" will be parsed as " and hashtag" skipping everything in between. Hence the bug mentioned above, where it ends up eval'ing an empty string. 2. strip strips leading/trailing whitespace - that includes newlines. So sub("\n", '') is unnecessary (besides, chomp would be more appropriate for removing a trailing newline). You should also use inspect when printing values in your "instead got" messages. Otherwise, nil becomes just an empty string, strings lose their quotes, and symbols lose their colons making it hard to tell what the actual result was. By the way, your heredocs should be indented. Overall, if you want inline (or at least "in-file") testing, you should rather use something like minitest/autorun (already included in the stdlib of recent Ruby versions). Or roll your own basic assert/assert_equal method: class Object def assert(value, expected, message = nil) if value != expected fail(message || "Assertion failed") end end end assert(1 + 1, 2, "Math failed us! 1 + 1 should be 2")  Of course, I'd rather repeat my suggestion that you instead look at test/unit, minitest, RSpec or any of the other existing testing libraries/frameworks. And of course there's the IRB console, for quickly trying things out. It just shows you the result, but you can always check it yourself, since the last result is stored in the _ variable: $ irb
irb(main):001:0> "hello".reverse
=> "olleh"
irb(main):002:0> _ == "oleh"
=> false


More typing, but very useful for quickly trying things out.