3
\$\begingroup\$

I wrote my first Ruby gem, which is for testing code in the shortest way possible.

Here's a usage example:

require 'arrow_test'

1 + 1 # -> 3

arrow_test

The code (including RDoc documentation):

 # Performs the logic of the arrow  test.
 #
 # Example:
 #   >> helper_arrow_test(["", "1+1 #-> 3"])
 #   => ["ERROR:\n        EXECUTING: 1  + 1\n        EXPECTED: 3\n        INSTEAD GOT: 2\n\n\n"]
 #
 # Arguments:
 #   lines: (String) The lines of code to be arrow tested.
 #   verbose: (Boolean) If true returns also correct tests.
def helper_arrow_test(lines, verbose=false)
  messages = []
  arrow_regex = Regexp.new('\ *\#\ *\-\ *\>\ *')
  thick_arrow_regex = Regexp.new('\ *\#\ *\=\ *\>\ *')
  lines
    .select{|line| arrow_regex.match(line) || thick_arrow_regex.match(line)}
    .each do |line|
      expression = line.split('#').first.strip
      result = line.split('#').last.sub("\n", '').sub("-",'').sub(">",'').sub('=','').strip
      got = eval(expression)
      actual = eval(result)
      if got != actual
        messages.push <<END
ERROR:
        EXECUTING: #{expression}
        EXPECTED: #{result}
        INSTEAD GOT: #{got}\n\n
END
      elsif verbose
        messages.push <<END
CORRECT:
        EXECUTING: #{expression}
        REALLY_IS: #{got}\n\n
END
       end
      end   
  messages
end

 # Performs Input/Output wrapping `helper_arrow_test`
 # Call this in your code.
 #
 # Example:
 #   require 'arrow_testing'
 #   1 + 1 # -> 3
 #   arrow_test
 #
 # Arguments:
 #   filename: (String) The filename to be tested. Default is $0: your current file.
 #   verbose: (Boolean) If true outputs also correct tests.
def arrow_test(filename=$0, verbose=false)
  puts helper_arrow_test(IO.foreach(filename).to_a, verbose)
end

I also have some tests using the UnitTestModule (testing my testing framework with itself seemed meaningless):

require 'test/unit'
require_relative 'lib/arrow_test'

class ArrowTestTest < Test::Unit::TestCase
  def test_empty
    assert(helper_arrow_test(['1 # -> 1']) == [], 'Empty is returned for no fails.')
  end

  def test_wrong_math
    result = ["ERROR:\n        EXECUTING: 1  + 1\n        EXPECTED: 3\n        INSTEAD GOT: 2\n\n\n"]
    assert(helper_arrow_test(['1  + 1 # -> 3']) == result, 'Wrong math is reported.')
  end

  def test_verbose_output
    result = ["CORRECT:\n        EXECUTING: 1\n        REALLY_IS: 1\n\n\n"]
    assert(helper_arrow_test(['1 # -> 1'], verbose=true) == result, "Verbose returns also correct expressions")
  end

  def test_thick_line_with_spaces
    assert(helper_arrow_test(['1 #   => 1']) == [], 'Thick line also works.')
  end
end

Fancily formatted documentation.

\$\endgroup\$
  • 1
    \$\begingroup\$ Why would testing with arrow_test be meaningless? I think it would be very meaningful: it shows that you trust it enough to test a non-trivial program. As it stands now, the only thing someone looking at your code will see is that you don't even trust your program enough to use it yourself. \$\endgroup\$ – Jörg W Mittag Jul 31 '15 at 11:15
5
\$\begingroup\$

I like your idea for the syntax, but I fail to see its usefulness. You're really only able to test "static" code. Yes, 1 + 1 is 2, but what should 1 + x be? Well, that depends on what x is at runtime. But comments aren't written at runtime.

There's also the issue of scope. E.g. something simple like this won't work:

foo = "hello"
foo.reverse # => "oleh"

foo doesn't exist in eval's scope, so you don't even get a failure message, just a NoMethodError exception caused by calling reverse on nil.

So you're limited to testing only the most simple things. Sure, you can add an extra line to e.g. call a method with hardcoded arguments (provided said method can even be called from eval) and verify the result against a hardcoded value, but then your code would be full of junk code when not running tests. Especially as you probably want to check multiple arguments and return values. And if the expression you run changes the program's state in any way, you can't make that call without altering things.

E.g. take file like this:

require 'arrow_test'

$count = 0

def increment
  $count += 1
end

increment #-> 1

arrow_test($0, true)

First off, the expectation fails because the code's run once before testing, so increment has been called once already. When the testing system then calls it a second time, $count becomes 2. So the test fails.

If you "fix" the expectation to be 2, it instead stops making any sense as a test, because at that point in the code, it should definitely be 1. A comment saying 2 would only confuse everyone. Regardless of what you do the program's state is altered simply by running the tests. And passing tests may mean breaking the program. That's pretty bad.

Lastly, there's the matter of syntax. I can't run your first code block as a file, despite the if __FILE__ == $0 at the end implying I should be able to (edit: that refers to a previous version of question; it was edited while I was writing my answer), because the documentation contains an example of the -> syntax. So it tries to verify that this:

 eval( ' #   >> helper_arrow_test(["", "1+1' )

or, rather (due to some bugs I'll get to later) this:

 eval( '' )

is equal to this:

eval( '3"])' )

Obviously, that doesn't work out.

Similarly, weird things will happen for anything spanning more than one line, e.g. a hash declaration, or a chained expression like the multiline lines.select {...}.each do .. end in your own code.

It does make sense that you picked expr # => result as your syntax since it's commonly used in documentation. But the problem is that... well, that it's commonly used in documentation. So you can't document your code without the test system misinterpreting it.

Your code isn't a full lexer/tokenizer, so it'll easily trip over all kinds of syntax. For instance:

str = "here's a string containing # -> but it's not even a comment"

In the end, you can only really test that, say, a constant is what it is (provided, of course, that it's a global constant) but that's not useful. Or you can test expressions that have no effect like 1 + 1. Expressions with no effect is something compilers/linters for other languages actually complain about, because they don't really make sense. It's considered a probable bug.

Sooo... yeah, I'm not too keen on it. Sorry.

Still, there's code to review.

Your regexes can be combined and simplified:

pattern = /^(.*)#\s*[=-]>(.*)$/

That'll match both -> and =>, and capture what comes before and after. Since captures are stored in the global $1, $2, $3 etc. variables, you can do stuff like:

if line =~ pattern
  expression = $1.strip
  expected = $2.strip
  # ...
end

Incidentally, expected is a better name than result. I'd think "result" would be the, well, the result of evaluating the expression.

Using captures also skips the whole split.last.sub.sub.etc. line.

That line is very buggy by the way:

  1. You split at every #, so a line like "#hashtag" # -> "#hashtag" will be parsed as " and hashtag" skipping everything in between. Hence the bug mentioned above, where it ends up eval'ing an empty string.

  2. strip strips leading/trailing whitespace - that includes newlines. So sub("\n", '') is unnecessary (besides, chomp would be more appropriate for removing a trailing newline).

You should also use inspect when printing values in your "instead got" messages. Otherwise, nil becomes just an empty string, strings lose their quotes, and symbols lose their colons making it hard to tell what the actual result was.

By the way, your heredocs should be indented.

Overall, if you want inline (or at least "in-file") testing, you should rather use something like minitest/autorun (already included in the stdlib of recent Ruby versions).

Or roll your own basic assert/assert_equal method:

class Object
  def assert(value, expected, message = nil)
    if value != expected
      fail(message || "Assertion failed")
    end
  end
end

assert(1 + 1, 2, "Math failed us! 1 + 1 should be 2")

Of course, I'd rather repeat my suggestion that you instead look at test/unit, minitest, RSpec or any of the other existing testing libraries/frameworks.

And of course there's the IRB console, for quickly trying things out. It just shows you the result, but you can always check it yourself, since the last result is stored in the _ variable:

$ irb
irb(main):001:0> "hello".reverse
=> "olleh"
irb(main):002:0> _ == "oleh"
=> false

More typing, but very useful for quickly trying things out.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.