2
\$\begingroup\$

I'm unit testing some webpages and I'm trying to figure out how to best isolate the portions that need to be tested. There are two goals: Don't impose on the webpage's content or format, and be as clear as possible to the webpage editors what's going on, so they don't inadvertently break the tests.

I've come up with something that works, but I wondering if I'm reinventing the wheel, or if there's just a better/more elegant/more standard way of doing this.

Below is how I've documented it:


Take this (Django) html template:

<!DOCTYPE html>
<html lang="en">
   <head>
       <title>My user profile</title>
       <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
       <meta name="viewport" content="width=device-width"/>
   </head>
<body>
   <h1>My user profile</h1>

   <p>Back to <a href="{% url 'birth_year_stats' %}">user-stats</a></p>

   <ul>
       <li>Username: {{ user.username }}</li>
       <li>Email: {{ user.email }}</li>
       <li>Name: {{ user.first_name }} {{ user.last_name }}</li>
       <li>Year of birth: {{ user.profile.birth_year }}</li>
   </ul>
</body>
</html>

Each of these Django variables (such as {{ first_name }}) will be tested, by comparing the value as stored in the database with a test user:

test_user = {"username":   "kermit", "password":  "timrek",
             "first_name": "Kermit", "last_name": "The Frog",
             "birth_year": 1955,     "email":     "[email protected]"}

which leads to a problem: How do you determine which string in the rendered template...

<ul>
   <li>Username: kermit</li>
   <li>Email: [email protected]</li>
   <li>Name: Kermit The Frog</li>
   <li>Year of birth: 1955</li>
</ul>

...refers to which variable? While we could test each with something like this Python code:

page_content_bytes = client.get(reverse('user_profile')).content
match = re.search(r'<li>Email: (.+?)</li>', str(page_content_bytes))
self.assertIsNotNone(match)
self.assertEqual(test_user['email'], match.group(1))

This imposes specific content and formatting onto the webpage--<li>Email:...</li>, can never be altered without breaking the tests. In addition, there's nothing making it clear that this must never be touched, or the tests will fail. To solve this problem, I've come up with a specialized html comment that clearly isolates each testable portion. So this:

<ul>
   <li>Username: {{ user.username }}</li>
   <li>Email: {{ user.email }}</li>
   <li>Name: {{ user.first_name }} {{ user.last_name }}</li>
   <li>Year of birth: {{ user.profile.birth_year }}</li>
</ul>

Becomes this:

<ul>
   <li>Username: <!-- UNITRQD-start: username -->{{ user.username }}<!-- UNITRQD-end --></li>
   <li>Email: <!-- UNITRQD-start: email -->{{ user.email }}<!-- UNITRQD-end --></li>
   <li>Name: <!-- UNITRQD-start: first_name -->{{ user.first_name }}<!-- UNITRQD-end --> <!-- UNITRQD-start: last_name -->{{ user.last_name }}<!-- UNITRQD-end --></li>
   <li>Year of birth: <!-- UNITRQD-start: birth_year -->{{ user.profile.birth_year }}<!-- UNITRQD-end --></li>
</ul>

Which, when rendered, becomes this:

<ul>
   <li>Username: <!-- UNITRQD-start: username -->kermit<!-- UNITRQD-end --></li>
   <li>Email: <!-- UNITRQD-start: email -->[email protected]<!-- UNITRQD-end --></li>
   <li>Name: <!-- UNITRQD-start: first_name -->Kermit<!-- UNITRQD-end --> <!-- UNITRQD-start: last_name -->The Frog<!-- UNITRQD-end --></li>
   <li>Year of birth: <!-- UNITRQD-start: birth_year -->1955<!-- UNITRQD-end --></li>
</ul>

Now it's clear exactly what's being tested, to both the testing code and to whomever edits the web page. The comments are admittedly distracting, but as long as each block is preserved--both the variable-value and all the text-and-spacing in the comments themselves--each of these block may be moved around freely, without affecting either the template or the tests.

The above template therefore becomes the following:

<!--
   UNITRQD blocks

   These blocks contain text that is expected by the unit tests~~
   including the text inside the comments. Changing them will cause the
   tests to fail. If one must be changed, it must first be communicated
   to the appropriate persons, so the tests can be adjusted.

   The benefit of these blocks is to minimize the portions in this template
   that must not be changed, but to make it exactly clear which piece
   refers to which test. Note that only dynamic information is contained in
   these testing blocks. No hard-coded, static, *publicly displayed* text
   or formatting is required by the tests.

   As long as each individual block is left untouched, they may be moved
   around freely.
 -->
<!DOCTYPE html>
<html lang="en">
   <head>
       <title>My user profile</title>
       <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
       <meta name="viewport" content="width=device-width"/>
   </head>
<body>
   <!-- UNITRQD: user-profile page -->
   <h1>My user profile</h1>

   <p>Back to <a href="{% url 'birth_year_stats' %}">user-stats</a></p>

   <ul>
       <li>Username: <!-- UNITRQD-start: username -->{{ user.username }}<!-- UNITRQD-end --></li>
       <li>Email: <!-- UNITRQD-start: email -->{{ user.email }}<!-- UNITRQD-end --></li>
       <li>Name: <!-- UNITRQD-start: first_name -->{{ user.first_name }}<!-- UNITRQD-end --> <!-- UNITRQD-start: last_name -->{{ user.last_name }}<!-- UNITRQD-end --></li>
       <li>Year of birth: <!-- UNITRQD-start: birth_year -->{{ user.profile.birth_year }}<!-- UNITRQD-end --></li>
   </ul>
</body>
</html>

And now you can test each field with, for example:

"""  
   ': email -->(.+?)<!'

   Captures all text between ': email -->', and the 
   IMMEDIATELY-FOLLOWING '<!'. The question-mark in '.+?' is 
   'reluctant'. Eliminating it would capture all text through the *final*
   '<!' in the document.
"""
page_content_bytes = client.get(reverse('user_profile')).content
match = re.search(r': email -->(.+?)<!',
                str(page_content_bytes))
self.assertIsNotNone(match)
self.assertEqual(test_user[email], match.group(1))
\$\endgroup\$
2
  • \$\begingroup\$ This is something of a nitpick, but almost by definition, you can't unit test a web page. A unit test works on an individual unit of code, and a web page is not an individual unit. You can unit test a function called by a web page but not the web page itself. This is more like component interface testing where you are trying to test one aspect of the final result. \$\endgroup\$
    – Brythan
    Oct 16, 2014 at 23:01
  • \$\begingroup\$ I should've just said "testing". This is functional, UI testing...confirming the expected information makes it to the user. \$\endgroup\$ Oct 16, 2014 at 23:47

1 Answer 1

2
\$\begingroup\$

It's an interesting solution, and an admirable effort. Unfortunately, this is over-engineering, it increases the complexity of your project, and may also weaken your security.

The biggest problem with including markers like <!-- UNITRQD-start: username --> in the code is that it leaks internal implementation details on the user interface. This example might not reveal a lot: it's kind of obvious that you would have a table with a field "username". But as the practice spreads in the project, revealing more and more intimate details about your design, attackers might be able to piece together enough information to help them hack your site.

You can work around this issue by implementing deployment scripts that wipe out these markers when released in production. But that's non-negligible extra work, and adds to the overall complexity of your project.

This imposes specific content and formatting onto the webpage

First of all, the ultimate target of your testing is not 100% clear. What are you really trying to test here? It seems you want to make sure that your elements like email are between specific invisible markers. But is this is a good target for testing? It sounds a bit analogous to micro-managing. I'm not sure what these tests will really achieve. Have you considered the possibility of going too far? Isn't there a better target for testing?

<li>Email:...</li>, can never be altered without breaking the tests.

Well, what's wrong with breaking tests? If the code evolves, it's normal that tests have to evolve with it. Imagine that your project has 100% coverage: all execution paths are verified by unit tests. At that point you cannot change anything without breaking tests.

In addition, there's nothing making it clear that this must never be touched, or the tests will fail.

In the code I write tests for, I don't normally add a friendly comment saying "hey, there are some unit tests for this code, check it out". So it seems normal to me that you never know that by touching something some tests might break. You know that afterwards, when running the tests.

Conclusions

  • It's ok for tests to break. If you find that you spend too much time fixing tests, then maybe you're testing the wrong things. Take a step back, and find better targets to test, better identifying characteristics. If you want to test that nobody removed {{ user.username }} from the template, write a test for exactly that, on the template file itself before rendering

  • Normally there are no signs attached to code saying "beware, this stuff is unit tested". Instead, after an editor is done editing, they should run the test suites before they commit. Even better: make it automated in pre-commit hooks, or in a centralized continuous integration system. It's ok that a CI will catch errors a bit late, as long as you enforce a policy of cleaning up failed builds ASAP.

  • For a software that reports errors, in general, it is recommended to be careful what you include in error messages, to avoid revealing too much. The code under review doesn't report errors, but I think we can extend that concept to testing techniques as well. It's probably best to avoid revealing anything about the internal design.

  • It is a recommended practice to not reveal internal implementation details in error messages, and I think your technique violates that principle, therefore it should be avoided.

  • Keep your testing simple. (Keep everything simple, in general.) I think you're clever technique can cause more problems than it solves. Remember Occam's Razor: simple solutions tend to be the best. This technique in question is not exactly simple.

\$\endgroup\$
8
  • \$\begingroup\$ Lots to think about. First, I get the security issue, and perhaps any hidden markers in the html are a mistake, just as a matter of policy. That said, while my example website is only a demo in a tutorial (and therefore what I'm testing for may be overkill in a real world project), it's still a valid concept: how do you test the critical portions of a dynamically-generated webpage? \$\endgroup\$ Oct 16, 2014 at 22:22
  • \$\begingroup\$ As far as "so let the tests break". If my code causes the tests to fail, then of course, because that's just part of the development process. But if a non-programmer webpage-editor breaks the tests only because they've changed some content or formatting, I consider that a very bad thing, with little exception. On the other hand, if they obliterate or mangle a variable in the template (such as changing {{ email }} to {{ em ail }}), then that should break the tests, because they're touching stuff they shouldn't. \$\endgroup\$ Oct 16, 2014 at 22:23
  • \$\begingroup\$ What I can do is, first, to just consider the variables themselves as untouchable (educate the webpage-editors of such), as opposed to the variable in a marker-comment (<!-- UNITRQD-start: email -->{{ email }}<!-- UNITRQD-end -->). Then, instead of scraping the variable-text (as exists between the markers), and then ensuring it equals value X, I could just search the page for X, with word-boundaries or whatever. \$\endgroup\$ Oct 16, 2014 at 22:24
  • \$\begingroup\$ I don't understand you point about error messages. These are unit tests. They fail or don't fail outside of the user's world. \$\endgroup\$ Oct 16, 2014 at 22:24
  • \$\begingroup\$ For a software that reports errors, in general, it is a well-known recommendation to be careful what you include in the error message, to not reveal too much. I used this as a supporting argument, to extend it to other techniques that may divulge about the internal design. Of course there's no error reporting in the code under review. I will reword that part to make this point clearer. \$\endgroup\$
    – janos
    Oct 16, 2014 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.