Testing and QA that doesn't suck (so your app won't)

The problem/challenge of proper interface QA

We all know we need to test. While in the building phase of a browser app I find that building an extensive test suite isn't necessarily worth the effort. Interfaces are inherently hard to test in that the variability of human input is part of what makes a good test for an interface.

Also, there are things that are just a bit hard to test. For example, if you support drag and drop actions in your interface it's not so easy to write a test that properly validates that it works well.

Then there's the "problem" of CSS changes. The DOM can be in perfect order but if the styles are off things can look quite broken, or you can have a layout issue where a control is covered or unclickable.

Some people build really elaborate QA systems that load their app into a headless browser and takes screenshots that are compared against reference images, etc. But the amount of effort and setup required for that is simply not practical in most cases.

There are also tools like Selenium that will script a browser for you, but it's a whole lot of work and setup, and then every time you want to change something, if your tests are too specific they'll need to be constantly updated. And if they're too general they'll miss stuff.

While headless browser testing is a really cool idea (PhantomJS, etc.), it doesn't really help you know how your app works in other browsers.

Ultimately, I don't believe you can actually do proper testing of an interface without a human.

So, there must be a balance that can be struck between human approval and oversight and taking advantage of the things computers are good at like process, consistency, and automation.

Meet the SpaceMonkey

So we've been building one, we're calling it SpaceMonkey. There wasn't anything out there that did this in a way that was browser-agnostic, simple, and blended automation with human approval.

We've still got a lot of work to do on it to make it awesome, but it's included in the app generator if you select the "express" option, and it seemed worth mentioning nonetheless.

It uses QUnit, the clientside testing tool made by the jQuery team. But rather than run simple programmatic tests. It loads your app into an <iframe> and walks you through the QA process that you've defined for it. You can automate high-level tests like filling in and submitting forms, etc. But, in addition, at the times you've specified it will stop and ask the user for visual verification or to perform some user action such as a drag-and-drop interactions or scrolling-related features that may be hard to test programatically. So some of the unit tests are considered a pass or fail based on a human's answer to "did it work?"

This approach makes it possible to define an explicit set of app interactions to test. So it's like a partially automated QA checklist.

Testing in other browsers will just involve running through the test sequence in other those other browsers. Using a service like Browserling or BrowserStack lets us walk through this process in whatever browsers we're targeting.

SpaceMonkey is still under heavy development, but a clienttests folder and sample QA workflow is included in the app generated by the humanjs app generator.

You end up writing tests that look like this:

// Load the app into our iframe
monkey.loadApp('/', {
    height: 500,
    width: 600,
    bugUrl: 'https://github.com/henrikjoreteg/humanjs-sample-app/issues/new'
});

// This is a normal QUnit test
test('basic app experience', function () {
    // Here we start a chain of interactions
    monkey
        // We can log out messages to the console
        .log('starting')
        // Wait for things to appear
        .waitForVisible('#pages .page')
        // Ask for user confirmation
        .confirm('The app loaded to the home page.')
        .confirm('The "home" nav tab is active')
        // Navigate to different urls
        .goToPage('/collections')
        .confirm('Collection demo page visible')
        .confirm('List of people are visible each with avatars')
        // We can also provide a set of instructions of something
        // that is about to happen. And instruct the user to look
        // for specific behaviors.
        .instruct('Five users will be added. Please ensure.', [
            'each one is added at the bottom of the list',
            'each has a readable name an avatar'
        ])
        // Simulate clicks
        .click('button.add')
        .click('button.add')
        .click('button.add')
        .click('button.add')
        .click('button.add')
        .confirm('Everything look ok?')
        .confirm('I can visually re-arrange them by pressing, "shuffle"')
        .confirm('I can hit reset and they disappear, and fetch and they come back.')
        .confirm('I can delete them by clicking "delete"')
        .goToPage('/info')
        .confirm('Info page is visible')
        // Call this when done
        .destroy();
});

To see it in action:

1. Install the app generator using npm: npm i humanjs -g
2. Run the generator and answer its questions: humanjs
3. Run the generated app and visit http://localhost:3000/test in a browser

As I mentioned, we still have a lot of work to do in making SpaceMonkey a well-structured standalone library. But it seemed useful and functional enough to include.

And if nothing else, the general idea might be enough to tempt you to build it :)

Unit testing modules that require a browser

For tests that are not QA, but are in fact unit tests, you need a way to write tests that will run in a browser and ideally, we'd be able to run them from the command line.

Turns out, we can. Using a set of tools developed largely by James Halliday, a.k.a "substack".

Here's a sample from ampersand-state:

var test = require('tape');
var State = require('../ampersand-state');

var Person = State.extend({
    props: {
        name: 'string'
    }
});

test('init with values', function (t) {
    var person = new Person({name: 'henrik'});
    t.ok(person);
    t.equal(person.name, 'henrik');
    t.end();
});

test('after init, change should be empty until a set op', function (t) {
    var person = new Person({name: 'phil'});
    t.deepEqual(person._changed, {});
    t.notOk(person.changedAttributes());
    t.end();
});

The "tape" module is capable of running in the browser or the server with browserify.

So, with the help of a few handy dev dependencies, we can run our tests in phantom.js, or in a browser.

First, run the following to install a few dependencies we'll need:

npm i --save-dev run-browser browserify tape-run

This will install them and save them as devDependencies in your package.json.

Next, we can add some scripts to our package.json like this:

  ...
  "scripts": {
    "start": "run-browser test/index.js",
    "test": "browserify test/index.js | tape-run"
  },
  ...

With this setup we can run npm start and open localhost:3000 in whatever browser we want to run the tests in.

We can also just run npm test and it will run it with phantom.js (which you may have to go install separately if you don't already have it) and you should see the same type of output in your terminal.

This output format is a standard test output format called TAP.

As a result, there are various tools to dress up that output. So, if you want prettier terminal output, you can also install tap-spec from npm and update your "test" script to look like this:

"test": "browserify test/index.js | tape-run | tap-spec"

By using tape and writing tests in this way, we can also use a service like testling that runs our tests in different browsers for us.

Testling can be configured to automatically run our test suite using the list of browsers that we specify each time we push to github. This approach is extremely helpful when developing small modules for use in a browser. See the testling quick start guide for more info on how to set this up for your project.

For a full working example of these tools being used, see ampersand-view or almost any of the ampersand.js tools.