Code Coverage in Node.js

Quick Intro

• @ Microsoft for 3 years (OS/Kernel development)
• @ Splunk since summer (DevPlatform)
• From Israel

I needed a way to get code coverage for the Splunk JavaScript SDK

Nothing quite fit what I needed, so I wrote Cover

A bit of history

JSCoverage

• Written in C
• Only line-based coverage (single statement per line)
• Requires modifying your code

Bunker

• By substack
• Pure JS using uglify-js
• Raw output (only JSON)
• A bit slow
• Had some parsing bugs
• Not trivial to use

runforcover

• By Chris Dickinson
• Pure JS, based on bunker
• Hooked into require
• Only console output
• Only line-level coverage

Cover

• Originally a fork of runforcover
• Significantly changed
• Based on the awesome Esprima JS parser
• Hooks into require
• Console, JSON and HTML outputs
• Standalone program
  (e.g. cover run nodeunit)
• Line- and block-based coverage

How does it work?

• Hook into require
• Parse, instrument and re-generate each JS file
• Collect generated statistics
• Print report

Hooking into require

• Use require.extensions['.js']
• Check to see if file should be ignored (i.e. not covered)
• Run it through instrumentation code
• Return transformed code to Node.js

require.extensions['.js'] = function(module, filename) {
    
    // Check if file is ignored
    ...

    // Read from disk
    var data = stripBOM(fs.readFileSync(filename, 'utf8').trim());
    data = data.replace(/^\#\!.*/, '');
    
    // Instrument
    var instrumented = instrument(data);    
    var newCode = addInstrumentationHeader(template, filename, instrumented, pathToCoverageStore);
    
    ...
    
    // Return to Node.js
    return module._compile(newCode, filename);
};    
  

Parse, instrument and regenerate code

Statement transformation

• This:

  var a = 5;

• Becomes:

  
  {
      __statement_ZyyqFc(1);
      var a = 5;
  }
    

Expression transformation

• This:

  foo();

• Becomes:

  
  {
      __statement_ZyyqFc(2);
      __expression_kC$jur(3), foo();
  }

Block transformation

• This:

  
  function foo() {
      ...
  };

• Becomes:

  
  function foo() {
      __block_n53cJc(1);
      ...
  }

All together now

• This:

  
  if (foo()) 
      var c = 5;

• Becomes:

  
  if (__expression_YgAcZn(8), foo()) {
      __block_n53cJc(2);
      {
          __statement_LpWHVj(9);
          var c = 5;
      }
  }

How instrumenting works

• Get AST from Esprima
• Traverse AST recursively, applying above transformations
• Can see all of this in instrument.js
• About 400 lines of meaningful code

That's pretty much it

There's a bit more...

• Engineering to get command-line tool to work
• A lot of futzing around with AST to get HTML report
• But the interesting parts are the require hook and instrumentation

Some thoughts...

100% code coverage doesn't mean your code is 100% correct

The trouble is that high coverage numbers are too easy to reach with low quality testing.

Martin Fowler

Coverage can be useful for detecting areas where you need more testing

Block-level coverage is a much more useful metric to track

Next steps

Make it run in the browser

• Esprima can run in all modern browser
• Would need to change how the "hook" works, but not a big deal
• Abstract away Node.js specific parts.

Get it ready for wider release

• Already used in the wild by Esprima, Joyent and others
• However, it is missing more docs...
• And more tests...
• Probably need to run Cover on Cover (INCEPTION!)
• Hook into Travis-CI

How you can help

• Happily looking for more contributors
• Help write docs/tests. Great way to get involved
• It's a fairly small code-base
• Happy to walk people through all the code

Most importantly... try it out!

npm install cover -g

Questions!