POSTS

Stabilizing Tests for Karma Coverage

Loading the canvas libs

Now that Karma is all wired up, I can start poking at the tests.

The first problem I need to fix involves canvas-libs. knobsAndLevers uses the supporting object in canvas-libs to set a randomized initial spawn interval for the creatures. The usage is:

this.spider.initialInterval = supporting.getRandom(this.spider.interval.min, this.spider.interval.max);

But knobsAndLevers.init() fails, because supporting is not available:

$ node_modules/.bin/karma start my.conf.js
17 03 2018 07:54:43.213:INFO [karma]: Karma v2.0.0 server started at http://0.0.0.0:9876/
17 03 2018 07:54:43.216:INFO [launcher]: Launching browser Chrome with unlimited concurrency
17 03 2018 07:54:43.221:INFO [launcher]: Starting browser Chrome
17 03 2018 07:54:43.637:INFO [Chrome 55.0.2883 (Linux 0.0.0)]: Connected on socket XC_wBQ5rcv01IYm7AAAA with id 45373
...
Chrome 55.0.2883 (Linux 0.0.0) Testing knobs and levers generation gamePieceTopLimit should not be undefined FAILED
        ReferenceError: supporting is not defined
            at <Jasmine>
            at Object.init (app/scripts/knobs-and-levers.js:16:35)
            at UserContext.<anonymous> (test/spec/knobs-and-levers-spec.js:4:13)
            at <Jasmine>
...
Chrome 55.0.2883 (Linux 0.0.0): Executed 35 of 38 (33 FAILED) ERROR (0.033 secs / 0 secs)

I could include canvas-libs in my dependencies somehow. First I’d have to explore publishing npm packages, but for now, (partly because I don’t want to inflict my library on the world) I can just pass the local path to canvas-libs through my.conf.js files list:

files: [
  ...
  '../canvas-libs/app/scripts/*.js',
  ...
]

After adding canvas-libs to the file list, the results are:

Chrome 55.0.2883 (Linux 0.0.0): Executed 5 of 38 (2 FAILED) (skipped 1) ERROR (0.003 secs / 0 secs)

Only 2 failed! But I don’t know whether more would have failed had the ERROR not occurred.

I don’t really understand why it only shows 38 tests, since there are 59 tests according to the spec-runner.html execution. Why is it only of 38, and why did this last one only execute 5 tests? It’s likely crapping out because of ERROR, meaning there’s a compilation problem with my code, but where are the missing tests?

Here’s the error:

Chrome 55.0.2883 (Linux 0.0.0) ERROR
  {
    "message": "An error was thrown in afterAll\nReferenceError: Sound is not defined\n    at <Jasmine>\n    at Suite.describe (test/spec/sound-spec.js:4:19)\n    at <Jasmine>\n    at test/spec/sound-spec.js:2:1",
    "str": "An error was thrown in afterAll\nReferenceError: Sound is not defined\n    at <Jasmine>\n    at Suite.describe (test/spec/sound-spec.js:4:19)\n    at <Jasmine>\n    at test/spec/sound-spec.js:2:1"
  }

This is odd, since Sound is defined in canvas-libs, which should have been included. Maybe the inclusion order in the Karma files list matters (ugh).

It also appears the sequence of tests is randomized (that’s a good thing), since the error occurs at different times in the sequence if run multiple times. I run it again and get things like:


Chrome 55.0.2883 (Linux 0.0.0): Executed 37 of 38 (skipped 1) ERROR (0.037 secs / 0 secs)
...
Chrome 55.0.2883 (Linux 0.0.0): Executed 0 of 38 ERROR (0.002 secs / 0 secs)

It’s all over the board.

The order of the files entries in the karma config does appear to matter. If I put canvas-libs first, I no longer get the Sound missing error, and consistently get this result:

Chrome 55.0.2883 (Linux 0.0.0): Executed 38 of 39 (2 FAILED) (skipped 1) ERROR (0.042 secs / 0 secs)

Progress!

Fixing the failing test

canvas-libs seems to be loading fine now, and I just have one more error and one more failure:

Chrome 55.0.2883 (Linux 0.0.0) Testing game-area functions getYVertices returns a reasonable array of vertices FAILED
        Expected 0 to be 17.
            at <Jasmine>
            at UserContext.it (test/spec/game-area-spec.js:27:30)
            at <Jasmine>
Chrome 55.0.2883 (Linux 0.0.0) ERROR
  {
    "message": "An error was thrown in afterAll\nUncaught ReferenceError: knobsAndLevers is not defined",
    "str": "An error was thrown in afterAll\nUncaught ReferenceError: knobsAndLevers is not defined"
  }
Chrome 55.0.2883 (Linux 0.0.0): Executed 38 of 39 (1 FAILED) (skipped 1) ERROR (0.037 secs / 0 secs)

I’m not certain what afterAll is, but it’s probably a cleanup script run by Jasmine when all tests are complete.

This error was caused by the this. reference on knobsAndLevers, meaning the wrong instance of knobsAndLevers was being accessed. xVertices and yVertices tests both failed because of this.

Aha! The discrepancy between browser Jasmine tests and Karma Jasmine tests (59 vs 39) was because I have to explicitly include subdirectories. I started seeing 59 after I added this to the files list: test/spec/components/*.js. This could also be why I’m getting these weird errors: not all of my source .js files are being included. So I added app/source/components/*.js as well.

Fixing the execution ERROR

The ERROR is caused by load order.

Chrome 55.0.2883 (Linux 0.0.0) ERROR
  {
    "message": "An error was thrown in afterAll\nUncaught ReferenceError: knobsAndLevers is not defined",
    "str": "An error was thrown in afterAll\nUncaught ReferenceError: knobsAndLevers is not defined"
  }

What’s really frustrating is Karma doesn’t give any indication where the knobsAndLevers are being accessed first, and I tried a lot of different things to see if the problem was with my variable names in the tests (renamed them all) or specific tests (xit’d them all) before hitting on the load order issue.

Now that it’s loading knobs-and-levers first, then the rest of scripts/*.js (which also loads knobs-and-levers again), I get an error on /components/metrics.js, which means it got through the rest of the scripts at the root of the scripts dir, and centipede, display-object-prototype, interval-creatures, and lasers inside components.

But in the files list in the Karma config, I just have: app/scripts/*.js, and it’s probably loading init.js right after hud.js. This became obvious when I loaded knobs-and-levers.js explicitly before loading the rest of the files. So this tells me either I need to separate critical dependencies and load them first through a subdir, or I need to put init.js in a subdir and load it last. The first probably makes more sense long term, but the second is how I have it working now. I’ll keep the reorg on a back burner and just isolate the init.js loading, controlling the load order within that. This should limit the troubleshooting to the order of execution in that file alone.

Moving init.js to a subdir and loading it last also solved the cryptic knobsAndLevers issue, so I no longer need to initialize it first.

The reason these errors don’t show up in normal execution even though I also have the scripts defined in alphabetical order in spec-runner.html, is because in spec-runner tests, I never actually load init.js. Instead, I just initialize things in the beforeEach sections as they are needed. I can probably forego loading the init dir altogether, but it will be good to run it to see if there are any runtime errors.

Coverage time!

I think we’re (finally) at a point where the Karma configuration is stable and we can start analyzing the code!