Build and test infrastructure update

[kum-deepak]

No description provided.

[gordonwoodhull]

Specifically, phantomjs seems to be dead, but both Firefox and Chrome can be run headless, and I believe there should be npm packages to allow our Jasmine tests to run on either one.

For ideological reasons, and also because I believe it is more standards-compliant, I'd prefer to use Firefox. But Chrome would also be great, and may have a stronger community supporting automated testing. I have not done significant research and would be happy to go either way.

[kum-deepak]

I have been using Chrome for automated test cases on Travis. That project uses Angular, which comes preconfigured with Karma. Test cases use Jasmine.

So, I think it will be possible to use Karma with Jasmine. That will give us flexibility in using any of the supported browsers.

[kum-deepak]

Karma browser support: http://karma-runner.github.io/0.12/config/browsers.html

[gordonwoodhull]

That's impressive browser support, and I think I've heard good things about Karma. I don't quite understand how it works though.

Can it be configured to run as part of grunt as we're doing now? Will it work on Travis? It seems to be different from the headless testing we are doing now; maybe it's at a different level in the technology stack?

[kum-deepak]

Karma takes a very interesting approach, browser is actually launched by Karma (in default configuration) and once the browser is launched it runs the test case (Jasmine). Karma itself does not provide testing framework (i.e., it is not a replacement of Jasmine), it however allows a good way for nodejs to integrate and remote-control a browser.

During running test cases console output from the browser gets merged into the stdout of the shell. I like this part of getting the console output easily.

Karma is invoked from command line so it should integrate well with grunt.

It does work with Travis. There are travis setup guides for Karma with Firefox/Chrome. Basically the browser is added in the the apt packages needed. xvfb needs to be started before firing off the tests.

I am using it with Travis now https://github.com/stomp-js/ng2-stompjs/blob/master/.travis.yml

[gordonwoodhull]

So it eliminates installing a headless browser as a Node dependency, and instead uses whatever browsers are available on the system? Very cool.

Sounds like we could maybe have a grunt task that tests on all browsers available locally on the dev machine. That would help a lot.

Console output would be excellent too. Have you encountered anything that actually tests console output? We have a few situations where dc.js can produce invalid but harmless SVG producing hundreds of warnings, and it would be nice to be able to assert this shouldn't happen.

[kum-deepak]

I think testing on multiple browsers should be possible.

On the console output, I have not tried myself, but we should be able to stub/spy console. I will try when I am working on this task.

[kum-deepak]

Please check #1393

[Justus-Maier]

Since you are already integrating Angular's Unit Test Runner called Karma, how about also adding end2end tests with protractor? This might be the right integration test for user interaction or complex crossfilter brushes ;)
Thumbs up for tests any way 👍

[gordonwoodhull]

Hmm, protractor looks nice - I wonder if it works without Angular? As we don't use it. Looks like Karma came from testing Angular but isn't dependent.

We do quite a bit of interaction testing using straight D3, but it's really hard. TBH I'm not sure if this is a tooling issue or if testing interactive charts is inherently difficult.

[kum-deepak]

I do not think we need Protractor for testing out interactions. Protractor allows loading web pages one after the other during the test scenario, which is not what we need. If we do not need to load a new page (or change the view - as in Angular), we should be just fine with Jasmine.

The issue is not actually tooling. We can do complex test cases using Jasmine. Writing those tests are inherently difficult and fragile.

There are some test cases for zoom in dc that send simulated mouse clicks. This approach - with lots of patience - can send and simulate mouse drags, clicks etc.

If there is sufficient interest, we can build the entire stocks demo as test bed and test interactions using Jasmine.

[gordonwoodhull]

Karma is working great for us. I don't think we need any further upgrades for now. Thanks @kum-deepak!