Testing YUI Everywhere

Reid Burke

YUIConf 2013

Testing is useful

You are not paid to write tests

Possible to test all environments

YUI testing in 2012

Single CI job

Did not use Yeti

Did not test AJAX

One browser at a time

Only unit tests, no functional tests

Only Node.js, IE 7, Firefox 3.6 on every commit

Test Fests before releases

Test Fests were day-long manual testing sessions

Test Fests are evil

We had to automate them

Joy of a single CI job

Broken Travis build email

Broken Travis build

YUI outgrew a single CI job

Introducing

YUI testing in 2013

Way better than last year

YUI tested by CI slave

PhantomJS, Node.js

YUI tested by Selenium

IE 6, IE 7, IE 8, IE 9, IE 10, Chrome, Firefox

YUI tested by Sauce Labs

Safari, iOS

Testing powered by Yeti

AJAX testing with echoecho

Browser launching

Parallel testing

Functional testing

11 of 15 tested on every commit

yuilibrary.com/yui/environments/

4 of 15 tested before each release

Automated but tested outside of CI

100% automated

More speed powered by Yeti

8 parallel instances of IE, Firefox & Chrome

3 parallel instances of Safari & iOS

Parallel builds, too

More environments

YUI tested by CI slave

PhantomJS, Node.js

YUI tested by Selenium

IE 6, IE 7, IE 8, IE 9, IE 10, Chrome, Firefox

YUI tested by Sauce Labs

Safari, iOS

100K+ tests per commit

Over 3 hours of testing in 90 minutes

More complex

21 jobs x 3 branches = 63 jobs

~3 VMs x 63 jobs = 189 browser VMs

~189 browser VMs + 63 slaves = ~252 machines

Over 252 moving parts

Testing became harder

Flaky, too many jobs & too many test results

Impossible workflow

Failing build email

Failing build email console log

Unstable build email

Unstable Jenkins build

Unstable Jenkins tests

Unstable Jenkins unit tests

Unstable Jenkins unit test detail

Repeat for a dozen builds

Use the dashboard?

Did my latest commit pass? Did code break the build? Or is it flaky?

Confusing CI dashboard

Flaky infrastructure

Sauce Labs, Jenkins, build slaves, Selenium, Yeti server, what will fail first?

Flaky tests

98% of the time, it works every time

Build numbers instead of commits

Build #581 was for which commit again?

yui-dev_3_x-selleck-selastic-ie-576

yui-dev_3_x-selleck-selastic-ie-archaic-581

Too many results

Jenkins becomes slow

Nobody responds to build failures

Too hard to understand what needs fixing

Our solution: yo/tests

Classify flaky tests

yo/tests understands what tests are flaky

Hide flaky test results

Do not alert developers to flaky test failures

Prevents panic from bad tests

Hide flaky infrastructure

Engineers do not care if a CI slave is down

Prevents panic from bad infrastructure

Commits, not build numbers

As builds complete, they are organized around commits

Developers know exactly what code broke

Unstable build email

Red means code broke? Bad infra? What commit was this?

Confusing CI dashboard

Now, with yo/tests

yo/tests branch overview

yo/tests commit detail

yo/tests recent builds

yo/tests commit unstable tests

Shipping quality YUI releases

release-3.13.0

yo/tests release overview

yo/tests release rows

yo/tests builds

Highlight to take action

IE 60 failing tests

4 failing tests

0failing tests

Did my commit pass?

yo/tests branch overview

yo/tests commit detail

Wins

Prevent panic from bad tests

Prevent panic from bad infrastructure

Developers know exactly what code broke

How To Win

Classify flaky tests

Hide flaky test or infra failures

Commits, not build numbers

We are not done

This is an HTML presentation

Made with YUI, Shifter, Upstage, and Jade