Sauce Labs presents

Cross-BrowserTesting with Yeti

DeveloperWeek 2014

Reid Burke@reid

Testing at Yahoo

YUI

We like automated testing

Headless

PhantomJS, Node.js

Your audience does not use PhantomJS

You need real browsers

Real browsers

IE 6, 7, 8, 9, 10, & 11, Chrome, Firefox Safari, iOS

Automating them is hard

Sauce Labs

Sauce Labs ♥ open source

Yeti makes testing easy

Automates tests written with QUnit, Jasmine, Mocha, Dojo, or YUI Test

No Selenium required on your computer

Works with Jenkins & Selenium

New: built-in code coverage

Open source

yeti.cx

Hands-on with Yeti

yeti.cx

How to spend your time

You can test everywhere

But not everything

Testing is a tool

Tested code ≠ working code

You are not paid to test

You are paid to write code that works

How deep are your unit tests?

Kent Beck on Stack Overflow

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence
If I don't typically make a kind of mistake (like setting the wrong variables in a constructor), I don't test for it.
When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.

Kent Beck, known for advancing Test-Driven Development

Write code that works

Testing shows the presence, not the absence of bugs

— Edsger W. Dijkstra, NATO Software Engineering Conference 1969 (p. 16)

Avoid over-testing

U.S. Department of Homeland Security

Security checkpoint at an airport

Testing like the TSA

Automate testing on every push

Another tool

Continuous Integration

Clean environment for your tests

Keeps you & your contributors honest

Joy of a single CI job

Broken Travis build email

Broken Travis build

Joy of a single environment

Multiple environments, multiple builds

Way more noise

More builds to check

More emails from CI

You make web apps

How many environments do you have?

Start small

Use tools like Yeti, Travis, Sauce Labs to prevent draining yourself on infrastrcture

PhantomJS

Joy of a single environment

Testing at Yahoo scale

Yeti helps us

AJAX testing with echoecho

Browser launching

Parallel testing

Code coverage

Yeti does parallel testing

Per build, we do...

3 parallel instances of browsers

Parallel builds, too

100K+ tests per push

Over 3 hours of testing in 90 minutes

Our audience does not use PhantomJS

We need real browsers

Provided by Sauce Labs, one less thing to worry about

Setting up automated browsers is hard

Handling test results from multiple environments is harder

More testing, more problems

Impossible workflow

Failing build email

Failing build email console log

Unstable build email

Unstable Jenkins build

Unstable Jenkins tests

Unstable Jenkins unit tests

Unstable Jenkins unit test detail

Repeat for a dozen builds

Use the dashboard?

Did my latest commit pass? Did code break the build? Or is it flaky?

Confusing CI dashboard

Flaky infrastructure

21 jobs x 3 branches = 63 jobs

~3 VMs x 63 jobs = 189 browser VMs

~189 browser VMs + 63 slaves = ~252 machines

Over 252 moving parts

Flaky tests

Build numbers instead of commits

Build #581 was for which commit again?

yui-dev_3_x-selleck-selastic-ie-576

yui-dev_3_x-selleck-selastic-ie-archaic-581

Nobody responds to build failures

Too hard to understand what needs fixing

Understanding what needs fixing

yo/tests

Classify flaky tests

yo/tests understands what tests are flaky

Hide flaky test results

Do not alert developers to flaky test failures

Prevents panic from bad tests

Hide flaky infrastructure

Engineers do not care if a CI slave is down

Prevents panic from bad infrastructure

Commits, not build numbers

As builds complete, they are organized around commits

Developers know exactly what code broke

Unstable build email

Red means code broke? Bad infra? What commit was this?

Confusing CI dashboard

yo/tests

yo/tests branch overview

yo/tests commit detail

yo/tests recent builds

yo/tests commit unstable tests

Shipping quality YUI releases

release-3.13.0

yo/tests release overview

yo/tests release rows

yo/tests builds

Highlight to take action

IE 60 failing tests

4 failing tests

0failing tests

Did my commit pass?

yo/tests branch overview

yo/tests commit detail

How we keep sane

Classify flaky tests

Hide flaky test or infra failures

Commits, not build numbers

New to testing?

Start with Yeti.

Need browsers?

Use Sauce Labs.

More testing, more complex

Make tools to help you understand complex results

Write code that works

Do not test like the TSA

rburke@yahoo-inc.com

We are not done

Follow @reid