Reid Burke

  • Debugging Travis builds

    Important Update December 19, 2013 — Travis infrastructure has changed since this post was written. This post remains unchanged for posterity; however, be aware that this setup no longer represents the current state of Travis.

    Yeti uses Travis CI to the max by spawning lots of PhantomJS processes that are tested asynchronously. When tests start failing in Travis, but not anywhere else, debugging can be infuriating.

    You can setup a Travis box locally to debug the problem. Trung Lê wrote about debugging with a local Travis VM in August 2012 but that post is outdated and didn’t work. You can expect sweeping changes to the Travis build infrastructure in the near future, so this blog post will also be outdated soon.

    For now, we have a bug that needs fixing. Let’s setup a local Travis VM for Node.js debugging.

    My assumptions about your computer

    1. Plenty of disk space
    2. OS X (but this should work with other systems, too)

    Install Vagrant & VirtualBox

    Visit the Vagrant website and the VirtualBox website for installers.

    Download the jvm box

    The jvm box is where Node.js builds run. Download travis-jvm.box. (3.2 GB)

    Prepare the jvm box

    Import the VM box:

    vagrant box add travis-jvm travis-jvm.box

    Verify it’s there:

    vagrant box list

    Start the jvm box

    Create a working directory to hold the Vagrantfile for the travis-jvm box.

    mkdir boxes
cd boxes

    Create the Vagrantfile and get the box ready to use:

    vagrant init travis-jvm

    The username for the VM is travis, so add this line to your Vagrantfile’s do-block:

    config.ssh.username = "travis"

    Start the VM:

    vagrant up

    Things should work nicely. (If not, read on.)

    You can access the VM with this command:

    vagrant ssh

    You can now follow the steps of your Travis build and debug your problem.

    If you have problems running vagrant ssh, you can try using a username and password for logging in instead with this command:

    vagrant ssh -p -- -l travis

    The password is travis.

    More fancy debugging

    The VM uses a NAT-only network. You may not be able to access it from your computer. For the problems I debug, I need to be able to access the the Node web server inside the VM from a browser on OS X.

    This means you should setup a host-only network interface, in addition to the NAT-only interface.

    Add this line to your Vagrantfile’s do-block:

    config.vm.network :hostonly, "192.168.89.10"

    This will setup a host-only network and give the IP address 192.168.89.10 to the VM. (If you are on a network that uses 192.168.89.xxx already, change the IP subnet to something else, because it cannot conflict with another subnet your host computer is using.)

    Reload the VM:

    vagrant reload

    You should be able to access the VM from the host at 192.168.89.10.

    Sometimes running vagrant up with a host-only network fails because of a bug in net-ssh and Vagrant with the message “Waiting for VM to boot. This can take a few minutes.” This bug is quite annoying, but a detailed workaround procedure that clears DHCP leases is available that’ll allow you to start the VM.

    Teardown

    You can pause the VM with vagrant suspend and bring it back to life with vagrant resume. If you’d like to power off the VM, do so over SSH. If you can’t, vagrant halt will power it off.

    If you want to start from a clean slate, you can destroy it with vagrant destroy. Re-creating a fresh VM is as easy as vagrant up.

    You can learn more about these commands by reading the Vagrant teardown documentation.

    Do not debug on travis-ci.org

    When you have a Travis-only problem, try creating a local environment as described above first. Don’t try to use the hosted Travis service to debug complicated failures. I have made this mistake many times:

    1. Noticed my build is failing in Travis.
    2. Make sure my own environments and clean installs do not fail. (The problem is sometimes here!)
    3. Create a branch named fix-travis-foo or something.
    4. Start making commits and pushes to this temporary branch attempting to fix the problem.
    5. Wait for the build to finish. If the build failed, repeat step 4.

    First of all, this takes a long time. Travis isn’t terribly slow for CI purposes, but when you’re trying to actively debug something, the last thing you need is having your app tests go from 15 seconds to 5 minutes.

    Most of all, my problems are usually with PhantomJS. When I can fully control the VM, I can add remotely debug the headless browser on OS X. I can also capture packets creatively with tcpdump(8). Using those tools, I found some pretty heinous race conditions in the Yeti test suite that would have been nearly impossible to find by trial-and-error.

  • Checkpoint

    When I travel home to Illinois, I frequently will work on Yeti for a few days and meetup with Dav. On this trip I’ve done both at the same time. I’ll meetup with him again tomorrow, hoping to finish up some test automation greatness before we take a break for Christmas.

    This time of year is really busy. I have lots of folks to shop for, lots of things on my personal to-do list, and in order to visit here I left a home full of mostly unpacked boxes. This week my employer needed me to list my accomplishments, which is difficult because I’m still trying to accomplish things, but the due date for this report was on my day off. Today.

    I had to have my family stop at a Starbucks off I-64 so I could submit what I typed up. When I returned, my sister Jessie wanted to type her own report.

    The Typist

    What I like about Jessie is that she reminds me of what’s possible when you don’t have lots to do.

    In this moment, she’s typing for my sake even though she prefers playing with other things. It’s more for me than her. But soon, she starts to enjoy the challenge of typing and I start to enjoy helping her find the words.

    When I begin my day focused on others instead of my own list of stuff, I’m typically more pleased with the result. That’s because instead of churning through what you think is important, you automatically have to do what someone else will value.

    Jessie does that because when chooses me over herself, it’s natural for her to give me something I value. Time. Patience. Kindness. As I can’t help but to do the same in return, it reminds me that sometimes my priorities are off.

    My challenge is to make moments like this more than a reminder. It’s a checkpoint. More breaks from your endless to-do list give you space to really change.

    We’re all not perfect. But these moments remind me of our potential. It’s a bit unsettling, but I don’t want to keep working harder at my own priorities if it means I’m ultimately not helping others.

    As this year draws to an end, reserve some time without your to-do list to remember what matters most.

  • Yeti at YUIConf

    Markandey Singh posted a short video of my YUIConf 2012 talk. Dav Glass is seen running around with a camera to show the audience YUI animation tests running with Yeti on various devices. Dav shipped 5 tablets, 1 phone, and an AirPort Extreme to California to make this demo happen.

    The Write Code That Works talk demonstrated Yeti in the context of software testing’s purpose. I also presented a few approaches for testing efficiently.

    After YUIConf, I landed a pull request to add Mocha, Jasmine, and full QUnit support to Yeti, making it more useful since this video. Thanks to Ryan Seddon for making that happen!

    The full session video will be available in the upcoming weeks. In the meantime, check out the slides or the Write Code That Works blog post which was the basis for the talk.

  • Write Code That Works

    Dav Glass and I visited the Yammer office in San Francisco this week to discuss build & test tools we use at YUI.

    We showed off Shifter for building YUI, Grover for testing headlessly, Yeti for testing in browsers, and Istanbul for statement-level code coverage. We use Travis for running most of this stuff in public CI. We now require 70% statement coverage before a new YUI module is allowed to land in the core and nobody can commit when the Travis or internal CI build is broken, unless the commit fixes the build.

    This is all very impressive. But @mde was quick to notice that we didn’t drop everything to get to this point—before diving in, you first need to prioritize what you work on. I couldn’t agree more.

    When you’re starting from scratch, you start to love the metrics. Green dots for passing builds. Green coverage reports when you hit 80% of your statements. The increasing number of passing tests. I’m all for having good code coverage, but before you go crazy, you should be careful that you don’t start writing tests for the wrong part of your code.

    Your code is not uniform

    Your code has various levels of quality starting at the first commit you make. You will write some code that’ll last for weeks or months, and some code that’ll need a rewrite next week. You need to embrace this kind of change and understand where it happens in your project.

    Node.js solves this problem quite well with the notion of a Stability Index.

    Throughout the documentation, you will see indications of a section’s stability. The Node.js API is still somewhat changing, and as it matures, certain parts are more reliable than others. Some are so proven, and so relied upon, that they are unlikely to ever change at all. Others are brand new and experimental, or known to be hazardous and in the process of being redesigned. The Stability Index ranges from Deprecated (0) and Experimental (1) for unstable code to Locked (5) for code that only changes for the most serious bugs.

    It’s a good idea for any post-1.0 project to assign a Stability Index to the APIs in your own code. Not only is it a clear message to those using your APIs, but it’s also a clear message for your team. It tells you where you should—and shouldn’t—write tests.

    More stable, more tests

    If you write tests like they cost nothing, you’re going to find yourself writing tests instead of writing code that works.

    Kent Beck’s wisdom says it best:

    I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it. This answer is good and I’ll take it a step further: You should prioritize writing tests for parts of your code with a higher Stability Index. Especially if you’re just starting on a new project.

    If you’re writing tests for code that’s rapidly changing, you’re going to spend more of your time writing tests instead of shipping features. For code that’s brand new, I typically only test-first a small amount of code and wait a while before hitting that green bar on a code coverage report.

    Don’t get sucked into the allure of metrics too early. Remember what your job is: writing code that works. Code coverage and good testing tools are very important, but not if they get in the way of building what you’re supposed to build.

  • Best Trumps Easy

    I work at Yahoo!, building open source software. I build Yeti, but I work alongside the team building YUI. The engineers who built this team, and continue to work on this team, are the reason I have chosen to stay at Yahoo! building Yeti: they choose what’s best over what’s easiest.

    You work with people too. You know your people are not perfect, but they’re going somewhere great, which is why you’ve decided to join them.

    I’m going to tell you that building software is very hard, but the most challenging part of my job isn’t building code. It’s building people.

    Your people are not a test framework or a programming language: they are human beings. Like your favorite code, they don’t always meet your expectations. They will let you down.

    People tell stories. What you do next will define the stories they tell.

    You probably rely on these people, so when they fail you, it’s going to affect you. A lot. You probably don’t deserve to be subjected to their actions.

    It’s easy to react.

    I’m telling you, the best people never settle on what’s easy.

    The best people never coddle or spin. They’re honest and speak their mind, but only after giving it 5 minutes. They think instead of react.

    The best people deliver criticism inwardly, one-on-one, to the person who needs it. The attitude is service and respect.

    The best people will embrace the opportunity to be a mentor instead of the opportunity to stand up for what they deserve.

    Next time your people don’t meet your expectations, I encourage you to see an opportunity to invest in people. It will be hard. It may take up a lot of your time and nobody but them will appreciate your investment. Yet serving others this way will reward you.

    You never know when you’ll need it yourself.

    It’ll also reward them. Honest feedback delivered this way is very desirable and mutually beneficial.

    If you want this kind of culture in your team or community, I’d encourage you to be the first one to start. Give more than they deserve, seek to understand what they care about, and after careful consideration, deliver your feedback to them personally.

    It’ll be the start of a conversation you won’t regret.