Stream Concat Anti-Pattern

NodeConf 2013 was great. You should go. I learned a lot of new stuff. Yesterday, I applied things I learned there for the first time.

I often write code that looks like something like this:

var stdout = "",
    child = cp.spawn(process.execPath, args);

child.stdout.setEncoding("utf8");
child.stdout.on("data", function (chunk) {
    stdout += chunk;
});
child.on("close", function () {
    cb(null, stdout);
});

I want to buffer stdout data that comes from the child process until it exits. Typically, I do this by appending data as it comes in (chunk) to a string variable (stdout).

This is an anti-pattern. There’s no need for the data I’m accumulating in stdout to be a string. When I convert Buffers into strings with setEncoding("utf8"), and then append them to create an even bigger string, I’m needlessly copying data into V8 when it’s more efficient to keep it outside.

We want to keep as much data as we can inside of Buffers, which corresponds to memory outside of the V8 heap. Node makes this easy. Since I’m collecting a lot of data here, I could do this better by rewriting that code to the following:

var stdout = [],
    child = cp.spawn(process.execPath, args);

child.stdout.on("data", function (chunk) {
    stdout.push(chunk);
});
child.on("close", function () {
    if (Buffer.isBuffer(stdout[0])) {
        cb(null, Buffer.concat(stdout));
        return;
     }
     cb(null);
});

Instead of taking large strings and appending them to enlarge another larger string, I can simply gather all of the Buffer chunks in an array until I’m ready to concat them all into a single large Buffer.

If I needed to read part of the output, perhaps the last few characters, I can use the Buffer’s slice method to convert a small part of the total results to a string for inspection.

A smart way to do this easily is to use the concat-stream module, which does this all for you.

If you liked this, you’ll love going on substack’s stream-adventure. You’re guaranteed to learn something, so give it a try.

Do For One

Sometimes I cruise around on my Tikit. Yesterday I didn’t fare too well.

San Thomas El Camino Intersection

While making a left turn, my tiny 16″ wheel sank into the ground at 15 mph. The pothole blended in nicely with years of oil stains from years of cars waiting to turn left. I couldn’t keep the bike under control with one hand signaling left. Next thing I knew, I was on the ground.

I banged up my knee and had some scrapes. Nothing too bad. But what surprised me was that the driver of the car next to me immediately got out and helped me off the ground. He insisted on giving me a bottled water. He made sure I was okay before taking off. Another woman who saw me limping away stopped her car to give me bandages. I was stoked to see these strangers help me out. But as an introverted guy, inside my car driving by, would I do the same?

Cars, phones, earbuds, the iPad I’m typing on: All of these things can be world-canceling devices. That isn’t always bad, of course. But it’s easier to miss the opportunity for better use of our time. Jonathan Safran Foer, in his NYT article How Not to Be Alone, shared his experience using a phone to avoid a difficult moment with another person:

The phone didn’t make me avoid the human connection, but it did make ignoring her easier in that moment, and more likely, by comfortably encouraging me to forget my choice to do so. My daily use of technological communication has been shaping me into someone more likely to forget others. The flow of water carves rock, a little bit at a time. And our personhood is carved, too, by the flow of our habits.

…everyone is always in need of something that another person can give, be it undivided attention, a kind word or deep empathy. There is no better use of a life than to be attentive to such needs. There are as many ways to do this as there are kinds of loneliness, but all of them require attentiveness, all of them require the hard work of emotional computation and corporeal compassion.

While we must balance the time we spend on others with time we spend on ourselves, the best parts of my life are those spent helping others. I find myself leveraging modern tools to ignore others, almost by habit, missing the best use of my life.

The nice strangers who took time out of their day to help me, garnering ire from the traffic behind them, are inspirations to a life-long introvert like me who would struggle to decide to open my car door.

We’re carved by our habits. Our big decisions are made by all of the little choices we make. Nobody will ever see most of these choices, but that’s okay. Deciding to engage with a messy world in small things will carve out an attitude of compassion that can make a big difference over time.

You can’t help everyone, but why not do for one what you wish you could do for all?

I’m Going to Zimbabwe

Important update, May 9, 2013: Due to political unrest, this trip will not be happening this year and is tentatively rescheduled for 2014. All funds raised will be put toward a future trip. The original post is below.

I am going to Zimbabwe for two weeks this July to serve alongside Hands of Hope Africa as they care for orphans and children at risk.

Zimbabwe has more orphans per capita than any other country in the world. There are 1.4 million kids under the age of 17 who are orphaned.

I’ll be working at the WestGate Haven Home, which is home to 12 school-aged girls who lost their parents and other close family members to AIDS.

Girls at WestGate Haven

I’m going to be with about a dozen other people traveling from San Jose, CA. It’s a huge privilege to be able to go and experience the world in a totally different context.

After I visited Peru last year, the trip left a lasting impression that made a difference in my life back home. We took deserving kids on a countryside retreat for 3 days—swimming, wheelchair races, horseback riding—which left a lasting impression on the kids and staff too. That makes me excited to serve again.

Learn more about my trip and why I’m going. Since announcing the trip yesterday on Twitter, I’ve already received a few donations toward the trip’s $4,000 cost, which is fantastic! Thank you.

The Real Meaning of Friendship

Douglas Gresham remembers the friendships of C.S. Lewis in the 1920s:

Now friendship in those days was a bit different from what it is today; friends did not have to agree on everything and often agreed on practically nothing. They were people with whom you could argue all day and yet never get irritated or angry at all. In today’s world we seem to have lost the real meaning of friendship. If someone disagrees with us, it is fashionable today to dislike them for it. This is silly and robs us of the best kind of friends we could find, for if we are always agreed with, we can never really have a serious conversation; we cannot learn from someone who agrees with what we say.

Don’t filter bubble your friends.

Build Modules, Not Examples

I’ve been thinking out loud this afternoon about how modules make Node.js successful with @ekashida, YUI’s newest team member.

YUI is a powerful open-source library for developing web apps. Node and YUI both document their APIs very well. Unlike Node, we also have 250 examples to help people get started. Yet lots of people have trouble getting started anyway.

How do we make that easier?

We’re not the only web app library with this problem. Ember has received criticism around getting started and quickly responded with a roadmap to improve it. In their response, Tom Dale said:

Ember promises—and, we think, delivers—tremendous value. But ramping up to that point is not easy, and we received this feedback repeatedly and take it very seriously.

We get the same feedback for YUI. We also take it seriously and a fellow team member is working on better documentation. Ember’s action items include a Getting Started guide, a short screencast, live examples to run demo code, and other things. These are good and will help.

But then, I think about Node. Their official newbie documentation begins and ends with Hello World on their homepage. Node’s newbies do not get started with an example. They get started with npm.

Modules Are The Example

Node has effectively outsourced their examples to the community using npm.

You may be thinking: “Reid, modules are not examples. They are tools you use to build an example, or a real app.” Well, you’d be right. Node’s community has built enough modules to help people build apps they care about with a tiny learning curve. Instead of reading about code, these newbies are building something with a tool—the module—that lets them start building what they care about right away.

Few newcomers to Node.js use require("http") to build their first web app. If the barrier-to-entry was to understand the HTTP module, a lot of people would walk away. They don’t want to learn about that module, they want routing. View rendering. You know, a web app.

The example is Express. Newbies have a project in mind and it’s probably something that needs a router, view rendering, and middleware. Express delivers that. You don’t need to learn about Node yet, you learn Express.

Once you’re up and running, you have real code and real problems. Not theoretical problems in a narrative, but problems that are running on your computer. Problems getting in the way of building what you care about. That’s motivation.

Since you’ve already spent an hour getting the basics setup because the Express API was a lot easier to get started with, you’re now motivated to dig deeper into Node to find the next solution.

Building A Ramp, Not A Cliff

Understanding everything Node has to offer isn’t something you learn overnight. But that isn’t necessary. A lot of people have built modules on top of Node’s core that make all kinds of common problems easy. These modules are easy to download and use. Newbies are engaged immediately. Then, they’re are progressively lured into deeper water. The community grows.

If you need to learn a million things before you can get started, you’ll never start. If that’s your project’s learning curve, you’re presenting newbies with a cliff. Some people will climb it, some will seek climbing experts to get to the top. Some may have the patience to read a book about climbing, or watch a screencast featuring someone else climbing.

But most people want to get started today, so they’ll leave and find an alternative way to the top that’s easier to climb.

Node provides many ramps. They aren’t ramps you merely read about: they’re ramps you walk on, build on, get experience with, and get more complicated as you go up. It doesn’t fix every problem and some people will still be confused. But it works pretty well.

At YUI, I’ll be encouraging our team to focus on building example modules: the ramps to mastery. YUI is already modular, but that isn’t the same. I’m talking about bundling modules in YUI’s core into a fully-baked solution to a common problem. Other people on my team are hard at work on the first product, and now I know why I’m so excited: it’s a ramp.

Own Your Email

If you’re reading my blog, you might be a professional working in technology. You’re likely to care about your online identity, and if you do, your publishing and communications must happen from your own domain.

If you care about your online presence, you must own it. I do, and that’s why my email address has always been at my own domain, not the domain of any employer or webmail service.

You might think your @gmail.com address will be fine indefinitely, but if I used a webmail address from the best webmail provider at the time I broke away from my university address and formed my own identity, it would have ended in @hotmail.com. And that wasn’t very long ago.

Own Your Identity by Marco Arment

I’ve had email at my domain for many years, so I don’t face the headache some do with switching today. If your email ends with someone else’s domain, bite the bullet and make the switch. The best time may have been in the past, but the second best time is right now.

fastmail

I’m currently trying out FastMail for hosting my email. I previously used Google Apps for Your Domain. Since late last year, that product has been focused on businesses. I’ve wanted to try the benefits of paid email (no ads) with a bring-your-own-domain service intended for individuals.

Here’s some factors that went into my decision to try FastMail:

Here’s what I don’t like, and why you shouldn’t switch:

  • I’ve had a couple spam messages come into my FastMail inbox, which rarely happens in GMail. You can train a personal Bayes filter over time, and tweak the spam score sensitivity, but I doubt it’ll be as good.
  • No 2-factor auth without expensive SMS messaging, but I don’t need this as much since I use 1Password.
  • Their documentation is pretty atrocious. They document everything but you’ll spend time digging for it.
  • Filters are not as easy to setup, but you get to edit the code behind them which is based on standard Sieve.

The good: Their website is fast. Gravatars are used in their website and I prefer its look-and-feel to GMail. If you don’t like it, you can run your own custom CSS and JavaScript to customize it.

We’ll see how this works out. If you decide to switch, read their migration instructions to get properly relocated from your current IMAP provider. Make sure you put SPF and DomainKeys into your DNS configuration while you’re changing MX records. The SPF record to set is v=spf1 include:spf.messagingengine.com -all (on TXT and SPF, if your DNS does both). The DomainKeys TXT record to set is on the “Virtual Domains” advanced settings screen once you sign in.

No matter what you choose, go forth and own your email. Even if my trial of FastMail goes south, they’re just a provider behind my own domain, and I can even go back to GMail. Switching providers is easy once your email is on your domain, so get to it.

Write-only

I’m taking a short break from most of the web and social media from February 13 – 23. I plan to share stuff, but I’ll be treating most websites as write-only. I’ll be working on Yeti at work (writing a lot of unit tests) and video editing at home (lots of Final Cut Pro projects), so I’ll continue to be quite busy. Email and bug reports are always open.

While I didn’t decide to do this myself, it’s timely. I’m distracted and controlled by an insatiable interest in what other folks create. I don’t use a feed reader and rarely find myself saving to Instapaper. My attention is divided between what I find important and Hacker News, The Verge, Flickr, Twitter, Facebook, and so on. It has grown to become taxing.

The habits are hard to change, but I’m not doing it alone. My pastor, Jay Kim, has suggested spending a part of lent by taking a break from media, or a media fast.

Fasting reveals what controls us. Far too often, we self-medicate with movies, television, music, social media, etc. We spend too little time alone and unplugged.

I’ll still be busy, but I’ll be spending more time unplugged and undistracted by the noise of the web for a little bit. I hope to return better focused on what I find important. If you’re feeling the same way, consider taking the next week or so off media with me. You can email me if you don’t want to do it alone. Here’s to more time spent offline.

Debugging Travis builds

Important Update December 19, 2013 — Travis infrastructure has changed since this post was written. This post remains unchanged for posterity; however, be aware that this setup no longer represents the current state of Travis.

Yeti uses Travis CI to the max by spawning lots of PhantomJS processes that are tested asynchronously. When tests start failing in Travis, but not anywhere else, debugging can be infuriating.

You can setup a Travis box locally to debug the problem. Trung Lê wrote about debugging with a local Travis VM in August 2012 but that post is outdated and didn’t work. You can expect sweeping changes to the Travis build infrastructure in the near future, so this blog post will also be outdated soon.

For now, we have a bug that needs fixing. Let’s setup a local Travis VM for Node.js debugging.

My assumptions about your computer

  1. Plenty of disk space
  2. OS X (but this should work with other systems, too)

Install Vagrant & VirtualBox

Visit the Vagrant website and the VirtualBox website for installers.

Download the jvm box

The jvm box is where Node.js builds run. Download travis-jvm.box. (3.2 GB)

Prepare the jvm box

Import the VM box:

vagrant box add travis-jvm travis-jvm.box

Verify it’s there:

vagrant box list

Start the jvm box

Create a working directory to hold the Vagrantfile for the travis-jvm box.

mkdir boxes
cd boxes

Create the Vagrantfile and get the box ready to use:

vagrant init travis-jvm

The username for the VM is travis, so add this line to your Vagrantfile’s do-block:

config.ssh.username = "travis"

Start the VM:

vagrant up

Things should work nicely. (If not, read on.)

You can access the VM with this command:

vagrant ssh

You can now follow the steps of your Travis build and debug your problem.

If you have problems running vagrant ssh, you can try using a username and password for logging in instead with this command:

vagrant ssh -p -- -l travis

The password is travis.

More fancy debugging

The VM uses a NAT-only network. You may not be able to access it from your computer. For the problems I debug, I need to be able to access the the Node web server inside the VM from a browser on OS X.

This means you should setup a host-only network interface, in addition to the NAT-only interface.

Add this line to your Vagrantfile’s do-block:

config.vm.network :hostonly, "192.168.89.10"

This will setup a host-only network and give the IP address 192.168.89.10 to the VM. (If you are on a network that uses 192.168.89.xxx already, change the IP subnet to something else, because it cannot conflict with another subnet your host computer is using.)

Reload the VM:

vagrant reload

You should be able to access the VM from the host at 192.168.89.10.

Sometimes running vagrant up with a host-only network fails because of a bug in net-ssh and Vagrant with the message “Waiting for VM to boot. This can take a few minutes.” This bug is quite annoying, but a detailed workaround procedure that clears DHCP leases is available that’ll allow you to start the VM.

Teardown

You can pause the VM with vagrant suspend and bring it back to life with vagrant resume. If you’d like to power off the VM, do so over SSH. If you can’t, vagrant halt will power it off.

If you want to start from a clean slate, you can destroy it with vagrant destroy. Re-creating a fresh VM is as easy as vagrant up.

You can learn more about these commands by reading the Vagrant teardown documentation.

Do not debug on travis-ci.org

When you have a Travis-only problem, try creating a local environment as described above first. Don’t try to use the hosted Travis service to debug complicated failures. I have made this mistake many times:

  1. Noticed my build is failing in Travis.
  2. Make sure my own environments and clean installs do not fail. (The problem is sometimes here!)
  3. Create a branch named fix-travis-foo or something.
  4. Start making commits and pushes to this temporary branch attempting to fix the problem.
  5. Wait for the build to finish. If the build failed, repeat step 4.

First of all, this takes a long time. Travis isn’t terribly slow for CI purposes, but when you’re trying to actively debug something, the last thing you need is having your app tests go from 15 seconds to 5 minutes.

Most of all, my problems are usually with PhantomJS. When I can fully control the VM, I can add remotely debug the headless browser on OS X. I can also capture packets creatively with tcpdump(8). Using those tools, I found some pretty heinous race conditions in the Yeti test suite that would have been nearly impossible to find by trial-and-error.