Stream Concat Anti-Pattern

July 3, 2013

NodeConf 2013 was great. You should go. I learned a lot of new stuff. Yesterday, I applied things I learned there for the first time.

I often write code that looks like something like this:

var stdout = "",
    child = cp.spawn(process.execPath, args);

child.stdout.setEncoding("utf8");
child.stdout.on("data", function (chunk) {
    stdout += chunk;
});
child.on("close", function () {
    cb(null, stdout);
});

I want to buffer stdout data that comes from the child process until it exits. Typically, I do this by appending data as it comes in (chunk) to a string variable (stdout).

This is an anti-pattern. There’s no need for the data I’m accumulating in stdout to be a string. When I convert Buffers into strings with setEncoding("utf8"), and then append them to create an even bigger string, I’m needlessly copying data into V8 when it’s more efficient to keep it outside.

We want to keep as much data as we can inside of Buffers, which corresponds to memory outside of the V8 heap. Node makes this easy. Since I’m collecting a lot of data here, I could do this better by rewriting that code to the following:

var stdout = [],
    child = cp.spawn(process.execPath, args);

child.stdout.on("data", function (chunk) {
    stdout.push(chunk);
});
child.on("close", function () {
    if (Buffer.isBuffer(stdout[0])) {
        cb(null, Buffer.concat(stdout));
        return;
     }
     cb(null);
});

Instead of taking large strings and appending them to enlarge another larger string, I can simply gather all of the Buffer chunks in an array until I’m ready to concat them all into a single large Buffer.

If I needed to read part of the output, perhaps the last few characters, I can use the Buffer’s slice method to convert a small part of the total results to a string for inspection.

A smart way to do this easily is to use the concat-stream module, which does this all for you.

If you liked this, you’ll love going on substack’s stream-adventure. You’re guaranteed to learn something, so give it a try.

Technology