Buffer to Stream in Node

This series of posts catalogs basic functions in Node via the built-in APIs. There are many libraries that provide syntactic sugar over these functions. However, it's a good exercise to understand how to do them manual. As serverless architectures increase in popularity, for performance reasons it's important to reduce dependencies and size of the deployment packages.

This post talk about converting between Streams and Buffers. This is a common task when working with many of the built-in APIs.

Stream to Buffer

The first operation is converting a read stream to a buffer. The most efficient operations with Streams are piping them to another stream. This is common with file system access where you don't want to bloat memory usage.

When working with HTTP requests however, you may want to directly convert the response stream into a JSON object or parse url encoded values.

To accomplish this, you'll read the data supplied by the buffers data event. Streams will either supply a string or a Buffer as the value of the data event. This depends on whether the Buffer has an encoding set. By default it will output Buffers during the data event.

This means you can do the following to read from the stream.

function streamToBuffer(stream) {  
  return new Promise((resolve, reject) => {
    let buffers = [];
    stream.on('error', reject);
    stream.on('data', (data) => buffers.push(data))
    stream.on('end', () => resolve(Buffer.concat(buffers))

This function obtains a stream (likely from HTTP or FS access). Then it will push each Buffer into an array of Buffers. When the stream has been fully read, it will combine all of those Buffers with Buffer.concat.

This would obviously cause some issues for large streams, but it works great for small streams that need to be converted and carried around in memory in subsequent pieces of the application.

Buffer to Stream

The next piece is converting a Buffer into a stream. This is useful when you want to pipe data into an exist stream.

let Duplex = require('stream').Duplex;  
function bufferToStream(buffer) {  
  let stream = new Duplex();
  return stream;

This operation creates a duplex string and simply writes the buffer to it. The stream can now be used as any write stream normally would be.

comments powered by Disqus