Processing Data Chunks

Learn how to create an API endpoint that processes data chunks.

Table of Contents

Next.js (/app)

Chunks in web streams are fundamental data units that can be of many different types depending on the content, such as String for text or Uint8Array for binary files. While standard Function responses contain full payloads of data processed on the server, streamed responses typically send data in chunks over time.

To do this, you create a ReadableStream and add a data source, then transform the stream's data chunks before they're read by the client. Finally, you write stream the data chunk by chunk as a Function response.

Jump to the full example to see the finished recipe.

Create a ReadableStream

Create a ReadableStream and add a data source. In this case, you'll create your own data by encoding text with TextEncoder:

// TextEncoder objects turn text content
// into streams of UTF-8 characters.
// You'll add this encoder to your stream
const encoder = new TextEncoder();
// This is the stream object, which clients can read from
// when you send it as a Function response
const readableStream = new ReadableStream({
  // The start method is where you'll add the stream's content
  start(controller) {
    const text = 'Stream me!';
    // Queue the encoded content into the stream
    controller.enqueue(encoder.encode(text));
    // Prevent more content from being
    // added to the stream
    controller.close();
  },
});

Transform the stream's data chunks

You then need to transform the stream's data chunks before they're read by the client. First, you'll decode the chunks with TextDecoder, then transform the text to uppercase before encoding the text again:

// TextDecoders can decode streams of
// encoded content. You'll use this to
// transform the streamed content before
// it's read by the client
const decoder = new TextDecoder();
// TransformStreams can transform a stream's chunks
// before they're read in the client
const transformStream = new TransformStream({
  transform(chunk, controller) {
    // Decode the content, so it can be transformed
    const text = decoder.decode(chunk);
    // Make the text uppercase, then encode it and
    // add it back to the stream
    controller.enqueue(encoder.encode(text.toUpperCase()));
  },
});

Write stream the data chunk by chunk

Finally, write stream the data chunk by chunk as a Function response:

// Finally, send the streamed response. Result:
// "STREAM ME!" will be displayed in the client
return new Response(readableStream.pipeThrough(transformStream), {
  headers: {
    'Content-Type': 'text/html; charset=utf-8',
  },
});

Full example

The final file will look like this:

Next.js (/app)

Next.js (/pages)

Other frameworks

app/api/chunk-example/route.ts

// This method must be named GET
export async function GET() {
  // TextEncoder objects turn text content
  // into streams of UTF-8 characters.
  // You'll add this encoder to your stream
  const encoder = new TextEncoder();
  // This is the stream object, which clients can read from
  // when you send it as a Function response
  const readableStream = new ReadableStream({
    // The start method is where you'll add the stream's content
    start(controller) {
      const text = 'Stream me!';
      // Queue the encoded content into the stream
      controller.enqueue(encoder.encode(text));
      // Prevent more content from being
      // added to the stream
      controller.close();
    },
  });
 
  // TextDecoders can decode streams of
  // encoded content. You'll use this to
  // transform the streamed content before
  // it's read by the client
  const decoder = new TextDecoder();
  // TransformStreams can transform a stream's chunks
  // before they're read in the client
  const transformStream = new TransformStream({
    transform(chunk, controller) {
      // Decode the content, so it can be transformed
      const text = decoder.decode(chunk);
      // Make the text uppercase, then encode it and
      // add it back to the stream
      controller.enqueue(encoder.encode(text.toUpperCase()));
    },
  });
 
  // Finally, send the streamed response. Result:
  // "STREAM ME!" will be displayed in the client
  return new Response(readableStream.pipeThrough(transformStream), {
    headers: {
      'Content-Type': 'text/html; charset=utf-8',
    },
  });
}

If you're not using a framework, you must either add "type": "module" to your package.json or change your JavaScript Functions' file extensions from .js to .mjs

Build your app and visit localhost:3000/api/chunk-example. You should see the text "STREAM ME!" in the browser.

See Understanding Chunks to learn more.