ReactJS

Building a Multi-Response Streaming API with Node.js, Express, and React

Published Aug 9, 2023

Updated Aug 17, 2023

6 min read

This article was written over 18 months ago and may contain information that is out of date. Some content may be relevant but please refer to the relevant official documentation or available resources for the latest information.

Introduction

As web applications become increasingly complex and data-driven, efficient and effective data transfer methods become critically important. A streaming API that can send multiple responses to a single request can be a powerful tool for handling large amounts of data or for delivering real-time updates. In this article, we will guide you through the process of creating such an API.

We will use video streaming as an illustrative example. With their large file sizes and the need for flexible, on-demand delivery, videos present a fitting scenario for showcasing the power of multi-response streaming APIs. The backend will be built with Node.js and Express, utilizing HTTP range requests to facilitate efficient data delivery in chunks.

Next, we'll build a React front-end to interact with our streaming API. This front-end will handle both the display of the streamed video content and its download, offering users real-time progress updates.

By the end of this walkthrough, you will have a working example of a multi-response streaming API, and you will be able to apply the principles learned to a wide array of use cases beyond video streaming.

Let's jump right into it!

Hands-On

Implementing the Streaming API in Express

In this section, we will dive into the server-side implementation, specifically our Node.js and Express application. We'll be implementing an API endpoint to deliver video content in a streaming fashion.

Assuming you have already set up your Express server with TypeScript, we first need to define our video-serving route. We'll create a GET endpoint that, when hit, will stream a video file back to the client.

Please make sure to install cors for handling cross-origin requests, dotenv for loading environment variables, and throttle for controlling the rate of data transfer. You can install these with the following command:

yarn add cors dotenv throttle @types/cors @types/dotenv @types/throttle

import cors from 'cors';
import 'dotenv/config';
import express, { Request, Response } from 'express';
import fs from 'fs';
import Throttle from 'throttle';

const app = express();
const port = 8000;

app.use(cors());

app.get('/video', (req: Request, res: Response) => {
  // Video by Zlatin Georgiev from Pexels: https://www.pexels.com/video/15708449/
  // For testing purposes - add the video in you `static` folder
  const path = 'src/static/pexels-zlatin-georgiev-15708449 (2160p).mp4';
  const stat = fs.statSync(path);
  const fileSize = stat.size;
  const range = req.headers.range;
  if (range) {
    const parts = range.replace(/bytes=/, '').split('-');
    const start = parseInt(parts[0], 10);
    const end = parts[1] ? parseInt(parts[1], 10) : fileSize - 1;

    const chunksize = end - start + 1;
    const file = fs.createReadStream(path, { start, end });
    const head = {
      'Content-Range': `bytes ${start}-${end}/${fileSize}`,
      'Accept-Ranges': 'bytes',
      'Content-Length': chunksize,
      'Content-Type': 'video/mp4',
    };

    res.writeHead(206, head);
    file.pipe(res);
  } else {
    const head = {
      'Content-Length': fileSize,
      'Content-Type': 'video/mp4',
    };

    res.writeHead(200, head);
    fs.createReadStream(path).pipe(res);
  }
});

app.listen(port, () => {
  console.log(`Server listening at ${process.env.SERVER_URL}:${port}`);
});

In the code snippet above, we are implementing a basic video streaming server that responds to HTTP range requests. Here's a brief overview of the key parts:

File and Range Setup: We start by determining the path to the video file and getting the file size. We also grab the range header from the request, which contains the range of bytes the client is requesting.
Range Requests Handling: If a range is provided, we extract the start and end bytes from the range header, then create a read stream for that specific range. This allows us to stream a portion of the file rather than the entire thing.
Response Headers: We then set up our response headers. In the case of a range request, we send back a '206 Partial Content' status along with information about the byte range and total file size. For non-range requests, we simply send back the total file size and the file type.
Data Streaming: Finally, we pipe the read stream directly to the response. This step is where the video data actually gets sent back to the client. The use of pipe() here automatically handles backpressure, ensuring that data isn't read faster than it can be sent to the client.

With this setup in place, our streaming server is capable of efficiently delivering large video files to the client in small chunks, providing a smoother user experience.

Implementing the Download API in Express

Now, let's add another endpoint to our Express application, which will provide more granular control over the data transfer process. We'll set up a GET endpoint for '/download', and within this endpoint, we'll handle streaming the video file to the client for download.

app.get('/download', (req: Request, res: Response) => {
  // Again, for testing purposes - add the video in you `static` folder
  const path = 'src/static/pexels-zlatin-georgiev-15708449 (2160p).mp4';
  const stat = fs.statSync(path);
  const fileSize = stat.size;

  res.writeHead(200, {
    'Content-Type': 'video/mp4',
    'Content-Disposition': 'attachment; filename=video.mp4',
    'Content-Length': fileSize,
  });

  const readStream = fs.createReadStream(path);
  const throttle = new Throttle(1024 * 1024 * 5); // throttle to 5MB/sec - simulate lower speed

  readStream.pipe(throttle);

  throttle.on('data', (chunk) => {
    Console.log(`Sent ${chunk.length} bytes to client.`);
    res.write(chunk);
  });

  throttle.on('end', () => {
    console.log('File fully sent to client.');
    res.end();
  });
});

This endpoint has a similar setup to the video streaming endpoint, but it comes with a few key differences:

Response Headers: Here, we include a 'Content-Disposition' header with an 'attachment' directive. This header tells the browser to present the file as a downloadable file named 'video.mp4'.
Throttling: We use the 'throttle' package to limit the data transfer rate. Throttling can be useful for simulating lower-speed connections during testing, or for preventing your server from getting overwhelmed by data transfer operations.
Data Writing: Instead of directly piping the read stream to the response, we attach 'data' and 'end' event listeners to the throttled stream. On the 'data' event, we manually write each chunk of data to the response, and on the 'end' event, we close the response.

This implementation provides a more hands-on way to control the data transfer process. It allows for the addition of custom logic to handle events like pausing and resuming the data transfer, adding custom transformations to the data stream, or handling errors during transfer.

Utilizing the APIs: A React Application

Now that we have a server-side setup for video streaming and downloading, let's put these APIs into action within a client-side React application. Note that we'll be using Tailwind CSS for quick, utility-based styling in our components.

Our React application will consist of a video player that uses the video streaming API, a download button to trigger the download API, and a progress bar to show the real-time download progress.

First, let's define the Video Player component that will play the streamed video:

import React from 'react';

const VideoPlayer: React.FC = () => {
  return (
    <div className="mb-3">
      <video controls width="860">
        <source src="http://localhost:8000/video" type="video/mp4" />
        Your browser does not support the video tag.
      </video>
    </div>
  );
};

export default VideoPlayer;

In the above VideoPlayer component, we're using an HTML5 video tag to handle video playback. The src attribute of the source tag is set to the video endpoint of our Express server. When this component is rendered, it sends a request to our video API and starts streaming the video in response to the range requests that the browser automatically makes.

Next, let's create the DownloadButton component that will handle the video download and display the download progress:

import React, { useState } from 'react';

const DownloadButton: React.FC = () => {
  const [downloadProgress, setDownloadProgress] = useState(0);

  const handleDownload = async () => {
    try {
      const response = await fetch('http://localhost:8000/download');
      const reader = response.body?.getReader();

      if (!reader) {
        return;
      }

      const contentLength = +(response.headers?.get('Content-Length') || 0);
      let receivedLength = 0;
      let chunks = [];

      while (true) {
        const { done, value } = await reader.read();

        if (done) {
          console.log('Download complete.');

          const blob = new Blob(chunks, { type: 'video/mp4' });
          const url = window.URL.createObjectURL(blob);
          const a = document.createElement('a');
          a.style.display = 'none';
          a.href = url;
          a.download = 'video.mp4';
          document.body.appendChild(a);
          a.click();
          window.URL.revokeObjectURL(url);

          setDownloadProgress(100);
          break;
        }
        chunks.push(value);
        receivedLength += value.length;
        const progress = (receivedLength / contentLength) * 100;
        setDownloadProgress(progress);
      }
    } catch (err) {
      console.error(err);
    }
  };

  return (
    <div className="flex gap-3 items-center">
      <button
        onClick={handleDownload}
        className="py-2 px-3 bg-indigo-500 text-white text-sm font-semibold rounded-md shadow focus:outline-none"
      >
        Download Video
      </button>
      {downloadProgress > 0 && downloadProgress < 100 && (
      <p className="flex gap-3 items-center">
        Download progress: <progress value={downloadProgress} max={100} />
      </p>
      )}
      {downloadProgress === 100 && <p>Download complete!</p>}
    </div>
  );
};

export default DownloadButton;

In this DownloadButton component, when the download button is clicked, it sends a fetch request to our download API. It then uses a while loop to continually read chunks of data from the response as they arrive, updating the download progress until the download is complete. This is an example of more controlled handling of multi-response APIs where we are not just directly piping the data, but instead, processing it and manually sending it as a downloadable file.

Bringing It All Together

Let's now integrate these components into our main application component.

import React from 'react';
import VideoPlayer from './components/VideoPlayer';
import DownloadButton from './components/DownloadButton';

function App() {
  return (
    <div className="flex flex-col items-center justify-center min-h-screen bg-gray-100 py-2">
      <h1 className="text-3xl font-bold mb-5">My Video Player</h1>
      <VideoPlayer />
      <DownloadButton />
    </div>
  );
}

export default App;

In this simple App component, we've included our VideoPlayer and DownloadButton components. It places the video player and download button on the screen in a neat, centered layout thanks to Tailwind CSS.

Here is a summary of how our system operates:

The video player makes a request to our Express server as soon as it is rendered in the React application. Our server handles this request, reading the video file and sending back the appropriate chunks as per the range requested by the browser. This results in the video being streamed in our player.
When the download button is clicked, a fetch request is sent to our server's download API. This time, the server reads the file, but instead of just piping the data to the response, it controls the data sending process. It sends chunks of data and also logs the sent chunks for monitoring purposes. The React application collects these chunks and concatenates them, displaying the download progress in real-time. When all chunks are received, it compiles them into a Blob and triggers a download in the browser.

This setup allows us to build a full-featured video streaming and downloading application with fine control over the data transmission process. To see this system in action, you can check out this video demo.

Conclusion

While the focus of this article was on video streaming and downloading, the principles we discussed here extend beyond just media files. The pattern of responding to HTTP range requests is common in various data-heavy applications, and understanding it can be a useful tool in your web development arsenal.

Finally, remember that the code shown in this article is just a simple example to demonstrate the concepts. In a real-world application, you would want to add proper error handling, validation, and possibly some form of access control depending on your use case.

I hope this article helps you in your journey as a developer. Building something yourself is the best way to learn, so don't hesitate to get your hands dirty and start coding!

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

William Mimura
@mimurawil

Communication Between Client Components in Next.js

Communication Between Client Components in Next.js In recent years, Next.js has become one of the most popular React frameworks for building server-rendered applications. With the introduction of the App Router in Next.js 13, the framework has taken a major leap forward by embracing a new approach to building web applications: the concept of server components and client components. This separation of concerns allows developers to strategically decide which parts of their application should be rendered on the server and which are then hydrated in the browser for interactivity. The Challenge: Communicating Between Client Components While server components offer numerous benefits, they also introduce a new challenge: how can client components within different boundaries communicate with each other? For instance, let's consider a scenario where you have a button (a client component) and a separate client component that displays the number of times the button has been clicked. In a regular React application, this would typically be accomplished by lifting the state to a common ancestor component, allowing the button to update the state, which is then passed down to the counter display component. However, the traditional approach may not be as straightforward in a Next.js application that heavily relies on server components, with client components scattered across the page. This blog post will explore three ways to facilitate communication between client components in Next.js, depending on where you want to store the state. Lifting State to a Common Client Component One approach to communicating between client components is to lift the state to a common client component. This could be a React context provider, a state management system like Zustand, or any other solution that allows you to share state across components. The key aspect is that this wrapper component should be higher up in the tree (perhaps even in the layout) and accept server components as children. Next.js allows you to interleave client and server components as much as you want, as long as server components are passed to client components as props or as component children. Here's how this approach might look in practice. First of all, we'd create a wrapper client component that holds the state: ` This wrapper component can be included in the layout: ` Any server components can be rendered within the layout, potentially nesting them several levels deep. Finally, we'd create two client components, one for the button and one for the counter display: ` ` The entire client and server component tree is rendered on the server, and the client components are then hydrated in the browser and initialized. From then on, the communication between the client components works just like in any regular React application. Check out the page using this pattern in the embedded Stackblitz window below: Using Query Params for State Management Another approach is to use query params instead of a wrapper client component and store the state in the URL. In this scenario, you have two client components: the button and the counter display. The counter value (the state) is stored in a query param, such as counterValue. The client components can read the current counter value using the useSearchParams hook. Once read, the useRouter hook can update the query param, effectively updating the counter value. However, there's one gotcha to this approach. If a route is statically rendered, calling useSearchParams will cause the client component tree up to the closest Suspense boundary to be client-side rendered. Next.js recommends wrapping the client component that uses useSearchParams in a boundary. Here's an example of how this approach might look. The button reads the current counter value and updates it on click by using the router's replace function: ` The counter display component is relatively simple, only reading the counter value: ` And here is the page that is a server component, hosting both of the above client components: ` Feel free to check out the above page in the embedded Stackblitz below: Storing State on the Server The third approach is to store the state on the server. In this case, the counter display component accepts the counter value as a prop, where the counter value is passed by a parent server component that reads the counter value from the database. The button component, when clicked, calls a server action that updates the counter value and calls revalidatePath() so that the counter value is refreshed, and consequently, the counter display component is re-rendered. It's worth noting that in this approach, unless you need some interactivity in the counter display component, it doesn't need to be a client component – it can be purely server-rendered. However, if both components need to be client components, here's an example of how this approach might look. First, we'll implement a server action that updates the counter value. We won't get into the mechanics of updating it, which in a real app would require a call to the database or an external API - so we're only commenting that part. After that, we revalidate the path so that the Next.js caches are purged, and the counter value is retrieved again in server components that read it. ` The button is a simple client component that calls the above server action when clicked. ` The counter display component reads the counter value from the parent: ` While the parent is a server component that reads the counter value from the database or an external API. ` This pattern can be seen in the embedded Stackblitz below: Conclusion Next.js is a powerful framework that offers various choices for implementing communication patterns between client components. Whether you lift state to a common client component, use query params for state management, or store the state on the server, Next.js provides all the tools you need to add a communication path between two separate client component boundaries. We hope this blog post has been useful in demonstrating how to facilitate communication between client components in Next.js. Check out other Next.js blog posts we've written for more insights and best practices. You can also view the entire codebase for the above snippets on StackBlitz....

Jun 21, 2024

5 mins

NextJSReact

Remix's evolution to a Vite plugin

The Remix / React Router news is a beautiful illustration of software evolution. You start with a clear idea of what you’re building, but it’s impossible to anticipate exactly what it will become. If you haven’t heard the news yet, Ryan Florence announced at React Conf 2024 that Remix will be a Vite plugin for React Router. It’s a sort of surprising announcement but it makes a lot of sense if you’ve been following along as Remix the project has evolved. In this post we’ll recap the evolution of the project and then dig into what this change means for the future of Remix in the Server Components era of React. 🗓️ October 2021 > Remix is open sourced In the beginning, Remix cost money. In October 2021 they received some seed funding and the project was open sourced. Remix was an exciting new framework for server-rendering React websites that included API’s for loading data on the server and submitting data to the server with forms. It started a trend in the front-end framework space that a lot of other popular frameworks have modeled after or taken inspiration from. 🗓️ March 2022 - September 2022 > Remixing React Router At some point they realized that the API’s that they had created for Remix were a more natural fit in React Router. In this post Ryan details the plans to move these Remix API’s into React Router. Several months later, React Router 6.4 is released with the action, loader, and other Remix API’s baked in. > "We've always thought of Remix as "just a compiler and server for React Router" After this release, Remix is now mostly as they described - a compiler and server for React Router. 🗓️ October 2023 - February 2024 > Remix gets Vite support If you’re not familiar with Vite, it’s an extremely popular tool for front-end development that handles things like development servers and code bundling/compiling. A lot of the modern front-end frameworks are built on top of it. Remix announced support for Vite as an alternative option to their existing compiler and server. Since Remix is now mostly “just a compiler and server for React Router” - what’s left of Remix now that Vite can handle those things? 🗓️ May 2024 > Remix is React Router The Remix team releases a post detailing the announcement that Ryan made at React Conf. Looking back through the evolution of the project this doesn’t seem like an out of the blue, crazy announcement. Merging Remix API’s into React Router and then replacing the compiler and server with Vite didn’t leave much for Remix to do as a standalone project. 🐐 React Router - the past, present, and future of React React Router is one of the oldest, and most used packages in the React ecosystem. > Since Remix has always been effectively "React Router: The Framework", we wanted to create a bridge for all these React Router projects to be able to upgrade to Remix. With Remix being baked into React Router and now supporting SPA mode, legacy React Router applications have an easier path for migrating to the next generation of React. https://x.com/ryanflorence/status/1791488550162370709 Remix vs React 19 If you’ve taken a look at React 19 at all, it turns out it now handles a lot of the problems that Remix set out to solve in the first place. Things like Server Components, Server Actions, and Form Actions provide the same sort of programming model that Remix popularized. https://x.com/ryanflorence/status/1791484663883829258 Planned Changes for Remix and React Router Just recently, a post was published giving the exact details of what the changes to Remix and React Router will look like. tl;dr For React Router * React Router v6 to v7 will be a non-breaking upgrade * The Vite plugin from Remix is coming to React Router in v7 * The Vite plugin simply makes existing React Router features more convenient to use, but it isn't required to use React Router v7 * v7 will support both React 18 and React 19 For Remix * What would have been Remix v3 is React Router v7 * Remix v2 to React Router v7 will be a non-breaking upgrade * Remix is coming back better than ever in a future release with an incremental adoption strategy enabled by these changes For Both * React Router v7 comes with new features not in Remix or React Router today: RSC, server actions, static pre-rendering, and enhanced Type Safety across the board The show goes on I love how this project has evolved and I tip my hat to the Remix/React Router team. React Router feels like a cozy blanket in an ecosystem that is going through some major changes which feels uncertain and a little scary at times. The Remix / React Router story is still being written and I’m excited to see where it goes. I’m looking forward to seeing the work they are doing to support React Server Components. A lot of people are pretty excited about RSC’s and there is a lot of room for unique and interesting implementations. Long live React Router 🙌....

Jul 10, 2024

4 mins

ReactViteRemix

Enhancing Your Playwright Workflow: A Guide to the VSCode Extension

Introduction In my last post, Quick Guide to Playwright Fixtures: Enhancing Your Tests, I delved into some of the enhancements we've been implementing in our end-to-end (E2E) tests using Playwright. As I refine our testing strategies, I've come across a tool that has quickly become an essential part of my workflow: the Playwright VSCode extension. If you're like me and constantly looking for ways to streamline testing and debugging, you'll appreciate any tool that can make the process more efficient and enjoyable. That's where this extension comes in. It's not just about writing tests - it's about enhancing the entire development experience. In this post, I'll walk you through getting started with the Playwright VSCode extension, sharing some tips and tricks that have made a real difference in my day-to-day work. Installing the Extension & Basic Setup Before diving into the Playwright VSCode extension, it's essential to have Playwright installed on your machine. If you haven't done so already, you can quickly install it by running: ` This command will set up Playwright and ensure all necessary dependencies are installed. Once Playwright is ready, the VSCode extension will be installed next. Open Visual Studio Code, navigate to the Extensions view by clicking on the Extensions icon in the Activity Bar on the side of the window, and search for “Playwright”. The official extension, ID: ms-playwright.playwright / named: “Playwright Test for VSCode” by Microsoft, should appear at the top of the list. Click "Install," and you're all set. With the extension installed, you can start leveraging its powerful features to enhance your Playwright testing workflow within VSCode. Running the Tests and Identifying Outputs To run a test, simply open the test file in VSCode. The Playwright extension will automatically detect test files and display a "Run" icon next to each test and test suite. You can click on this icon to run individual tests or test suites. Alternatively, you can run all the tests in your project using the Playwright Test Explorer, accessible from the sidebar. Once you start running your tests, the extension provides real-time feedback within the editor. You'll see the status of each test - whether it passes, fails, or is skipped - right next to the corresponding test in your code. This immediate feedback loop is incredibly helpful for catching issues as you write your tests. The output of your tests will be displayed in the VSCode terminal. You'll see detailed information about each test run. Debugging Step-by-Step I find debugging particularly useful when a test fails unexpectedly or when I want to verify that certain actions are being performed as intended. Instead of guessing what might be wrong, I can see exactly what's going on in each test step, making debugging a much more straightforward and less frustrating process. To start debugging, you can easily set a breakpoint in your test file by clicking on the left margin next to the line number where you'd like the execution to pause. Once your breakpoints are in place, you can initiate the debug process by selecting the "Debug" option next to the test you'd like to investigate. Once the debugger is running, the extension allows you to step through your code, inspect variables, and evaluate expressions - all within VSCode. This real-time insight into your test execution is a game-changer, enabling you to pinpoint issues more effectively and confidently refine your tests. Using the Pick Locator Tool Another handy feature is the "Pick Locator" tool. If you've ever struggled with selecting the right element in your tests, this tool can be a time saver. It helps you generate reliable locators by letting you interact directly with the webpage elements you want to target. To use the Pick Locator tool, click the "Pick Locator" button in the Playwright Test Explorer. This will open a new window where you can navigate to the site you're testing. As you hover over elements on the page, the tool will suggest locators, allowing you to select the most appropriate one for your test. While the Pick Locator tool is handy, it’s important to ensure that the locators you generate are robust and maintainable. This is especially true when integrating them with the fixtures I discussed in my previous blog post. Combining the proper locators with well-designed fixtures can create more reliable and reusable test setups, ultimately making your E2E tests more efficient. Conclusion The Playwright VSCode extension has quickly become indispensable in my development workflow. It significantly enhanced my experience of writing and running Playwright tests. Whether you’re just starting with Playwright or looking to optimize your existing tests, this extension offers a range of features that can save you time and effort. Combining these tools with thoughtful test design, such as leveraging fixtures, you can create a more efficient and effective testing process. I hope this guide has given you a good overview of what the Playwright VSCode extension can do and how it can benefit your work. If you haven’t tried it yet, I highly recommend giving it a go. And as always, feel free to explore further and experiment with the features that best suit your needs....

Sep 6, 2024

5 mins

VSCodeTestingPlaywright

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging)

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging) In the first part of my series on the importance of a scientific mindset in software engineering, we explored how the principles of the scientific method can help us evaluate sources and make informed decisions. Now, we will focus on how these principles can help us tackle one of the most crucial and challenging tasks in software engineering: debugging. In software engineering, debugging is often viewed as an art - an intuitive skill honed through experience and trial and error. In a way, it is - the same as a GP, even a very evidence-based one, will likely diagnose most of their patients based on their experience and intuition and not research scientific literature every time; a software engineer will often rely on their experience and intuition to identify and fix common bugs. However, an internist faced with a complex case will likely not be able to rely on their intuition alone and must apply the scientific method to diagnose the patient. Similarly, a software engineer can benefit from using the scientific method to identify and fix the problem when faced with a complex bug. From that perspective, treating engineering challenges like scientific inquiries can transform the way we tackle problems. Rather than resorting to guesswork or gut feelings, we can apply the principles of the scientific method—forming hypotheses, designing controlled experiments, gathering and evaluating evidence—to identify and eliminate bugs systematically. This approach, sometimes referred to as "scientific debugging," reframes debugging from a haphazard process into a structured, disciplined practice. It encourages us to be skeptical, methodical, and transparent in our reasoning. For instance, as Andreas Zeller notes in the book _Why Programs Fail_, the key aspect of scientific debugging is its explicitness: Using the scientific method, you make your assumptions and reasoning explicit, allowing you to understand your assumptions and often reveals hidden clues that can lead to the root cause of the problem on hand. Note: If you'd like to read an excerpt from the book, you can find it on Embedded.com. Scientific Debugging At its core, scientific debugging applies the principles of the scientific method to the process of finding and fixing software defects. Rather than attempting random fixes or relying on intuition, it encourages engineers to move systematically, guided by data, hypotheses, and controlled experimentation. By adopting debugging as a rigorous inquiry, we can reduce guesswork, speed up the resolution process, and ensure that our fixes are based on solid evidence. Just as a scientist begins with a well-defined research question, a software engineer starts by identifying the specific symptom or error condition. For instance, if our users report inconsistencies in the data they see across different parts of the application, our research question could be: _"Under what conditions does the application display outdated or incorrect user data?"_ From there, we can follow a structured debugging process that mirrors the scientific method: - 1. Observe and Define the Problem: First, we need to clearly state the bug's symptoms and the environment in which it occurs. We should isolate whether the issue is deterministic or intermittent and identify any known triggers if possible. Such a structured definition serves as the groundwork for further investigation. - 2. Formulate a Hypothesis: A hypothesis in debugging is a testable explanation for the observed behavior. For instance, you might hypothesize: _"The data inconsistency occurs because a caching layer is serving stale data when certain user profiles are updated."_ The key is that this explanation must be falsifiable; if experiments don't support the hypothesis, it must be refined or discarded. - 3. Collect Evidence and Data: Evidence often includes logs, system metrics, error messages, and runtime traces. Similar to reviewing primary sources in academic research, treat your raw debugging data as crucial evidence. Evaluating these data points can reveal patterns. In our example, such patterns could be whether the bug correlates with specific caching mechanisms, increased memory usage, or database query latency. During this step, it's essential to approach data critically, just as you would analyze the quality and credibility of sources in a research literature review. Don't forget that even logs can be misleading, incomplete, or even incorrect, so cross-referencing multiple sources is key. - 4. Design and Run Experiments: Design minimal, controlled tests to confirm or refute your hypothesis. In our example, you may try disabling or shortening the cache's time-to-live (TTL) to see if more recent data is displayed correctly. By manipulating one variable at a time - such as cache invalidation intervals - you gain clearer insights into causation. Tools such as profilers, debuggers, or specialized test harnesses can help isolate factors and gather precise measurements. - 5. Analyze Results and Refine Hypotheses: If the experiment's outcome doesn't align with your hypothesis, treat it as a stepping stone, not a dead end. Adjust your explanation, form a new hypothesis, or consider additional variables (for example, whether certain API calls bypass caching). Each iteration should bring you closer to a better understanding of the bug's root cause. Remember, the goal is not to prove an initial guess right but to arrive at a verifiable explanation. - 6. Implement and Verify the Fix: Once you're confident in the identified cause, you can implement the fix. Verification doesn't stop at deployment - re-test under the same conditions and, if possible, beyond them. By confirming the fix in a controlled manner, you ensure that the solution is backed by evidence rather than wishful thinking. - Personally, I consider implementing end-to-end tests (e.g., with Playwright) that reproduce the bug and verify the fix to be a crucial part of this step. This both ensures that the bug doesn't reappear in the future due to changes in the codebase and avoids possible imprecisions of manual testing. Now, we can explore these steps in more detail, highlighting how the scientific method can guide us through the debugging process. Establishing Clear Debugging Questions (Formulating a Hypothesis) A hypothesis is a proposed explanation for a phenomenon that can be tested through experimentation. In a debugging context, that phenomenon is the bug or issue you're trying to resolve. Having a clear, falsifiable statement that you can prove or disprove ensures that you stay focused on the real problem rather than jumping haphazardly between possible causes. A properly formulated hypothesis lets you design precise experiments to evaluate whether your explanation holds true. To formulate a hypothesis effectively, you can follow these steps: 1. Clearly Identify the Symptom(s) Before forming any hypothesis, pin down the specific issue users are experiencing. For instance: - "Users intermittently see outdated profile information after updating their accounts." - "Some newly created user profiles don't reflect changes in certain parts of the application." Having a well-defined problem statement keeps your hypothesis focused on the actual issue. Just like a research question in science, the clarity of your symptom definition directly influences the quality of your hypothesis. 2. Draft a Tentative Explanation Next, convert your symptom into a statement that describes a _possible root cause_, such as: - "Data inconsistency occurs because the caching layer isn't invalidating or refreshing user data properly when profiles are updated." - "Stale data is displayed because the cache timeout is too long under certain load conditions." This step makes your assumption about the root cause explicit. As with the scientific method, your hypothesis should be something you can test and either confirm or refute with data or experimentation. 3. Ensure Falsifiability A valid hypothesis must be falsifiable - meaning it can be proven _wrong_. You'll struggle to design meaningful experiments if a hypothesis is too vague or broad. For example: - Not Falsifiable: "Occasionally, the application just shows weird data." - Falsifiable: "Users see stale data when the cache is not invalidated within 30 seconds of profile updates." Making your hypothesis specific enough to fail a test will pave the way for more precise debugging. 4. Align with Available Evidence Match your hypothesis to what you already know - logs, stack traces, metrics, and user reports. For example: - If logs reveal that cache invalidation events aren't firing, form a hypothesis explaining why those events fail or never occur. - If metrics show that data served from the cache is older than the configured TTL, hypothesize about how or why the TTL is being ignored. If your current explanation contradicts existing data, refine your hypothesis until it fits. 5. Plan for Controlled Tests Once you have a testable hypothesis, figure out how you'll attempt to _disprove_ it. This might involve: - Reproducing the environment: Set up a staging/local system that closely mimics production. For instance with the same cache layer configurations. - Varying one condition at a time: For example, only adjust cache invalidation policies or TTLs and then observe how data freshness changes. - Monitoring metrics: In our example, such monitoring would involve tracking user profile updates, cache hits/misses, and response times. These metrics should lead to confirming or rejecting your explanation. These plans become your blueprint for experiments in further debugging stages. Collecting and Evaluating Evidence After formulating a clear, testable hypothesis, the next crucial step is to gather data that can either support or refute it. This mirrors how scientists collect observations in a literature review or initial experiments. 1. Identify "Primary Sources" (Logs, Stack Traces, Code History): - Logs and Stack Traces: These are your direct pieces of evidence - treat them like raw experimental data. For instance, look closely at timestamps, caching-related events (e.g., invalidation triggers), and any error messages related to stale reads. - Code History: Look for related changes in your source control, e.g. using Git bisect. In our example, we would look for changes to caching mechanisms or references to cache libraries in commits, which could pinpoint when the inconsistency was introduced. Sometimes, reverting a commit that altered cache settings helps confirm whether the bug originated there. 2. Corroborate with "Secondary Sources" (Documentation, Q&A Forums): - Documentation: Check official docs for known behavior or configuration details that might differ from your assumptions. - Community Knowledge: Similar issues reported on GitHub or StackOverflow may reveal known pitfalls in a library you're using. 3. Assess Data Quality and Relevance: - Look for Patterns: For instance, does stale data appear only after certain update frequencies or at specific times of day? - Check Environmental Factors: For instance, does the bug happen only with particular deployment setups, container configurations, or memory constraints? - Watch Out for Biases: Avoid seeking only the data that confirms your hypothesis. Look for contradictory logs or metrics that might point to other root causes. You keep your hypothesis grounded in real-world system behavior by treating logs, stack traces, and code history as primary data - akin to raw experimental results. This evidence-first approach reduces guesswork and guides more precise experiments. Designing and Running Experiments With a hypothesis in hand and evidence gathered, it's time to test it through controlled experiments - much like scientists isolate variables to verify or debunk an explanation. 1. Set Up a Reproducible Environment: - Testing Environments: Replicate production conditions as closely as possible. In our example, that would involve ensuring the same caching configuration, library versions, and relevant data sets are in place. - Version Control Branches: Use a dedicated branch to experiment with different settings or configuration, e.g., cache invalidation strategies. This streamlines reverting changes if needed. 2. Control Variables One at a Time: - For instance, if you suspect data inconsistency is tied to cache invalidation events, first adjust only the invalidation timeout and re-test. - Or, if concurrency could be a factor (e.g., multiple requests updating user data simultaneously), test different concurrency levels to see if stale data issues become more pronounced. 3. Measure and Record Outcomes: - Automated Tests: Tests provide a great way to formalize and verify your assumptions. For instance, you could develop tests that intentionally update user profiles and check if the displayed data matches the latest state. - Monitoring Tools: Monitor relevant metrics before, during, and after each experiment. In our example, we might want to track cache hit rates, TTL durations, and query times. - Repeat Trials: Consistency across multiple runs boosts confidence in your findings. 4. Validate Against a Baseline: - If baseline tests manifest normal behavior, but your experimental changes manifest the bug, you've isolated the variable causing the issue. E.g. if the baseline tests show that data is consistently fresh under normal caching conditions but your experimental changes cause stale data. - Conversely, if your change eliminates the buggy behavior, it supports your hypothesis - e.g. that the cache configuration was the root cause. Each experiment outcome is a data point supporting or contradicting your hypothesis. Over time, these data points guide you toward the true cause. Analyzing Results and Iterating In scientific debugging, an unexpected result isn't a failure - it's valuable feedback that brings you closer to the right explanation. 1. Compare Outcomes to the hypothesis. For instance: - Did user data stay consistent after you reduced the cache TTL or fixed invalidation logic? - Did logs show caching events firing as expected, or did they reveal unexpected errors? - Are there only partial improvements that suggest multiple overlapping issues? 2. Incorporate Unexpected Observations: - Sometimes, debugging uncovers side effects - e.g. performance bottlenecks exposed by more frequent cache invalidations. Note these for future work. - If your hypothesis is disproven, revise it. For example, the cache may only be part of the problem, and a separate load balancer setting also needs attention. 3. Avoid Confirmation Bias: - Don't dismiss contrary data. For instance, if you see evidence that updates are fresh in some modules but stale in others, you may have found a more nuanced root cause (e.g., partial cache invalidation). - Consider other credible explanations if your teammates propose them. Test those with the same rigor. 4. Decide If You Need More Data: - If results aren't conclusive, add deeper instrumentation or enable debug modes to capture more detailed logs. - For production-only issues, implement distributed tracing or sampling logs to diagnose real-world usage patterns. 5. Document Each Iteration: - Record the results of each experiment, including any unexpected findings or new hypotheses that arise. - Through iterative experimentation and analysis, each cycle refines your understanding. By letting evidence shape your hypothesis, you ensure that your final conclusion aligns with reality. Implementing and Verifying the Fix Once you've identified the likely culprit - say, a misconfigured or missing cache invalidation policy - the next step is to implement a fix and verify its resilience. 1. Implementing the Change: - Scoped Changes: Adjust just the component pinpointed in your experiments. Avoid large-scale refactoring that might introduce other issues. - Code Reviews: Peer reviews can catch overlooked logic gaps or confirm that your changes align with best practices. 2. Regression Testing: - Re-run the same experiments that initially exposed the issue. In our stale data example, confirm that the data remains fresh under various conditions. - Conduct broader tests - like integration or end-to-end tests - to ensure no new bugs are introduced. 3. Monitoring in Production: - Even with positive test results, real-world scenarios can differ. Monitor logs and metrics (e.g. cache hit rates, user error reports) closely post-deployment. - If the buggy behavior reappears, revisit your hypothesis or consider additional factors, such as unpredicted user behavior. 4. Benchmarking and Performance Checks (If Relevant): - When making changes that affect the frequency of certain processes - such as how often a cache is refreshed - be sure to measure the performance impact. Verify you meet any latency or resource usage requirements. - Keep an eye on the trade-offs: For instance, more frequent cache invalidations might solve stale data but could also raise system load. By systematically verifying your fix - similar to confirming experimental results in research - you ensure that you've addressed the true cause and maintained overall software stability. Documenting the Debugging Process Good science relies on transparency, and so does effective debugging. Thorough documentation guarantees your findings are reproducible and valuable to future team members. 1. Record Your Hypothesis and Experiments: - Keep a concise log of your main hypothesis, the tests you performed, and the outcomes. - A simple markdown file within the repo can capture critical insights without being cumbersome. 2. Highlight Key Evidence and Observations: - Note the logs or metrics that were most instrumental - e.g., seeing repeated stale cache hits 10 minutes after updates. - Document any edge cases discovered along the way. 3. List Follow-Up Actions or Potential Risks: - If you discover additional issues - like memory spikes from more frequent invalidation - note them for future sprints. - Identify parts of the code that might need deeper testing or refactoring to prevent similar issues. 4. Share with Your Team: - Publish your debugging report on an internal wiki or ticket system. A well-documented troubleshooting narrative helps educate other developers. - Encouraging open discussion of the debugging process fosters a culture of continuous learning and collaboration. By paralleling scientific publication practices in your documentation, you establish a knowledge base to guide future debugging efforts and accelerate collective problem-solving. Conclusion Debugging can be as much a rigorous, methodical exercise as an art shaped by intuition and experience. By adopting the principles of scientific inquiry - forming hypotheses, designing controlled experiments, gathering evidence, and transparently documenting your process - you make your debugging approach both systematic and repeatable. The explicitness and structure of scientific debugging offer several benefits: - Better Root-Cause Discovery: Structured, hypothesis-driven debugging sheds light on the _true_ underlying factors causing defects rather than simply masking symptoms. - Informed Decisions: Data and evidence lead the way, minimizing guesswork and reducing the chance of reintroducing similar issues. - Knowledge Sharing: As in scientific research, detailed documentation of methods and outcomes helps others learn from your process and fosters a collaborative culture. Ultimately, whether you are diagnosing an intermittent crash or chasing elusive performance bottlenecks, scientific debugging brings clarity and objectivity to your workflow. By aligning your debugging practices with the scientific method, you build confidence in your solutions and empower your team to tackle complex software challenges with precision and reliability. But most importantly, do not get discouraged by the number of rigorous steps outlined above or by the fact you won't always manage to follow them all religiously. Debugging is a complex and often frustrating process, and it's okay to rely on your intuition and experience when needed. Feel free to adapt the debugging process to your needs and constraints, and as long as you keep the scientific mindset at heart, you'll be on the right track....

May 9, 2025

13 mins

JavaScript

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Building a Multi-Response Streaming API with Node.js, Express, and React

Introduction

Hands-On

Implementing the Streaming API in Express

Implementing the Download API in Express

Utilizing the APIs: A React Application

Bringing It All Together

Conclusion

William Mimura

You might also like

Communication Between Client Components in Next.js

Remix's evolution to a Vite plugin

Enhancing Your Playwright Workflow: A Guide to the VSCode Extension

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging)

Let's innovate together!

You might also like

Communication Between Client Components in Next.js

Remix's evolution to a Vite plugin

Enhancing Your Playwright Workflow: A Guide to the VSCode Extension

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging)