Javascript

Creating Custom GitHub Actions

Published Oct 27, 2021

Updated Feb 13, 2023

13 min read

This article was written over 18 months ago and may contain information that is out of date. Some content may be relevant but please refer to the relevant official documentation or available resources for the latest information.

Since its generally available release in Nov 2019, Github Actions has seen an incredible increase in adoptions.

Github Actions allows you to automate, customize and execute your software development workflows.

In this article, we will learn how to create our first custom Github Actions using Typescript. We will also show some of the best practices, suggested by Github, for publishing and versioning our actions.

Types of Actions

There two types of publishable actions: Javascript and Docker actions.

Docker containers provide a more consistent and reliable work unit than Javascript actions because they package the environment with the Github Actions code. They are ideal for actions that must run in a specific configuration.

On the other hand, JavaScript actions are faster than Docker actions since they run directly on a runner machine and do not have to worry about building the Docker image every time. Javascript actions can run in Windows, Mac, and Linux, while Docker actions can just run in Linux. But most importantly (for the purpose of this article), Javascript actions are easier to write.

There is a third kind of Action: the Composite run steps Actions. These help you reuse code inside your project workflows, and hide complexity when you do not want to publish the Action to the marketplace.

You can quickly learn how to create Composite run step Actions in this video, or by reading through the docs.

The Action

For this article, we will be creating a simple Javascript Action.

We will use the Typescript Action template to simplify our setup, and use TypeScript out of the box. The objective is to walk over the whole lifecycle of creating and publishing a custom GitHub Action.

We will be creating a simple action that counts the Lines of Code (LOC) of a given type of file, and throws if the sum of LOC exceeds a given threshold.

Keep in mind that the source code is not production-ready and should only be used for learning.

The Action will receive three params:

fileOrFolderToProcess (optional): The file or folder to process
filesAndFoldersToIgnore (optional): A list of directories to ignore. It supports glob patterns.
maxCount (required): The maximum number of LOC for the sum of files.

The Action recursively iterates over all files under the folder to calculate the total amount of Lines of Code for our project.

During the process, the Actions will skip the files and folders marked to ignore, and at the end, if the max count is reached, we throw an error.

Additionally, we will set the total LOC in an Action output no matter the result of the Action.

Setting up the Environment

JavaScript Github Actions are not significantly different from any other Javascript project. We will set up some minimal configuration, but you should feel free to add your favorite workflow.

Let us start by creating the repository. As mentioned, we will use the Typescript Github Actions template, which will provide some basic configuration for us.

We start by visiting https://github.com/actions/typescript-action. We should see something like this:

The first thing we need to do is add a start to the repo :). Once that is completed, we will then click on the "Use this template" button.

We are now in a regular "create new repository" page that we must fill.

We can then create our new repository by clicking the "Create repository from template" button.

Excellent, now our repository is created. Let us take a look at what this template has provided for us.

The first thing to notice is that Github recognizes that we are in a GitHub Actions source code. Because of that, GitHub provides a contextual button to start releasing our Action.

The file that allows this integration is the action.yml file. That is the action metadata container, including the name, description, inputs, and outputs. It is also where we will reference the entry point .js for our Action.

The mentioned entry point will be located in the dist folder, and the files contained there is the result of building our Typescript files.

Important! Github uses the dist folder to run the Actions. Unlike other repositories, this build bundle MUST be included in the repository, and should not be ignored.

Our source code lives in the source folder. The main.ts is what would be compiled to our Action entry point index.js. There is where most of our work will be focused.

Additional files and configurations

In addition to the main files, the TypeScript template also adds configuration files for Jest, TypeScript, Prettier and ESLint.

A Readme template and a CODEOWNERS file are included, along with a LICENSE.

Lastly, it will also provide us with a GitHub CI YAML file with everything we need to e2e test our Action.

Final steps

To conclude our setup walkthrough, let us clone the repository. I will be using mine, but you should replace the repository with yours.

git clone https://github.com/NachoVazquez/loc-alarm.git

Navigate to the cloned project folder, and install the dependencies.

npm i

Now we are ready to start implementing our Action.

The implementation

First we must configure our action.yml file and define our API.

The metadata

The first three properties are mostly visual metadata for the Workspace, and the Actions tab.

name: 'Lines Of Code Alert'
description: 'Github Action that throws an error when the specified maximum LOC count is reached by any file'
author: 'Nacho Vazquez -- This Dot, Inc'

The name property is the name of your Action. GitHub displays the name in the Actions tab to help visually identify actions in each job.

GitHub will also use the name, the description, and the author of the Action to inform users about the Action goal in the Actions Marketplace.

Ensure a short and precise description; Doing so will help the users of the Action quickly identify the problem that the Action is solving.

Next, we define our inputs. Like we did with the Action, we should write a short and precise description to avoid confusion about the usage of each input variable.

inputs:
  fileOrFolderToProcess:
    required: false
    description: 'The file or folder to process'
    default: '.'
  filesAndFoldersToIgnore:
    required: false
    description: 'A list of directories to ignore. Supports glob patterns.'
    default: '["node_modules", ".git", "dist", ".github"]'
  maxCount:
    required: true
    description: 'The maximum number of LOC for the sum of files'

We will mark our inputs as required or optional, according to what we already specified when describing our plans for the Action.

The default values help provide pre-configured data to our Action.

As with the inputs, we must define the outputs.

outputs:
  locs:
    description: 'The amount of LOC of your project without comments and empty lines'

Actions that run later in a workflow can use the output data set in our Action run.

If you don't declare an output in your action metadata file, you can still set outputs and use them in a workflow.

However, it would not be evident for a user searching for the Action in the Marketplace since GitHub cannot detect outputs that are not defined in the metadata file.

Finally, we define the application running the Action and the entry point for the Action itself.

runs:
  using: 'node12'
  main: 'dist/index.js'

Now, let's see everything together so we can appreciate the big picture of our Action metadata.

name: 'Lines Of Code Alert'
description: 'Github Action that throws an error when the specified maximum LOC count is reached by any file'
author: 'Nacho Vazquez -- This Dot, Inc'
inputs:
  fileOrFolderToProcess:
    required: false
    description: 'The file or folder to process'
    default: '.'
  filesAndFoldersToIgnore:
    required: false
    description: 'A list of directories to ignore. Supports glob patterns.'
    default: '["node_modules", ".git", "dist", ".github"]'
  maxCount:
    required: true
    description: 'The maximum number of LOC for the sum of files'
outputs:
  locs:
    description: 'The amount of LOC of your project without comments and empty lines'
runs:
  using: 'node12'
  main: 'dist/index.js'

The Code

Now that we have defined all our metadata and made GitHub happy, we can start coding our Action.

Our code entry point is located at src/maint.ts. Let's open the file in our favorite IDE and start coding.

Let's clean all the unnecessary code that the template created for us. We will, however, keep the core tools import.

import * as core from '@actions/core'

The core library will give us all the tools we need to interact with the inputs and outputs, force the step to fail, add debugging information, and much more. Discover all the tools provided by the Github Actions Toolkit.

After cleaning up all of the example code, the initial step would be extracting and transforming our inputs to a proper form.

// extract inputs
const filesAndFoldersToIgnore = JSON.parse(
  core.getInput('filesAndFoldersToIgnore')
)
const maxCount: number = +core.getInput('maxCount')
const fileOrFolderToProcess: string = core.getInput('fileOrFolderToProcess')

With our inputs ready, we need to start thinking about counting our LOC while enforcing the input restrictions.

Luckily there is a couple of libraries that can do this for us. For this example, we will be using node-sloc, but feel free to use any other.

Go on and install the dependency using npm or any package manager that you prefer.

npm install --save node-sloc

Import the library.

import sloc from 'node-sloc'

And the rest of the implementation is straightforward.

// calculate loc stats
const stats = await sloc({
  path: fileOrFolderToProcess,
  extensions: ['ts', 'html', 'css', 'scss'],
  ignorePaths: filesAndFoldersToIgnore,
  ignoreDefault: true
})

Great! We have our LOC information ready. Let's use it to set the output defined in the metadata before doing anything else.

// set the output of the action
core.setOutput('locs', stats?.sloc)

Additionally, we will also provide debuggable data. Notice that debug information is only available if the repository owner activated debug logging capabilities.

// debug information is only available when enabling debug logging https://docs.github.com/en/actions/managing-workflow-runs/enabling-debug-logging
core.debug(`LOC ${stats?.sloc?.toString() || ''}`)
core.debug(`Max Count ${maxCount.toString() || ''}`)

Here is the link if you are interested in debugging the Action yourself.

Finally, verify that the count of the LOC is not exceeding the threshold.

// verify that locs threshold is not exceeded
if ((stats?.sloc || 0) > maxCount) {
  core.debug('Threshold exceeded')
  throw new Error(
    `The total amount of lines exceeds the maximum allowed.
     Total Amount: ${stats?.sloc}
     Max Count: ${maxCount}`
  )
}

If the threshold is exceeded, we use the core.setFailed, to make this action step fail and, therefore, the entire pipeline fails.

catch (error) {
  core.setFailed(error.message)
}

Excellent! We just finished our Action. Now we have to make it available for everyone.

But first, lets configure our CI to perform an e2e test of our Action.

Go to the file .github/workflows/*.yml. I called mine ci.yml but you can use whatever name makes sense to you.

name: 'loc-alarm CI'

on: # rebuild any PRs and main branch changes
  pull_request:
  push:
    branches:
      - main

jobs:
  build: # make sure build/ci work properly
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: npm install
      - run: npm run build

  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: ./
        id: e2e
        with:
          maxCount: '1000'
      # Use the output from the `e2e` step
      - name: Get the LOC
        run: echo "The project has ${{ steps.e2e.outputs.locs }} lines of code"

Here, we are triggering the pipeline whenever a pull request is created with base branch main or the main branch itself is pushed.

Then, we run the base setup steps, like installing the packages, and building the action to verify that everything works as it should.

Finally, we run e2e jobs that will test the actions as we were running it in an external application.

That's it! Now we can publish our Action with confidence.

Publish and versioning

Something you must not forget before any release is to build and package your Action.

npm run build && npm run package

These commands will compile your TypeScript and JavaScript into a single file bundle on the dist folder.

With that ready, we can commit our changes to the main branch, and push to origin.

Go back to your browser and navigate to the Action repository.

First, go to the Actions tab and verify that our pipeline is green and the Action is working as expected.

After that check, go back to the "Code" tab, the home route of our repository.

Remember the "Draft a release" button? Well, it is time to click it.

We are now on the releases page. This is where our first release will be created.

Click on the terms and conditions link, and agree with the terms to publish your actions.

Check the "Publish this Action to the Github Marketplace" input, and fill in the rest of the information.

You can mark this as pre-release if you want to experiment with the Action before inviting users to use it.

And that's it! Just click the "Publish release" button.

Tada! Click in the marketplace button to see how your Action looks!

After the first release is out, you will probably start adding features or fixing bugs. There are some best practices that you should follow while maintaining your versioning.

Use this guide to keep your version under control. But the main idea is that the major tag- v1 for instance- should always be referencing the latest tag with the same major version.

This means that if we release v1.9.3 we should update v1 to the same commit as v1.9.3.

Our Action is ready. The obvious next step is to test it with a real application.

Using the Action

Now it is time to test our Action, and see how it works in the wild.

We are going to use our Plugin Architecture example application. If you have read that article yet, here is the link.

The first thing we need to do is create a new git branch.

After that, we create our ci.yml file under .github/workflows.

And we add the following pipeline code.

name: Plugin Architecture CI

on:
  pull_request:
  push:
    branches:
      - main

jobs:
  loc:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: NachoVazquez/loc-alarm@v1
        id: loc-alert
        with:
          maxCount: 1000
      - name: Print the LOC
        run: echo "The project has ${{ steps.loc-alert.outputs.locs }} lines of code."

Basically, we are just triggering this Action when a PR is created using main as the base branch, or if we push directly to main.

Then, we add a single job that will checkout the PR branch and use our Action with a max count of 200.

Finally, we print the value of our output variable.

Save, commit, and push.

Create your PR, go to the check tab, and see the result of your effort.

Great! We have our first failing custom GitHub action.

Now, 200 is a bit strict. Maybe 1000 lines of code are more appropriate.

Adjust your step, commit, and push to see your pipeline green and passing.

How great is that!?

Conclusion

Writing Custom GitHub Actions using JavaScript and TypeScript is really easy, but it can seem challenging when we are not familiar with the basics.

We covered an end-to-end tutorial about creating, implementing, publishing, and testing your Custom GitHub Action.

This is really just the beginning. There are unlimited possibilities to what you can create using GitHub Actions.

Use what you learned today to make the community a better place with the tools you can create for everyone.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Nacho Vazquez
Front-end architect, tech writer, meetup speaker, scuba diving lover. Passionate about Software architecture and sharing knowledge.
@nacho_vazquez14 @NachoVazquez

Exploring Angular Forms: A New Alternative with Signals

Exploring Angular Forms: A New Alternative with Signals In the world of Angular, forms are essential for user interaction, whether you're crafting a simple login page or a more complex user profile interface. Angular traditionally offers two primary approaches: template-driven forms and reactive forms. In my previous series on Angular Reactive Forms, I explored how to harness reactive forms' power to manage complex logic, create dynamic forms, and build custom form controls. A new tool for managing reactivity - signals - has been introduced in version 16 of Angular and has been the focus of Angular maintainers ever since, becoming stable with version 17. Signals allow you to handle state changes declaratively, offering an exciting alternative that combines the simplicity of template-driven forms with the robust reactivity of reactive forms. This article will examine how signals can add reactivity to both simple and complex forms in Angular. Recap: Angular Forms Approaches Before diving into the topic of enhancing template-driven forms with signals, let’s quickly recap Angular's traditional forms approaches: 1. Template-Driven Forms: Defined directly in the HTML template using directives like ngModel, these forms are easy to set up and are ideal for simple forms. However, they may not provide the fine-grained control required for more complex scenarios. Here's a minimal example of a template-driven form: ` ` 2. Reactive Forms: Managed programmatically in the component class using Angular's FormGroup, FormControl, and FormArray classes; reactive forms offer granular control over form state and validation. This approach is well-suited for complex forms, as my previous articles on Angular Reactive Forms discussed. And here's a minimal example of a reactive form: ` ` Introducing Signals as a New Way to Handle Form Reactivity With the release of Angular 16, signals have emerged as a new way to manage reactivity. Signals provide a declarative approach to state management, making your code more predictable and easier to understand. When applied to forms, signals can enhance the simplicity of template-driven forms while offering the reactivity and control typically associated with reactive forms. Let’s explore how signals can be used in both simple and complex form scenarios. Example 1: A Simple Template-Driven Form with Signals Consider a basic login form. Typically, this would be implemented using template-driven forms like this: ` ` This approach works well for simple forms, but by introducing signals, we can keep the simplicity while adding reactive capabilities: ` ` In this example, the form fields are defined as signals, allowing for reactive updates whenever the form state changes. The formValue signal provides a computed value that reflects the current state of the form. This approach offers a more declarative way to manage form state and reactivity, combining the simplicity of template-driven forms with the power of signals. You may be tempted to define the form directly as an object inside a signal. While such an approach may seem more concise, typing into the individual fields does not dispatch reactivity updates, which is usually a deal breaker. Here’s an example StackBlitz with a component suffering from such an issue: Therefore, if you'd like to react to changes in the form fields, it's better to define each field as a separate signal. By defining each form field as a separate signal, you ensure that changes to individual fields trigger reactivity updates correctly. Example 2: A Complex Form with Signals You may see little benefit in using signals for simple forms like the login form above, but they truly shine when handling more complex forms. Let's explore a more intricate scenario - a user profile form that includes fields like firstName, lastName, email, phoneNumbers, and address. The phoneNumbers field is dynamic, allowing users to add or remove phone numbers as needed. Here's how this form might be defined using signals: ` > Notice that the phoneNumbers field is defined as a signal of an array of signals. This structure allows us to track changes to individual phone numbers and update the form state reactively. The addPhoneNumber and removePhoneNumber methods update the phoneNumbers signal array, triggering reactivity updates in the form. ` > In the template, we use the phoneNumbers signal array to dynamically render the phone number input fields. The addPhoneNumber and removePhoneNumber methods allow users to reactively add or remove phone numbers, updating the form state. Notice the usage of the track function, which is necessary to ensure that the ngFor directive tracks changes to the phoneNumbers array correctly. Here's a StackBlitz demo of the complex form example for you to play around with: Validating Forms with Signals Validation is critical to any form, ensuring that user input meets the required criteria before submission. With signals, validation can be handled in a reactive and declarative manner. In the complex form example above, we've implemented a computed signal called formValid, which checks whether all fields meet specific validation criteria. The validation logic can easily be customized to accommodate different rules, such as checking for valid email formats or ensuring that all required fields are filled out. Using signals for validation allows you to create more maintainable and testable code, as the validation rules are clearly defined and react automatically to changes in form fields. It can even be abstracted into a separate utility to make it reusable across different forms. In the complex form example, the formValid signal ensures that all required fields are filled and validates the email and phone numbers format. This approach to validation is a bit simple and needs to be better connected to the actual form fields. While it will work for many use cases, in some cases, you might want to wait until explicit "signal forms" support is added to Angular. Tim Deschryver started implementing some abstractions around signal forms, including validation and wrote an article about it. Let's see if something like this will be added to Angular in the future. Why Use Signals in Angular Forms? The adoption of signals in Angular provides a powerful new way to manage form state and reactivity. Signals offer a flexible, declarative approach that can simplify complex form handling by combining the strengths of template-driven forms and reactive forms. Here are some key benefits of using signals in Angular forms: 1. Declarative State Management: Signals allow you to define form fields and computed values declaratively, making your code more predictable and easier to understand. 2. Reactivity: Signals provide reactive updates to form fields, ensuring that changes to the form state trigger reactivity updates automatically. 3. Granular Control: Signals allow you to define form fields at a granular level, enabling fine-grained control over form state and validation. 4. Dynamic Forms: Signals can be used to create dynamic forms with fields that can be added or removed dynamically, providing a flexible way to handle complex form scenarios. 5. Simplicity: Signals can offer a simpler, more concise way to manage form states than traditional reactive forms, making building and maintaining complex forms easier. Conclusion In my previous articles, we explored the powerful features of Angular reactive forms, from dynamic form construction to custom form controls. With the introduction of signals, Angular developers have a new tool that merges the simplicity of template-driven forms with the reactivity of reactive forms. While many use cases warrant Reactive Forms, signals provide a fresh, powerful alternative for managing form state in Angular applications requiring a more straightforward, declarative approach. As Angular continues to evolve, experimenting with these new features will help you build more maintainable, performant applications. Happy coding!...

Oct 4, 2024

5 mins

AngularJavaScript

Learning Paths for Next.JS Developers with Ankita Kulkarni

In this video, Rob Ocel and co-hosts Tracy Lee, Adam Rackis, and Danny Thompson talk with tech educator Ankita Kulkarni about her journey from engineering leader to full-time educator. Ankita shares insights on teaching Next.js, bridging practical knowledge gaps, and helping developers tackle real-world challenges. They discuss Next.js as a React-based framework, its benefits, and the challenges it presents for beginners. Chapters - Introduction to the Podcast and Guests 00:01 - Meet Ankita Kulkarni, Tech Educator 00:26 - Ankita's Transition to Full-Time Education 01:41 - Teaching Practical Knowledge in Next.js 03:19 - Effective Methods for Teaching Next.js 05:27 - Challenges of Being a Full-Time Educator 07:48 - Balancing Broad and Specific Examples 09:54 - Embracing Mistakes as a Teaching Tool 12:13 - Pair Programming and Mentorship 14:00 - Discussion on Next.js and Framework Adoption 16:48 - Advantages and Challenges of Next.js 18:12 - Choosing the Right Framework for Your Needs 20:35 - Impact of Next.js in React Documentation 22:26 - Learning Paths for New Developers 23:24 - The Rise of Full-Stack Web Development 25:09 - Benefits of Frameworks Abstracting Complexity 26:27 - OpenNext and Deployment Flexibility 28:06 - Ankita's Excitement for New Next.js Features 30:35 - The Future of Next.js Without Vercel 32:16 - Final Thoughts and Where to Find Everyone Online 34:21 Follow Ankita Kulkarni on Social Media Twitter: https://x.com/kulkarniankita9 YouTube: https://www.youtube.com/@kulkarniankita Sponsored by This Dot: thisdot.co...

Nov 12, 2024

2 mins

NextJSJavaScript

Deep Dive into Node.js with James Snell

Node.js is one of the most used engines globally, and it is unique in the approach it follows to make our code work. As developers, we usually ignore how the underlying tools we use in daily work. In this article, we will deep dive into the Node.js internals using James Snell's talk as our guide, but we will expand in some areas to clarify some of the concepts discussed. We will learn about the Event Loop and the asynchronous model of Node.js. We will understand Event Emitters and how they power almost everything inside Node, and then we will build on that knowledge to understand what Streams are, and how they work. Event Loop We will begin our journey inside Node.js by exploring the Event Loop. The Event Loop is one of the most critical aspects of Node.js to understand. The Event Loop is the big orchestrator- the mechanism in charge of scheduling Node.js' synchronous and asynchronous nature. This section teaches how everything related to the scheduling mechanism works. When the Node.js process begins, it starts multiple threads: the Main process thread and the Libuv pool threads with four worker threads by default. The Libuv thread concern is handling heavy load work like IO by reading files from the disc, running some encryption, or reading from a socket. We can see the Event Loop as a glorified for/while loop that lives inside the main thread. The loop process happens at the C++ level. The Event Loop must perform several steps to complete a full back-around iteration. These steps will involve performing different checks and listening to OS events, like checking if there are any timers expired that need to be fired. It will check if there is any pending IO to process or schedule. These tasks run at the C++ level, but the event associated with the steps usually involves a callback like the action to execute when a timer expires, or a new chunk of a file is ready to be processed. When this happens, the callback executes in Javascript. Because the event loop exists in the process main thread, every time one of the steps of the event loop is processed, the event loop and the main thread are blocked until that step completes. This means that even if the Libuv is still executing the IO operations while the main thread is completing one-step tasks, the result of the IO operations is not accessible until the main thread is unblocked and iterates to the step that handles that result. "There is no such thing as asynchronous Javascript" James Snell But if asynchronous doesn't exist in Node and Javascript. What are promises for? Promises are just a mechanism to defer processing to the future. The power of Promises is the ability to schedule code execution after a specific set of events happens. But we will learn more about Promises later in this article. Step Execution Now that we understand the event loop high-level picture, it is time to go one level deeper and understand what happens in each step of the Event Loop. We learned that each step of the event loop executes in C++, but what happens if that step needs to run some Javascript? The C++ step code will use the V8 APIs to execute the passed Javascript code in a synchronous call. That Javascript code could have calls to a Node API named process.nextTick(fn), which receives another function as an argument. > "nextTick is not a good name because no tick is involved." James Snell If present, the process.nextTick(fn) appends its argument function in queue data structure. Later, we will find that there is a second queue where the Javascript code can append functions. But for simplicity, let's assume for now that there is only one queue: the nextTick queue. Once the Javascript runs and completes filling the queue through the process.nextTick method, it will return control to the C++ layer. Now it is time for Node to drain the nextTick Queue synchronously, in the order they were added, by running each of the functions added to the nextTick queue. Only when the queue is empty can the event loop move to the next step and start again with the same process. Remember everything described before runs asynchronously. > "Any time you have Javascript running, anything else is happening" James Snell Therefore, the key to keeping your Javascript performant is to keep your functions small, and use a scheduling mechanism to defer work. But what is the scheduling mechanism? The scheduling mechanisms are the instruments through which Node.js can simulate asynchronicity by scheduling the execution of a given Javascript function to a given time in the future. The scheduling mechanisms are the nextTick queue and the Microtask queue. The difference between these two is the order in which they execute. NodeJS will only start draining the Microtasks queue after the nextTick queue is empty. And the nextTick queue after the call stack is empty. The call stack is a LIFO (Last In, First Out) stack data structure. The operations from the stack are completely synchronous and are scheduled to run ASAP. The v8 API that we saw before runs the Javascript code sent from the C++ layer by adding and removing the statements into the stack, and executing them as corresponding. We saw how the nextTick queue is filled by the V8 execution when processing one statement from the stack, and how it drains as soon as C++ processes the Javascript stack. The Microtask queue is the one that process Promises continuation like events, catches, and finallies in V8. It is drained immediately after the nextTick queue is empty. Let's paint a picture of how this works. The following represents a function body executing several tasks. ` But the final order in which Node will execute this code looks more like ` Notice how nextTick is processed after the stack of operations is emptied, and the Microstask operations (the promise continuations) are deferred to just after the nextTick queue is emptied. However, there are other scheduling mechanisms: - Timers like setTimeout, setInterval, etc - Inmediates, the execution order of which is combined with timers, but they are not timers. A setInmediate(fn) registers a callback that will execute at the end of the current event loop turn, or just before the next event loop turn starts. - Promises, which are not the same as promises event continuation, which is what the Microstask handles. - Callbacks on the native layers this is something that is not necessarily available to the javascript developer. - Workers threads are separate node instances with their own process main thread, their own Libuv threads, and their own event loop. It is basically a complete Node instance running in a separate thread. Because it is a separate Node instance, it can run Javascript Node in parallel to the main Node instance. > "NextTick and Immediate names should be inverted because NextTick operations happen immediately after the Javascript stack is empty and Immediate functions will happen just before the nextTick start." To find complementary resources, you can go to the Node.js documentation. Event Emitter Event emitters are one of the first Node APIs ever created. Almost everything in Node is an event emitter. It is the first thing that loads and is fundamental to how Node works. If you see an object with the following methods, you have an event emitter. ` But how do these work? Inside the event emitter object, we have a map/look-up table, which is just another object. Inside each map entry, the key is the event name, and the value is an array of callback functions. The on method on the event emitter object receives the event name as the first argument, and a callback function as its second. When called, this method adds the on callback function into the corresponding array of the look-up table. The once method behaves like the on method, but with a key difference; instead of storing the once callback function directly, it will wrap it inside another function, and then add the wrapped function to the array. When the wrapper function gets executed, it will execute the wrapped callback function, removing itself from the array. On the other hand, emit uses the event name to access the corresponding array. It will create a backup copy of the array, and iterate synchronously over each callback function of the array executing it. It is important to highlight that emit is synchronous. If any of the callback functions on the array take a long time to execute, the main thread will be blocked. Emit won't return until all of functions on the array have executed. The asynchronous behavior of the emit is an illusion. This effect is caused because internally, Node will invoke each callback function using a nextTick, and therefore deferring the function's execution into the future. You can find more info about Event Emitter at the Node.js documentation. Streams (Node | Web) Streams are one of the fundamental concepts that power Node.js applications. They are a way to handle reading/writing files, network communications, or any end-to-end information exchange efficiently. Streams are not a concept unique to Node.js. They were introduced in the Unix operating system decades ago, and programs can interact with each other by passing streams through the pipe operator (|). For example, traditionally, when you tell the program to read a file, it is read into memory, from start to finish, and then you process it. You read it piece by piece using streams, processing its content without keeping it all in memory. The Node.js stream module provides the foundation upon which all streaming APIs are built. All streams are instances of EventEmitter. Node Streams There are four Node stream types: Readable, Writable, Duplex, and Transform. All of them are Event Emitters. Readable: a stream you can pipe from but not pipe into (you can receive data, but not send data to it). When you push data into a readable stream, it is buffered until a consumer starts to read the data. Writable: a stream you can pipe into but not pipe from (you can send data but not receive from it). Duplex: a stream you can both pipe into and pipe from. Basically a combination of a Readable and Writable stream. Transform: a Transform stream is similar to a Duplex, but the output is a transform of its input. Readable Stream The Readable stream works through a queue and the highwatermark in an oversimplified way. The highwatermark will delimit how much data can be in the queue, but the Readable Stream will not enforce that limit. Every time data is pushed into the queue, the stream will give feedback to the client code, telling if the highwatermark was reached or not, and transferring the responsibility to the client. The client code must decide if it continues pushing data and overflowing the queue. This feedback is received through the push method, which is the mechanism for pushing data into the queue. If the push method returns true, the highwatermark has not been reached, and you can push more. If the push method returns false that means that the highwatermark has been reached, and you are not supposed to push more, but you can. Events When data has been pushed into the queue, the internal of the Readable Stream will emit a couple of events. The on:readable is part of the pull model; it alerts that this stream is readable and has data to be read. If you are listening to the on:readable event, a read method can be called. When the client code calls the read() method, it will get a chunk of data out of the queue, and it will dequeue it. Then, the client code can keep calling read until there is no more data in the queue. After emptying the queue, the client code can restart pulling data when the on:readable event triggers again. The on:read event is part of the push model. In this event, any new chunk of data that is pushed using the push method will be synchronously sent to its listeners. That means we don't need to call the read method. The data will arrive automatically. However, there is another crucial difference; the sent data will never be stored in the queue. The queue is only filled when there is no active on:read event listener. This is called the "flow" mode because the data just flows, and it doesn't get stored. Other events are the end event that would notice that there is no more data to be consumed from the stream. The 'close' event is emitted when the stream and any underlying resources (a file descriptor, for example) have been closed. The event indicates that no more events will be emitted, and no further computation will occur. The 'error' event may be emitted by a Readable implementation. Typically, this may occur if the underlying stream cannot generate data due to an underlying internal failure or when a stream implementation attempts to push an invalid chunk of data. The key to understanding the Readable Streams and their events is that the Readable Streams are just Event Emitters. Like Event Emitters, the Readable Streams don't have anything built into it that is asynchronous. It will not call any of the scheduling mechanisms. Therefore they operate purely synchronously, and to obtain an asynchronous behavior, the client code needs to defer when to call the different event APIs like the read() and push() methods. Let's understand how! When creating a Readable Stream, we need to provide a read function. The stream will repeatedly call this read function as long as its buffer is not full; however, after it calls it once, it will wait until we call push before calling it again. If we synchronously call push after the Readable Stream called read and the buffer is not full, the stream will synchronously call read again, up until we call push(null), marking the end of the stream. Otherwise, we can defer the push calls to some other point in time, effectively making the Stream read calls operate asynchronously; we might, for example, wait for some File IO callback to return. Examples Creating a Readable Stream ` ` Using a Readable Stream ` Writable Stream The Writable Streams works similarly. When creating a Writable Stream, we need to provide a _write(chunk, encoding, callback) function, and it will be an external write(chunck) function. When the external write function is called, it will just call the internal _write function. The internal _write function must call its callback argument function. Let's imagine that we write ten chunks; when the first chunk is written, if the internal _write function doesn't invoke the callback function, what will happen is that the chunks will accumulate in the Writable Stream internal buffer. When the callback is invoked, it will grab the next chunk and write it, draining the buffer. This means that if the _write is calling the callback function synchronously, then all those writes will happen synchronously; if it is deferring invoking that callback, then all those calls will happen asynchronously, but the data will accumulate in the internal buffer up until the highwatermark is hit. Even then, you can decide to keep incrementation the buffer queue. Events Like the Readable Stream, the Writable Stream is an Event Emitter that will provide its events listeners with useful alerts. The on:drain event notifies that the Writable buffer is longer, and more writes are allowed. If a call to the write function returns false, indicating backpressure, the 'drain' event will be emitted when it is appropriate to resume writing data to the stream. The 'close' event is emitted when the stream and any underlying resources (a file descriptor, for example) have been closed. The event indicates that no more events will be emitted, and no further computation will occur. The 'error' event is emitted if an error occurs while writing or piping data. The listener callback is passed a single Error argument when called. The 'finish' event is emitted after the stream.end() method has been called, and all data has been flushed to the underlying system. Examples Creating a Writable Stream ` Duplex Stream Duplex streams are streams that implement both the Readable and Writable interfaces. We can see a Duplex stream as an object containing both a Readable Stream and a Writable Stream. These two internal objects are not connected, and each has independent buffers. They share some events; for example, when the close event emits, both Streams are closed. They still emit the stream type-specific events like the readable and read for the Readable Stream, or the drain for the Writable Stream independently. However, this doesn't mean that data will be available on the Readable stream if we write into the Writable Stream. Those are independent. Duplex streams are especially useful for Sockets. Transform Stream Transform streams are Duplex streams where the output is related to the input. Contrary to the Duplex streams, when we write some chunk of data into the Writable stream, that data will be passed to the Readable stream by calling a transform function and invoking the push method on the readable side with the transformed data. As with Duplex, both the Readable and the Writable sides will have independent buffers with their own highwatermark. Examples Creating a Transform Stream ` Web Streams In Web streams, we still see some of the concepts we have seen for Node Streams. There exist the Readable, Writable, Duplex, and Transform streams. However, they also have significant differences. Unlike Node streams, web streams are not Event Emitter-based but Promise-based. Another big difference is that Node streams support multiple event listeners at the same time, and each of those listeners will get a copy of the data, while Web streams are restricted to a single listener at a time, and it has no events in it; it is purely Promise-based. While Node streams work entirely synchronously because Web streams are Promise-based, they work entirely asynchronously. This happens because the continuation is always deferred to the MicroTasks queue. It is essential to highlight that Node streams are significantly faster than Web streams, while Web streams are much more portable than Node streams. So keep that in mind when you are planning to select one or the other. Readable stream To understand the underlying implementation of how this works, let's take the Readable stream and dissect it. The developer can pass the underlayingSource object to the constructor when creating a new Readable stream. We can define the pull(controller) function inside this object, which receives the Controller object as its first and only parameter. This is the function where the client code can push the data into the stream by defining a custom data pulling logic. The Controller object, argument of the pull() function, contains the enqueue() function. This is the equivalent to the Node Readable stream push() function, and it is used to push the callback function return data into the Readable stream. The reader object, is the element through which the Web Readable stream enforces a single listener or reader at the time. It can be accessed through the getReader() method of the stream. Inside the reader object, we can find the read() function, which returns a promise providing access to the next chunk in the stream's internal queue. When the client code of the stream calls read() from the reader object, this will look into the Controller data queue and check if there is any data. If there is data in the queue, it will only dequeue the first chunk of data from the queue and resolve the read promise. If there is no data in the queue, the stream is going to call the pull(controller) function defined in the stream constructor argument object. Then the pull(controller) function will run and as part of its execution, it will call the Controller function enqueue() to push data into the stream. After pushing the data in, the pull function will resolve the initial read() Promise with the data pushed in. Because the pull(controller) function can be an async function, and return a Promise, if it calls once and that Promise is not resolved yet, and the client code continues calling the read() function, those read promises are going to accumulate in a read queue. Now we have our data queue and the read queue, and every time a new data is enqueued into the data queue, the stream logic will check if there is some pending read promise in the read queue. If the read queue is not empty, the first pending Promise will be dequeued and resolved with the data enqueued in the data queue. We can see this process as the read queue and the data queue trying to balance each other out. Consequently, the read queue will not accumulate read promises if the data queue has data. And the opposite is also true; we will not accumulate data in the data queue if the read queue has unresolved promises. Example Creating a readable stream ` Conclusions There is immense beauty in the depths for those curious developers who are not afraid of diving into the deep waters of the Node.js inner mechanics. In this article, we barely start exploring some of the solutions chosen by the Node.js core contributors, and we have found how elegant and straightforward many of those solutions are. There is still much to learn about the internals of Node and the Browser engines, but I think this article and its companion Javascript Marathon are a great starting point. If you get here, thank you, and I hope you got inspired to continue digging into the tools that allow us to unleash our passion and creativity....

May 24, 2022

16 mins

NodeJS

How Nuxt Studio is Redefining Developer & User Experience with Baptiste Leproux & Ferdinand Coumau

In this episode of the Modern Web Podcast, Danny Thompson sits down with Ferdinand Como and Baptiste Leproux from Nuxt Labs to uncover the story behind Nuxt Studio, a tool that's transforming how developers and non-technical users interact with Nuxt applications. Ferdinand and Baptiste share how Nuxt Studio bridges the gap between developer customization and user-friendly content management. Built to empower agencies, freelancers, and their clients, Nuxt Studio combines powerful features like live previews, Vue component integration, and schema-driven forms to make managing content seamless. The conversation also explores the broader mission of Nuxt Labs—building sustainable open-source tools that enhance developer experience and meet real-world needs. With insights into the future of Nuxt Studio and its potential to scale for larger organizations, this episode is a must-listen for anyone passionate about innovation in web development. Chapters 1. Introduction and Setting the Stage (00:00:00) 2. The Vision Behind Nuxt Studio (00:03:10) 3. Nuxt Studio’s Core Features (00:08:45) 4. Challenges in Building Nuxt Studio (00:16:20) 5. Target Audience and Use Cases (00:22:35) 6. Sustainability in Open Source (00:29:00) 7. The Future of Nuxt Studio (00:35:10) 8. Nuxt Studio’s Role in the Nuxt Ecosystem (00:42:30) 9. Closing Thoughts and What’s Next (00:48:00) 10. Sponsor Shoutout and Wrap-Up (00:53:20) Follow Baptiste Leproux and Ferdinand Coumau Baptiste Twitter: https://x.com/_larbish Ferdinand Twitter: https://x.com/CoumauFerdinand Baptiste Linkedin: https://www.linkedin.com/in/baptiste-leproux-618842b0/ Ferdinand Linkedin: https://www.linkedin.com/in/ferdinand-coumau-nuxt/ Sponsored by This Dot: thisdot.co...

Nov 20, 2024

2 mins

VueNuxtJS

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Creating Custom GitHub Actions

Types of Actions

The Action

Setting up the Environment

Additional files and configurations

Final steps

The implementation

The metadata

The Code

Publish and versioning

Using the Action

Conclusion

Nacho Vazquez

You might also like

Exploring Angular Forms: A New Alternative with Signals

Learning Paths for Next.JS Developers with Ankita Kulkarni

Deep Dive into Node.js with James Snell

How Nuxt Studio is Redefining Developer & User Experience with Baptiste Leproux & Ferdinand Coumau

Let's innovate together!

You might also like

Exploring Angular Forms: A New Alternative with Signals

Learning Paths for Next.JS Developers with Ankita Kulkarni

Deep Dive into Node.js with James Snell

How Nuxt Studio is Redefining Developer & User Experience with Baptiste Leproux & Ferdinand Coumau