General

Ensuring Accurate Workflow Status in GitHub for Enhanced Visibility

Published Mar 22, 2024

Updated Mar 22, 2024

3 min read

Introduction

In the world of software development, GitHub workflows are crucial for automating CI/CD processes. However, a key challenge emerges when these workflows report a 'success' despite underlying issues, like failed tests. This is especially common in scenarios involving tests (e.g., Cypress) and notifications (e.g., Slack) within the same workflow. This blog post aims to highlight the importance of accurate GitHub workflow statuses for better visibility and team response, and how to ensure your workflows reflect the true outcome of their runs.

The Problem with Misleading Workflow Statuses

Consider a scenario in a GitHub workflow where end-to-end tests are run using Cypress.

jobs:
  cypress_test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Use Node.js version from .nvmrc
        uses: actions/setup-node@v4
        with:
          node-version-file: ".nvmrc"

      - name: Clean install
        run: npm ci

      - name: Run Cypress tests
        id: cypress_run
        run: npm run cy:run
        continue-on-error: true

      - name: Send Slack message on fail
        if: steps.cypress_run.outcome == 'failure'
        uses: slackapi/slack-github-action@v1.24.0
        ... # Slack action configuration

If these tests fail, but the workflow proceeds to a subsequent step, like sending a notification via Slack, which completes successfully, the entire workflow might still show a green checkmark. This misleading success status suggests everything is functioning as intended, when in fact, there could be significant underlying issues.

The core issue is the determination of workflow success. Even if critical steps like testing fail, later steps without errors can override this, resulting in a false sense of security. This not only delays bug detection but can also lead to faulty code advancing in the CI/CD pipeline. It's crucial for the overall workflow status to accurately reflect failures in critical steps to ensure prompt and appropriate responses to issues.

Crafting a Solution and Best Practices

Ensuring Accurate Status Reporting

To address the issue of misleading workflow statuses, it’s essential to configure your GitHub Actions properly. The goal is to ensure that the workflow accurately reflects the success or failure of critical tasks, such as running tests, regardless of the success of subsequent steps.

Adjusting the Workflow

Conditional Notifications: First, set up notifications to execute conditionally based on the outcome of critical steps. This ensures you're alerted of the workflow status without altering the overall result. For example, sending a Slack message if a Cypress test fails:

- name: Send Slack message on fail
  if: steps.cypress_run.outcome == 'failure' # conditional test from cypress execution
  uses: slackapi/slack-github-action@v1.24.0
  ... # Slack action configuration

Explicit Failure Handling: After configuring conditional notifications, explicitly handle failure scenarios. If a critical step like a Cypress test fails, force the workflow to exit with a failure status. This step is crucial to ensure that the overall workflow reflects the true status:

- name: Force failed job if Cypress failed
  if: steps.cypress_run.outcome == 'failure'
  run: exit 1

Best Practices:

Clear Step Separation: Clearly separate and label each step in your workflow for easier readability and troubleshooting. Regular Reviews: Periodically review your workflows to ensure they are aligned with the latest project requirements and best practices. Document Workflow Logic: Maintain documentation for your workflows, especially for complex ones, to aid in understanding and future modifications.

By first setting up conditional notifications and then enforcing explicit failure handling, you maintain both alertness to issues and accuracy in workflow status. This approach ensures that failures in critical steps like tests are not overshadowed by subsequent successful steps, keeping the reported status of your workflow true to its actual state.

Conclusion

Accurate GitHub workflow statuses are vital for a transparent and efficient CI/CD process. By implementing conditional notifications and explicit failure handling, we ensure that our workflows truthfully represent the success or failure of critical tasks. This not only fosters better issue awareness and response but also upholds the integrity of our development practices. Embrace these steps as part of your commitment to maintaining clear and reliable automation processes in your projects. Happy coding!

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

William Mimura
@mimurawil

Integrating Playwright Tests into Your GitHub Workflow with Vercel

Vercel previews offer a great way to test PRs for a project. They have a predefined environment and don’t require any additional setup work from the reviewer to test changes quickly. Many projects also use end-to-end tests with Playwright as part of the review process to ensure that no regressions slip uncaught. Usually, workflows configure Playwright to run against a project running on the GitHub action worker itself, maybe with dependencies in Docker containers as well, however, why bother setting that all up and configuring yet another environment for your app to run in when there’s a working preview right there? Not only that, the Vercel preview will be more similar to production as it’s running on the same infrastructure, allowing you to be more confident about the accuracy of your tests. In this article, I’ll show you how you can run Playwright against the Vercel preview associated with a PR. Setting up the Vercel Project To set up a project in Vercel, we first need to have a codebase. I’m going to use the Next.js starter, but you can use whatever you like. What technology stack you use for this project won’t matter, as integrating Playwright with it will be the same experience. You can create a Next.js project with the following command: ` If you’ve selected all of the defaults, you should be able to run npm run dev and navigate to the app at http://localhost:3000. Setting up Playwright We will set up Playwright the standard way and make a few small changes to the configuration and the example test so that they run against our site and not the Playwright site. Setup Playwright in our existing project by running the following command: ` Install all browsers when prompted, and for the workflow question, say no since the one we’re going to use will work differently than the default one. The default workflow doesn’t set up a development server by default, and if that is enabled, it will run on the GitHub action virtual machine instead of against our Vercel deployment. To make Playwright run tests against the Vercel deployment, we’ll need to define a baseUrl in playwright.config.ts and send an additional header called X-Vercel-Protection-Bypass where we'll pass the bypass secret that we generated earlier so that we don’t get blocked from making requests to the deployment. I’ll cover how to add this environment variable to GitHub later. ` Our GitHub workflow will set the DEPLOYMENT_URL environment variable automatically. Now, in tests/example.spec.ts let’s rewrite the tests to work against the Next.js starter that we generated earlier: ` This is similar to the default test provided by Playwright. The main difference is we’re loading pages relative to baseURL instead of Playwright’s website. With that done and your Next.js dev server running, you should be able to run npx playwright test and see 6 passing tests against your local server. Now that the boilerplate is handled let’s get to the interesting part. The Workflow There is a lot going on in the workflow that we’ll be using, so we’ll go through it step by step, starting from the top. At the top of the file, we name the workflow and specify when it will run. ` This workflow will run against new PRs against the default branch and whenever new commits are merged against it. If you only want the workflow to run against PRs, you can remove the push object. Be careful about running workflows against your main branch if the deployment associated with it in Vercel is the production deployment. Some tests might not be safe to run against production such as destructive tests or those that modify customer data. In our simple example, however, this isn’t something to worry about. Installing Playwright in the Virtual Machine Workflows have jobs associated with them, and each job has multiple steps. Our test job takes a few steps to set up our project and install Playwright. ` The actions/checkout@v4 step clones our code since it isn’t available straight out of the gate. After that, we install Node v22 with actions/setup-node@v4, which, at the time of writing this article, is the latest LTS available. The latest LTS version of Node should always work with Playwright. With the project cloned and Node installed, we can install dependencies now. We run npm ci to install packages using the versions specified in the lock file. After our JS dependencies are installed, we have to install dependencies for Playwright now. sudo npx playwright install-deps installs all system dependencies that Playwright needs to work using apt, which is the package manager used by Ubuntu. This command needs to be run as the administrative user since higher privilege is needed to install system packages. Playwright’s dependencies aren’t all available in npm because the browser engines are native code that has native library dependencies that aren’t in the registry. Vercel Preview URL and GitHub Action Await Vercel The next couple of steps is where the magic happens. We need two things to happen to run our tests against the deployment. First, we need the URL of the deployment we want to test. Second, we want to wait until the deployment is ready to go before we run our tests. We have written about this topic before on our blog if you want more information about this step, but we’ll reiterate some of that here. Thankfully, the community has created GitHub actions that allow us to do this called zentered/vercel-preview-url and UnlyEd/github-action-await-vercel. Here is how you can use these actions: ` There are a few things to take note of here. Firstly, some variables need to be set that will differ from project to project. vercel_app in the zentered/vercel-preview-url step needs to be set to the name of your project in Vercel that was created earlier. The other variable that you need is the VERCEL_TOKEN environment variable. You can get this by going to Vercel > Account Settings > Tokens and creating a token in the form that appears. For the scope, select the account that has your project. To put VERCEL_TOKEN into GitHub, navigate to your repo, go to Settings > Secrets and variables > Actions and add it to Repository secrets. We should also add VERCEL_AUTOMATION_BYPASS_SECRETl. In Vercel, go to your project then navigate to Settings > Deployment Protection > Protection Bypass for Automation. From here you can add the secret, copy it to your clipboard, and put it in your GitHub action environment variables just like we did with VERCEL_TOKEN. With the variables taken care of, let’s take a look at how these two steps work together. You will notice that the zentered/vercel-preview-url step has an ID set to vercel_preview_url. We need this so we can pass the URL we receive to the UnlyEd/github-action-await-vercel action, as it needs a URL to know which deployment to wait on. Running Playwright After the last steps we just added, our deployment should be ready to go, and we can run our tests! The following steps will run the Playwright tests against the deployment and save the results to GitHub: ` In the first step, where we run the tests, we pass in the environment variables needed by our Playwright configuration that’s stored in playwright.config.ts. DEPLOYMENT_URL uses the Vercel deployment URL we got in an earlier step, and VERCEL_AUTOMATION_BYPASS_SECRET gets passed the secret with the same name directly from the GitHub secret store. The second step uploads a report of how the tests did to GitHub, regardless of whether they’ve passed or failed. If you need to access these reports, you can find them in the GitHub action log. There will be a link in the last step that will allow you to download a zip file. Once this workflow is in the default branch, it should start working for all new PRs! It’s important to note that this won’t work for forked PRs unless they are explicitly approved, as that’s a potential security hazard that can lead to secrets being leaked. You can read more about this in the GitHub documentation. One Caveat There’s one caveat that is worth mentioning with this approach, which is latency. Since your application is being served by Vercel and not locally on the GitHub action instance itself, there will be longer round-trips to it. This could result in your tests taking longer to execute. How much latency there is can vary based on what region your runner ends up being hosted in and whether the pages you’re loading are served from the edge or not. Conclusion Running your Playwright tests against Vercel preview deployments provides a robust way of running your tests against new code in an environment that more closely aligns with production. Doing this also eliminates the need to create and maintain a 2nd test environment under which your project needs to work....

Feb 25, 2025

7 mins

VercelPlaywrightGitHub

OAuth2 for JavaScript Developers

OAuth2 for JavaScript Developers OAuth is a staple of modern web development. It's the magic behind the "Log in with Facebook" or "Connect to Google" buttons you see everywhere. But what exactly is it, and how does it work, especially for a JavaScript developer? Let's dive in, using GitHub as our primary example. What is OAuth? At its core, OAuth (Open Authorization) is a protocol for authorization. It allows third-party applications to access user data without needing the user's password. Instead of users sharing their password with an app, they are granted a token that the app can use to act on their behalf. This offers several benefits; the most significant is that the app doesn't need to store user passwords, which greatly reduces potential attack vectors. Although OAuth is often used interchangeably with OAuth2, they are distinct. OAuth 1.0, introduced in 2007, is the first version of the protocol. While it provided a solid method for consumers to access protected resources without user credentials, it had complexities, such as the need for cryptographic libraries for request signing. Recognizing these challenges, the community introduced OAuth2 in 2012. OAuth2 simplified many aspects, eliminating the need for cryptographic signatures and emphasizing bearer tokens, which are easier to manage. OAuth2 provides flexibility with various methods of obtaining access tokens suitable for different application types. OAuth2 Steps Explained Using GitHub To explain how OAuth2 works, it's best to describe the entire process step-by-step. It's challenging to do this without code, so we'll reference our starter.dev backend showcase, which illustrates integrating a serverless backend with GitHub as an OAuth2 provider. Although the steps below are tailored to GitHub's OAuth2 implementation, it's important to note that the core flow is consistent across most OAuth2 services. For most of these services, you'll only need to adjust elements like URLs, query parameter names, and so on. Also, please note that our starter.dev backend showcase is designed to be deployed on Netlify. As such, all the API endpoints run as Netlify functions. When running the app locally, you must prepend the URL path with /.netlify/functions/server to access these endpoints. This step is unnecessary for the deployed version; API endpoints can be accessed directly (e.g., /api/auth/signin instead of /.netlify/functions/server/api/auth/signin) due to the redirects configured in the netlify.toml file. Step 0: Creating the App on GitHub The first step is to create your app with the provider. For GitHub, this is done on the Settings / Developer Settings / OAuth Apps page. This step is essential so that the provider can identify your app and possibly manage service-level parameters, like rate limits. In the screenshot above, when registering our app on GitHub, the authorization callback URL is a local URL. This is suitable for local development. However, in production, it should be a public API endpoint accessible to GitHub, like an API endpoint on Netlify. After registering the app, you'll receive a public client ID for your app and will need to generate a client secret (which should stay private). The client secret, used alongside the client ID, helps authenticate your application when trading an authorization code for an access token. Essentially, the client secret assures GitHub that the request for an access token genuinely comes from your app and not a malicious actor. Step 1: App Sends User to GitHub for Authorization The first step in the OAuth2 flow typically involves the user clicking a "Log in with GitHub" button on our web app or something similar. This action redirects the user from the web app to the OAuth2 provider's page (in this case, GitHub). For GitHub, the base URL for such a page is https://github.com/login/oauth/authorize, followed by query parameters that provide more details about the app. These parameters usually include the client ID — a mandatory identifier for your app — and several optional parameters, such as the redirect URI (where GitHub will send the user back) and scope (defining the permissions you're requesting). In our starter.dev example, clicking the "Log in with GitHub" button should lead the user to the /api/auth/signin route in our app. This route generates the GitHub authorization URL, populating it with all the necessary query parameters, and then redirects the user to that URL using a standard HTTP 303 redirect. ` Step 2: User Grants Permission In this step, the user arrives at the provider's authorization page and sees a prompt asking if they allow your app to access the specified data. The user can either approve or decline this request. For our scenario, this page would appear as follows: If either the cancel or authorize button is clicked, GitHub will invoke the callback URL specified when registering the app on GitHub. However, if the authorize button is clicked, GitHub will also send a one-time-use code query parameter that can be used to get the access token. Step 3: App Receives an Authorization Code After the user grants permission, GitHub redirects them to the specified callback URL. Attached to this URL is a one-time-use code query parameter. ` Step 3: App Exchanges Authorization Code for Access Token With this code, the app needs to make a POST request to the GitHub API to exchange the code for an access token. As shown below, the app sends the client ID, client secret, and the one-time code to GitHub and receives the access token in return. The access token should then be persisted in a store (such as Redis) or sent to the frontend as a cookie, as shown below: ` Step 4: App Accesses User Data with the Access Token Once you have the access token, you can use it to make authorized API requests on behalf of the user. All the app needs to do is include the access token in the Authorization header. ` And that is all. As long as the access token is valid, you can access any of the API services that were granted by the user in the authorization step. Conclusion OAuth2, though appearing complex initially, provides a strong method for authentication and authorization. For JavaScript developers, knowing its process helps in easy third-party integrations, giving users a safe way to access different services. Whether using GitHub or another platform, the main concepts are the same, allowing for better web applications. Check out our starter.dev backend showcase for the full code! There's also an associated StackBlitz project if you want to play with it....

Jan 19, 2024

5 mins

GitHubJavaScriptOAuth 2

Our Journey from Cypress to Playwright for E2E Testing

Our Journey from Cypress to Playwright for E2E Testing Introduction Adapting to unexpected technical challenges often leads to innovation. Our team faced such a moment when we discovered that our primary end-to-end (E2E) testing framework, Cypress, was incompatible with an essential component of our tech stack. This discovery led us to consider alternatives to resolve the compatibility issue and enhance our testing processes. Our choice was Playwright, a modern testing framework acclaimed for its robust features and compatibility with multiple browsers. The shift to Playwright was swift and efficient: all our E2E tests were successfully migrated within a few hours. This rapid transition highlighted our team's ability to adapt and integrate new technologies quickly. In this blog post, we will share the hands-on process we followed during the migration to Playwright, the challenges we encountered along the way, and the solutions we implemented to overcome them. Using a Test Conversion Tool We utilized a third-party test conversion tool to facilitate our migration from Cypress to Playwright, which significantly streamlined the process. Among several options available, we chose Ray's Cypress to Playwright Conversion Tool for its ease of use and effectiveness. This tool allowed us to automatically transpose our existing Cypress tests into the Playwright format. The tool processes each test script by identifying equivalent functions and commands in Playwright, translating our entire suite with minimal input. Below are some before and after images of our test scripts, demonstrating the tool's conversion capabilities. One test script written in Cypress: ` And its result converted in Playwright: ` While the tool performed impressively, converting most of our tests accurately had limitations. Certain custom helper functions and test aliases did not translate directly, and the tool clearly marked these for manual intervention. This feature was beneficial as it allowed us to quickly identify and address the segments that required our attention. Small Fixes and Fine-Tuning The initial output from the conversion tool was highly useful, yet it required some tweaks to align with Playwright's capabilities perfectly. The tool conveniently marked each unconverted function with the identifier FIXME_, making it straightforward to address these specific areas. Here’s how we tackled some of these functions: FIXME_visitWithTimeout: Originally a custom Cypress command designed to visit a URL with a specified timeout (cy.visit(url, { timeout: 30000 })), this was straightforwardly converted to Playwright's page.goto(url) FIXME_getDataCy: Another custom command (cy.get('[data-cy=""]')) was efficiently translated to Playwright’s page.locator('[data-testid=""]') FIXME_scrollTo: This native Cypress command, which scrolls the window to a specific position, proved unnecessary in Playwright. Given Playwright’s automatic detection of elements on the page, we opted to remove this command entirely ` Further refinements were made to the converted code, such as removing some "wait/delay" commands. Playwright’s robust handling of asynchronous events allowed us to streamline our tests by eliminating unnecessary pauses enhancing test execution speed and reliability. Conclusion The transition from Cypress to Playwright in our testing framework was driven by necessity but has significantly improved our testing practices. Utilizing a test conversion tool allowed us to migrate our entire suite of E2E tests efficiently, minimizing manual effort and accelerating the adoption of Playwright. Our experience highlights the importance of flexibility and the willingness to embrace new technologies in the face of unexpected challenges. The marked improvements in test execution speed and reliability, along with Playwright's advanced features, have made this transition a resounding success for our team. As we continue to refine our tests and explore the full capabilities of Playwright, we encourage other teams facing similar challenges to consider this approach. The tools and processes outlined here can serve as a roadmap for those looking to enhance their automated testing solutions....

May 31, 2024

3 mins

CypressTestingPlaywright

Internationalization in Next.js with next-intl

Internationalization in Next.js with next-intl Internationalization (i18n) is essential for providing a multi-language experience for global applications. next-intl integrates well with Next.js’ App Router, handling i18n routing, locale detection, and dynamic configuration. This guide will walk you through setting up i18n in Next.js using next-intl for URL-based routing, user-specific settings, and domain-based locale routing. Getting Started First, create a Next.js app with the App Router and install next-intl: ` Next, configure next-intl in the next.config.ts file to provide a request-specific i18n configuration for Server Components: ` Without i18n Routing Setting up an app without i18n routing integration can be advantageous in scenarios where you want to provide a locale to next-intl based on user-specific settings or when your app supports only a single language. This approach offers the simplest way to begin using next-intl, as it requires no changes to your app’s structure, making it an ideal choice for straightforward implementations. ` Here’s a quick explanation of each file's role: * translations/: Stores different translations per language (e.g., en.json for English, es.json for Spanish). Organize this as needed, e.g., translations/en/common.json. * request.ts: Manages locale-based configuration scoped to each request. Setup request.ts for Request-Specific Configuration Since we will be using features from next-intl in Server Components, we need to add the following configuration in i18n/request.ts: ` Here, we define a static locale and use that to determine which translation file to import. The imported JSON data is stored in the message variable, and is returned together with the locale so that we can access them from various components in the application. Using Translation in RootLayout Inside RootLayout, we use getLocale() to retrieve the static locale and set the document language for SEO and pass translations to NextIntlClientProvider: ` Note that NextIntlClientProvider automatically inherits configuration from i18n/request.ts here, but messages must be explicitly passed. Now you can use translations and other functionality from next-intl in your components: ` In case of async components, you can use the awaitable getTranslations function instead: ` And with that, you have i18n configured and working on your application! \ Now, let’s take it a step further by introducing routing. \ With i18n Routing To set up i18n routing, we need a file structure that separates each language configuration and translation file. Below is the recommended structure: ` We updated the earlier structure to include some files that we require for routing: * routing.ts: Sets up locales, default language, and routing, shared between middleware and navigation. * middleware.ts: Handles URL rewrites and locale negotiation. * app/[locale]/: Creates dynamic routes for each locale like /en/about and /es/about. Define Routing Configuration in i18n/routing.ts The routing.ts file configures supported locales and the default locale, which is referenced by middleware.ts and other navigation functions: ` This configuration lets Next.js handle URL paths like /about, with locale management managed by next-intl. Update request.ts for Request-Specific Configuration We need to update the getRequestConfig function from the above implementation in i18n/request.ts. ` Here, request.ts ensures that each request loads the correct translation files based on the user’s locale or falls back to the default. Setup Middleware for Locale Matching The middleware.ts file matches the locale based on the request: ` Middleware handles locale matches and redirects to localized paths like /en or /es. Updating the RootLayout file Inside RootLayout, we use the locale from params (matched by middleware) instead of calling getLocale() ` The locale we get from the params was matched in the middleware.ts file and we use that here to set the document language for SEO purposes. Additionally, we used this file to pass configuration from i18n/request.ts to Client Components through NextIntlClientProvider. Note: When using the above setup with i18n routing, next-intl will currently opt into dynamic rendering when APIs like useTranslations are used in Server Components. next-intl provides a temporary API that can be used to enable static rendering. Static Rendering for i18n Routes For apps with dynamic routes, use generateStaticParams to pass all possible locale values, allowing Next.js to render at build time: ` next-intl provides an API setRequestLocale that can be used to distribute the locale that is received via params in layouts and pages for usage in all Server Components that are rendered as part of the request. You need to call this function in every layout/page that you intend to enable static rendering for since Next.js can render layouts and pages independently. ` Note: Call setRequestLocale before invoking useTranslations or getMessages or any next-intl functions. Domain Routing For domain-specific locale support, use the domains setting to map domains to locales, such as us.example.com/en or ca.example.com/fr. ` This setup allows you to serve localized content based on domains. Read more on domain routing here. Conclusion Setting up internationalization in Next.js with next-intl provides a modular way to handle URL-based routing, user-defined locales, and domain-specific configurations. Whether you need URL-based routing or a straightforward single-locale setup, next-intl adapts to fit diverse i18n needs. With these tools, your app will be ready to deliver a seamless multi-language experience to users worldwide....

Apr 11, 2025

4 mins

NextJSAccessibility

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Ensuring Accurate Workflow Status in GitHub for Enhanced Visibility

Introduction

The Problem with Misleading Workflow Statuses

Crafting a Solution and Best Practices

Ensuring Accurate Status Reporting

Adjusting the Workflow

Best Practices:

Conclusion

William Mimura

You might also like

Integrating Playwright Tests into Your GitHub Workflow with Vercel

OAuth2 for JavaScript Developers

Our Journey from Cypress to Playwright for E2E Testing

Internationalization in Next.js with next-intl

Let's innovate together!

You might also like

Integrating Playwright Tests into Your GitHub Workflow with Vercel

OAuth2 for JavaScript Developers

Our Journey from Cypress to Playwright for E2E Testing

Internationalization in Next.js with next-intl