Javascript

Our Journey from Cypress to Playwright for E2E Testing

William Mimura

Published May 31, 2024

Updated May 31, 2024

3 min read

Our Journey from Cypress to Playwright for E2E Testing

Introduction

Adapting to unexpected technical challenges often leads to innovation. Our team faced such a moment when we discovered that our primary end-to-end (E2E) testing framework, Cypress, was incompatible with an essential component of our tech stack. This discovery led us to consider alternatives to resolve the compatibility issue and enhance our testing processes.

Our choice was Playwright, a modern testing framework acclaimed for its robust features and compatibility with multiple browsers. The shift to Playwright was swift and efficient: all our E2E tests were successfully migrated within a few hours. This rapid transition highlighted our team's ability to adapt and integrate new technologies quickly.

In this blog post, we will share the hands-on process we followed during the migration to Playwright, the challenges we encountered along the way, and the solutions we implemented to overcome them.

Using a Test Conversion Tool

We utilized a third-party test conversion tool to facilitate our migration from Cypress to Playwright, which significantly streamlined the process. Among several options available, we chose Ray's Cypress to Playwright Conversion Tool for its ease of use and effectiveness. This tool allowed us to automatically transpose our existing Cypress tests into the Playwright format.

The tool processes each test script by identifying equivalent functions and commands in Playwright, translating our entire suite with minimal input. Below are some before and after images of our test scripts, demonstrating the tool's conversion capabilities.

One test script written in Cypress:

describe("Newsletter page", () => {
  it("subscribing to email newsletter should show success message", () => {
    cy.visitWithTimeout("/newsletter");

    cy.getDataCy("newsletter-email-input").type(SIGN_UP_EMAIL_TEST);
    cy.getDataCy("newsletter-submit-button").wait(100).click();

    cy.contains("You subscribed to the newsletter successfully!");
  });

  it("home page should contain button to navigate to newsletter page", () => {
    cy.visitWithTimeout("/");

    // email registration signup is located at the bottom of the home page
    cy.scrollTo("bottom");

    cy.getDataCy("newsletter-sign-up-link").contains(
      "Sign up for our newsletters",
    );

    // adding a 100ms wait for safety, otherwise this command runs too fast and the click is not registered by the website
    cy.getDataCy("newsletter-sign-up-link").wait(100).click();

    cy.url().should("include", "/newsletter");
  });
});

And its result converted in Playwright:

import { test, expect } from "@playwright/test";

test.describe("Newsletter page", () => {
  test("subscribing to email newsletter should show success message", async ({
    page,
  }) => {
    page.FIXME_visitWithTimeout("/newsletter");
    page.FIXME_getDataCy("newsletter-email-input");
    await page.fill(SIGN_UP_EMAIL_TEST);
    page.FIXME_getDataCy("newsletter-submit-button");
    await page.waitForTimeout(100);
    await page.click();
    await expect(
      page.getByText(/You subscribed to the newsletter successfully!/).first()
    ).toBeVisible();
  });

  test("home page should contain button to navigate to newsletter page", async ({
    page,
  }) => {
    page.FIXME_visitWithTimeout("/");

    // email registration signup is located at the bottom of the home page
    await page.FIXME_scrollTo("bottom");
    page.FIXME_getDataCy("newsletter-sign-up-link");
    await expect(
      page.getByText(/Sign up for our newsletters/).first()
    ).toBeVisible();

    // adding a 100ms wait for safety, otherwise this command runs too fast and the click is not registered by the website
    page.FIXME_getDataCy("newsletter-sign-up-link");
    await page.waitForTimeout(100);
    await page.click();
    await expect(page).toHaveURL(/\/newsletter/);
  });
});

While the tool performed impressively, converting most of our tests accurately had limitations. Certain custom helper functions and test aliases did not translate directly, and the tool clearly marked these for manual intervention. This feature was beneficial as it allowed us to quickly identify and address the segments that required our attention.

Small Fixes and Fine-Tuning

The initial output from the conversion tool was highly useful, yet it required some tweaks to align with Playwright's capabilities perfectly. The tool conveniently marked each unconverted function with the identifier FIXME_<unconverted function>, making it straightforward to address these specific areas. Here’s how we tackled some of these functions:

FIXME_visitWithTimeout: Originally a custom Cypress command designed to visit a URL with a specified timeout (cy.visit(url, { timeout: 30000 })), this was straightforwardly converted to Playwright's page.goto(url) FIXME_getDataCy: Another custom command (cy.get('[data-cy="<a_selector>"]')) was efficiently translated to Playwright’s page.locator('[data-testid="<a_selector>"]') FIXME_scrollTo: This native Cypress command, which scrolls the window to a specific position, proved unnecessary in Playwright. Given Playwright’s automatic detection of elements on the page, we opted to remove this command entirely

import { test, expect } from "@playwright/test";

test.describe("Newsletter page", () => {
  test("subscribing to email newsletter should show success message", async ({
    page,
  }) => {
    await page.goto("/newsletter");
    await page
      .locator(`[data-testid="newsletter-email-input"]`)
      .fill(SIGN_UP_EMAIL_TEST);
    await page.locator(`[data-testid="newsletter-submit-button"]`).click();
    await expect(
      page.getByText(/You subscribed to the newsletter successfully!/).first(),
    ).toBeVisible();
  });

  test("home page should contain button to navigate to newsletter page", async ({
    page,
  }) => {
    await page.goto("/");

    await expect(
      page
        .locator(`[data-testid="newsletter-sign-up-link"]`)
        .getByText(/Sign up for our newsletters/)
        .first(),
    ).toBeVisible();

    await page.locator(`[data-testid="newsletter-sign-up-link"]`).click();
    await expect(page).toHaveURL(/\/newsletter/);
  });
});

Further refinements were made to the converted code, such as removing some "wait/delay" commands. Playwright’s robust handling of asynchronous events allowed us to streamline our tests by eliminating unnecessary pauses enhancing test execution speed and reliability.

Conclusion

The transition from Cypress to Playwright in our testing framework was driven by necessity but has significantly improved our testing practices. Utilizing a test conversion tool allowed us to migrate our entire suite of E2E tests efficiently, minimizing manual effort and accelerating the adoption of Playwright.

Our experience highlights the importance of flexibility and the willingness to embrace new technologies in the face of unexpected challenges. The marked improvements in test execution speed and reliability, along with Playwright's advanced features, have made this transition a resounding success for our team.

As we continue to refine our tests and explore the full capabilities of Playwright, we encourage other teams facing similar challenges to consider this approach. The tools and processes outlined here can serve as a roadmap for those looking to enhance their automated testing solutions.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

William Mimura
@mimurawil

Enhancing Your Playwright Workflow: A Guide to the VSCode Extension

Introduction In my last post, Quick Guide to Playwright Fixtures: Enhancing Your Tests, I delved into some of the enhancements we've been implementing in our end-to-end (E2E) tests using Playwright. As I refine our testing strategies, I've come across a tool that has quickly become an essential part of my workflow: the Playwright VSCode extension. If you're like me and constantly looking for ways to streamline testing and debugging, you'll appreciate any tool that can make the process more efficient and enjoyable. That's where this extension comes in. It's not just about writing tests - it's about enhancing the entire development experience. In this post, I'll walk you through getting started with the Playwright VSCode extension, sharing some tips and tricks that have made a real difference in my day-to-day work. Installing the Extension & Basic Setup Before diving into the Playwright VSCode extension, it's essential to have Playwright installed on your machine. If you haven't done so already, you can quickly install it by running: ` This command will set up Playwright and ensure all necessary dependencies are installed. Once Playwright is ready, the VSCode extension will be installed next. Open Visual Studio Code, navigate to the Extensions view by clicking on the Extensions icon in the Activity Bar on the side of the window, and search for “Playwright”. The official extension, ID: ms-playwright.playwright / named: “Playwright Test for VSCode” by Microsoft, should appear at the top of the list. Click "Install," and you're all set. With the extension installed, you can start leveraging its powerful features to enhance your Playwright testing workflow within VSCode. Running the Tests and Identifying Outputs To run a test, simply open the test file in VSCode. The Playwright extension will automatically detect test files and display a "Run" icon next to each test and test suite. You can click on this icon to run individual tests or test suites. Alternatively, you can run all the tests in your project using the Playwright Test Explorer, accessible from the sidebar. Once you start running your tests, the extension provides real-time feedback within the editor. You'll see the status of each test - whether it passes, fails, or is skipped - right next to the corresponding test in your code. This immediate feedback loop is incredibly helpful for catching issues as you write your tests. The output of your tests will be displayed in the VSCode terminal. You'll see detailed information about each test run. Debugging Step-by-Step I find debugging particularly useful when a test fails unexpectedly or when I want to verify that certain actions are being performed as intended. Instead of guessing what might be wrong, I can see exactly what's going on in each test step, making debugging a much more straightforward and less frustrating process. To start debugging, you can easily set a breakpoint in your test file by clicking on the left margin next to the line number where you'd like the execution to pause. Once your breakpoints are in place, you can initiate the debug process by selecting the "Debug" option next to the test you'd like to investigate. Once the debugger is running, the extension allows you to step through your code, inspect variables, and evaluate expressions - all within VSCode. This real-time insight into your test execution is a game-changer, enabling you to pinpoint issues more effectively and confidently refine your tests. Using the Pick Locator Tool Another handy feature is the "Pick Locator" tool. If you've ever struggled with selecting the right element in your tests, this tool can be a time saver. It helps you generate reliable locators by letting you interact directly with the webpage elements you want to target. To use the Pick Locator tool, click the "Pick Locator" button in the Playwright Test Explorer. This will open a new window where you can navigate to the site you're testing. As you hover over elements on the page, the tool will suggest locators, allowing you to select the most appropriate one for your test. While the Pick Locator tool is handy, it’s important to ensure that the locators you generate are robust and maintainable. This is especially true when integrating them with the fixtures I discussed in my previous blog post. Combining the proper locators with well-designed fixtures can create more reliable and reusable test setups, ultimately making your E2E tests more efficient. Conclusion The Playwright VSCode extension has quickly become indispensable in my development workflow. It significantly enhanced my experience of writing and running Playwright tests. Whether you’re just starting with Playwright or looking to optimize your existing tests, this extension offers a range of features that can save you time and effort. Combining these tools with thoughtful test design, such as leveraging fixtures, you can create a more efficient and effective testing process. I hope this guide has given you a good overview of what the Playwright VSCode extension can do and how it can benefit your work. If you haven’t tried it yet, I highly recommend giving it a go. And as always, feel free to explore further and experiment with the features that best suit your needs....

Sep 6, 2024

5 mins

VSCodeTestingPlaywright

Quick Guide to Playwright Fixtures: Enhancing Your Tests

Introduction Following our recent blog post on migrating E2E tests from Cypress to Playwright, we've identified opportunities to enhance our test scripts further. In this guide, we'll delve into the basics of Playwright fixtures, demonstrating their utility and flexibility in test environments. Playwright fixtures are reusable components that set up and tear down the environment or conditions necessary for tests. They are crucial for writing clean, maintainable, and scalable tests. Fixtures can handle tasks like opening a browser, initializing a database, or logging into an application—actions you might need before running your tests. As a practical example, we'll revisit one of the tests from our previous post, enhancing it with a new fixture to streamline the testing process and significantly improve maintainability. This post is designed to provide the foundational skills to integrate fixtures into your testing workflows effectively, giving you the confidence to manage and maintain your test scripts more efficiently. Creating Our First Fixture To illustrate the power of fixtures in Playwright, let’s consider a practical example from a test scenario in our project. Below is a snippet of a test case from our newsletter page: ` This test scenario navigates to the newsletter page, fills in an email, submits the form, and checks for a success message. To optimize our test suite, we'll refactor common actions like navigation, form completion, and submission into reusable fixtures. This approach makes our tests cleaner and more maintainable and reduces redundancy across similar test scenarios. Implementing the Fixture Here’s how the fixture looks: ` This fixture encapsulates the actions of navigating to the page, filling out the email field, and submitting the form. By abstracting these actions, we simplify and focus our test cases. Refactoring the Test With the fixture in place, let’s see how it changes our original test file: ` A beforeEach method to reset the state of the NewsletterPage fixture ensures a clean and consistent environment for each test scenario. This practice is crucial for maintaining the integrity and reliability of your tests. By leveraging the NewsletterPage fixture, each test within the "Newsletter page" suite starts with a clean and pre-configured environment. This setup improves test clarity and efficiency and aligns with best practices for scalable test architecture. Conclusion As we've seen, fixtures are powerful tools that help standardize test environments, reduce code redundancy, and ensure that each test operates in a clean state. By abstracting common setup and teardown tasks into fixtures, we can focus our testing efforts on what matters most, verifying the behavior and reliability of the software we're developing. Remember, the key to successful test management is choosing the right tools and using them wisely to create scalable, maintainable, and robust testing frameworks. Playwright fixtures offer a pathway towards achieving these goals, empowering teams to build better software faster and more confidently....

Jul 26, 2024

2 mins

Testing

Ensuring Accurate Workflow Status in GitHub for Enhanced Visibility

Introduction In the world of software development, GitHub workflows are crucial for automating CI/CD processes. However, a key challenge emerges when these workflows report a 'success' despite underlying issues, like failed tests. This is especially common in scenarios involving tests (e.g., Cypress) and notifications (e.g., Slack) within the same workflow. This blog post aims to highlight the importance of accurate GitHub workflow statuses for better visibility and team response, and how to ensure your workflows reflect the true outcome of their runs. The Problem with Misleading Workflow Statuses Consider a scenario in a GitHub workflow where end-to-end tests are run using Cypress. ` If these tests fail, but the workflow proceeds to a subsequent step, like sending a notification via Slack, which completes successfully, the entire workflow might still show a green checkmark. This misleading success status suggests everything is functioning as intended, when in fact, there could be significant underlying issues. The core issue is the determination of workflow success. Even if critical steps like testing fail, later steps without errors can override this, resulting in a false sense of security. This not only delays bug detection but can also lead to faulty code advancing in the CI/CD pipeline. It's crucial for the overall workflow status to accurately reflect failures in critical steps to ensure prompt and appropriate responses to issues. Crafting a Solution and Best Practices Ensuring Accurate Status Reporting To address the issue of misleading workflow statuses, it’s essential to configure your GitHub Actions properly. The goal is to ensure that the workflow accurately reflects the success or failure of critical tasks, such as running tests, regardless of the success of subsequent steps. Adjusting the Workflow Conditional Notifications: First, set up notifications to execute conditionally based on the outcome of critical steps. This ensures you're alerted of the workflow status without altering the overall result. For example, sending a Slack message if a Cypress test fails: ` Explicit Failure Handling: After configuring conditional notifications, explicitly handle failure scenarios. If a critical step like a Cypress test fails, force the workflow to exit with a failure status. This step is crucial to ensure that the overall workflow reflects the true status: ` Best Practices: Clear Step Separation: Clearly separate and label each step in your workflow for easier readability and troubleshooting. Regular Reviews: Periodically review your workflows to ensure they are aligned with the latest project requirements and best practices. Document Workflow Logic: Maintain documentation for your workflows, especially for complex ones, to aid in understanding and future modifications. By first setting up conditional notifications and then enforcing explicit failure handling, you maintain both alertness to issues and accuracy in workflow status. This approach ensures that failures in critical steps like tests are not overshadowed by subsequent successful steps, keeping the reported status of your workflow true to its actual state. Conclusion Accurate GitHub workflow statuses are vital for a transparent and efficient CI/CD process. By implementing conditional notifications and explicit failure handling, we ensure that our workflows truthfully represent the success or failure of critical tasks. This not only fosters better issue awareness and response but also upholds the integrity of our development practices. Embrace these steps as part of your commitment to maintaining clear and reliable automation processes in your projects. Happy coding!...

Mar 22, 2024

3 mins

GitHubGit

Introduction to Zod for Data Validation

As web developers, we're often working with data from external sources like APIs we don't control or user inputs submitted to our backends. We can't always rely on this data to take the form we expect, and we can encounter unexpected errors when it deviates from expectations. But with the Zod library, we can define what our data ought to look like and parse the incoming data against those defined schemas. This lets us work with that data confidently, or to quickly throw an error when it isn't correct. Why use Zod? TypeScript is great for letting us define the shape of our data in our code. It helps us write more correct code the first time around by warning us if we are doing something we shouldn't. But TypeScript can't do everything for us. For example, we can define a variable as a string or a number, but we can't say "a string that starts with user_id_ and is 20 characters long" or "an integer between 1 and 5". There are limits to how much TypeScript can narrow down our data for us. Also, TypeScript is a tool for us developers. When we compile our code, our types are not available to the vanilla JavaScript. JavaScript can't validate that the data we actually use in our code matches what we thought we'd get when we wrote our TypeScript types unless you're willing to manually write code to perform those checks. This is where we can reach for a tool like Zod. With Zod, we can write data schemas. These schemas, in the simplest scenarios, look very much like TypeScript types. But we can do more with Zod than we can with TypeScript alone. Zod schemas let us create additional rules for data parsing and validation. A 20-character string that starts with user_id_? It's z.string().startsWith('user_id_').length(20). An integer between 1 and 5 inclusive? It's z.number().int().gte(1).lte(5). Zod's primitives give us many extra functions to be more specific about *exactly* what data we expect. Unlike TypeScript, Zod schemas aren't removed on compilation to JavaScript—we still have access to them! If our app receives some data, we can verify that it matches the expected shape by passing it to your Zod schema's parse function. You'll either get back your data in exactly the shape you defined in your schema, or Zod will give you an error to tell you what was wrong. Zod schemas aren't a replacement for TypeScript; rather, they are an excellent complement. Once we've defined our Zod schema, it's simple to derive a TypeScript type from it and to use that type as we normally would. But when we really need to be sure our data conforms to the schema, we can always parse the data with our schema for that extra confidence. Defining Data Schemas Zod schemas are the variables that define our expectations for the shape of our data, validate those expectations, and transform the data if necessary to match our desired shape. It's easy to start with simple schemas, and to add complexity as required. Zod provides different functions that represent data structures and related validation options, which can be combined to create larger schemas. In many cases, you'll probably be building a schema for a data object with properties of some primitive type. For example, here's a schema that would validate a JavaScript object representing an order for a pizza: ` Zod provides a number of primitives for defining schemas that line up with JavaScript primitives: string, number, bigint, boolean, date, symbol, undefined, and null. It also includes primitives void, any, unknown, and never for additional typing information. In addition to basic primitives, Zod can define object, array, and other native data structure schemas, as well as schemas for data structures not natively part of JavaScript like tuple and enum. The documentation contains considerable detail on the available data structures and how to use them. Parsing and Validating Data with Schemas With Zod schemas, you're not only telling your program what data should look like; you're also creating the tools to easily verify that the incoming data matches the schema definitions. This is where Zod really shines, as it greatly simplifies the process of validating data like user inputs or third party API responses. Let's say you're writing a website form to register new users. At a minimum, you'll need to make sure the new user's email address is a valid email address. For a password, we'll ask for something at least 8 characters long and including one letter, one number, and one special character. (Yes, this is not really the best way to write strong passwords; but for the sake of showing off how Zod works, we're going with it.) We'll also ask the user to confirm their password by typing it twice. First, let's create a Zod schema to model these inputs: ` So far, this schema is pretty basic. It's only making sure that whatever the user types as an email is an email, and it's checking that the password is at least 8 characters long. But it is *not* checking if password and confirmPassword match, nor checking for the complexity requirements. Let's enhance our schema to fix that! ` By adding refine with a custom validation function, we have been able to verify that the passwords match. If they don't, parsing will give us an error to let us know that the data was invalid. We can also chain refine functions to add checks for our password complexity rules: ` Here we've chained multiple refine functions. You could alternatively use superRefine, which gives you even more fine grained control. Now that we've built out our schema and added refinements for extra validation, we can parse some user inputs. Let's see two test cases: one that's bound to fail, and one that will succeed. ` There are two main ways we can use our schema to validate our data: parse and safeParse. The main difference is that parse will throw an error if validation fails, while safeParse will return an object with a success property of either true or false, and either a data property with your parsed data or an error property with the details of a ZodError explaining why the parsing failed. In the case of our example data, userInput2 will parse just fine and return the data for you to use. But userInput1 will create a ZodError listing all of the ways it has failed validation. ` ` We can use these error messages to communicate to the user how they need to fix their form inputs if validation fails. Each error in the list describes the validation failure and gives us a human readable message to go with it. You'll notice that the validation errors for checking for a valid email and for checking password length have a lot of details, but we've got three items at the end of the error list that don't really tell us anything useful: just a custom error of Invalid input. The first is from our refine checking if the passwords match, and the second two are from our refine functions checking for password complexity (numbers and special characters). Let's modify our refine functions so that these errors are useful! We'll add our own error parameters to customize the message we get back and the path to the data that failed validation. ` Now, our error messages from failures in refine are informative! You can figure out which form fields aren't validating from the path, and then display the messages next to form fields to let the user know how to remedy the error. ` By giving our refine checks a custom path and message, we can make better use of the returned errors. In this case, we can highlight specific problem form fields for the user and give them the message about what is wrong. Integrating with TypeScript Integrating Zod with TypeScript is very easy. Using z.infer<typeof YourSchema> will allow you to avoid writing extra TypeScript types that merely reflect the intent of your Zod schemas. You can create a type from any Zod schema like so: ` Using a TypeScript type derived from a Zod schema does *not* give you any extra level of data validation at the type level beyond what TypeScript is capable of. If you create a type from z.string.min(3).max(20), the TypeScript type will still just be string. And when compiled to JavaScript, even that will be gone! That's why you still need to use parse/safeParse on incoming data to validate it before proceeding as if it really does match your requirements. A common pattern with inferring types from Zod schemas is to use the same name for both. Because the schema is a variable, there's no name conflict if the type uses the same name. However, I find that this can lead to confusing situations when trying to import one or the other—my personal preference is to name the Zod schema with Schema at the end to make it clear which is which. Conclusion Zod is an excellent tool for easily and confidently asserting that the data you're working with is exactly the sort of data you were expecting. It gives us the ability to assert at runtime that we've got what we wanted, and allows us to then craft strategies to handle what happens if that data is wrong. Combined with the ability to infer TypeScript types from Zod schemas, it lets us write and run more reliable code with greater confidence....

Nov 15, 2024

7 mins

JavaScriptZod

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Our Journey from Cypress to Playwright for E2E Testing

Our Journey from Cypress to Playwright for E2E Testing

Introduction

Using a Test Conversion Tool

Small Fixes and Fine-Tuning

Conclusion

William Mimura

You might also like

Enhancing Your Playwright Workflow: A Guide to the VSCode Extension

Quick Guide to Playwright Fixtures: Enhancing Your Tests

Ensuring Accurate Workflow Status in GitHub for Enhanced Visibility

Introduction to Zod for Data Validation

Let's innovate together!

You might also like

Enhancing Your Playwright Workflow: A Guide to the VSCode Extension

Quick Guide to Playwright Fixtures: Enhancing Your Tests

Ensuring Accurate Workflow Status in GitHub for Enhanced Visibility

Introduction to Zod for Data Validation