GraphQL

How to Optimize GraphQL Performance with Redis

Published May 15, 2023

Updated May 15, 2023

3 min read

This article was written over 18 months ago and may contain information that is out of date. Some content may be relevant but please refer to the relevant official documentation or available resources for the latest information.

Intro

GraphQL is a great way to build backend applications. It simplifies the project’s structure and scales easily, but there are some challenges that come with it. One of the biggest is how to optimize GraphQL at scale. In this post, we will show you how to use Redis as an optimization tool for your GraphQL servers.

What is Redis?

Redis is an open-source database that can be used with any programming language. It is well-suited for large-scale applications. It’s a key-value store that can be used to store and retrieve data. It is designed to be fast, with no single point of failure. It also provides a high degree of concurrency, which means it's possible for multiple clients to access the same data at the same time. Redis is also lightweight enough that you can run it in the cloud.

Why GraphQL?

GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.

Why Redis with GraphQL?

Redis can be your best friend when it comes to optimizing the performance of GraphQL. First, Redis provides a powerful system that helps you to cache queries and results so that they can be reused. This is crucial because there's no way to predict how often you'll need to run a query on a serverless architecture. Sometimes you use a 3rd party API that is slow, or there are request limits, or even the local databases could take a quite long time to return the data. If you're creating tons of different queries every day, this can add up quickly!

Different caching strategies

There are two main caching strategies that you can use with GraphQL and Apollo Server:

1. Caching the entire response (use Redis as a cache)

Caching the whole response is the most straightforward way to cache your GraphQL queries. When you cache the entire response, you're essentially caching the entire query, including all of the fields that are returned by the query. This is a great option if you're only interested in caching the data that are returned by the query, and you don't need to worry about caching any of the other data that is returned by the query, or if you have a repeatable query for different users.

Example of in-memory cache:

2. Caching individual fields (use Redis as a data store)

This is a more proper way to cache your GraphQL queries. It's also a more complex way to cache queries in Apollo Server. When you cache individual fields, you're caching the individual fields that are returned by the query. This is a great option if you're only interested in caching the data that are returned by the query, and you don't need to worry about caching any of the other data that are returned by the query.

What not to cache?

Redis is not built for large data. If you're storing critical business data, you're going to need a more traditional data solution.

If you're looking for a way to store complex data queries, look elsewhere. Redis is designed to be simple and fast, but that doesn't mean it's ready for just about anything. Even if you think your data could grow into a large enough set that it would benefit from relational databases, remember that Redis does not have support for building relational databases from scratch. It has no support for tables or relationships or any other kind of constraints that would be required if your data was stored in a relational database.

Conclusion

In this post, we showed you how to use Redis as an optimization tool for your GraphQL. We also showed you how to use Redis as a cache and as a data store. We hope you found this post helpful! Also, check out our GraphQL Serverless Contentful starter kit on Starter.dev If you have any questions or comments, please feel free to send them to us by email at hi@thisdot.co. Thanks for reading!

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Mark Shenouda
@MarkSShenouda @markshenouda

How to create and use custom GraphQL Scalars

How to create and use custom GraphQL Scalars In the realm of GraphQL, scalars form the bedrock of the type system, representing the most fundamental data types like strings, numbers, and booleans. As explored in our previous post, "Leveraging GraphQL Scalars to Enhance Your Schema," scalars play a pivotal role in defining how data is structured and validated. But what happens when the default scalars aren't quite enough? What happens when your application demands a data type as unique as its requirements? Enter the world of custom GraphQL scalars. These data types go beyond the conventional, offering the power and flexibility to tailor your schema to precisely match your application's unique needs. Whether handling complex data structures, enforcing specific data formats, or simply bringing clarity to your API, custom scalars open up a new realm of possibilities. In this post, we'll explore how to understand, create, and effectively utilize custom scalars in GraphQL. From conceptualization to implementation, we'll cover the essentials of extending your GraphQL toolkit, empowering you to transform abstract ideas into concrete, practical solutions. So, let's embark together on the journey of understanding and utilizing custom GraphQL scalars, enhancing and expanding the capabilities of your GraphQL schema. Understanding Custom Scalars Custom scalars in GraphQL extend beyond basic types like String or Int, allowing data to be defined, validated, and processed more precisely. They're instrumental when default types don't quite capture the complexity or specificity of the data, such as with specialized date formats or unique identifiers. The use of custom scalars brings several benefits: * Enhanced Clarity: They offer a clearer representation of what data looks like and how it behaves. * Built-in Validation: Data integrity is bolstered at the schema level. * Flexibility: They can be tailored to specific data handling needs, making your schema more adaptable and robust. With this understanding, we'll explore creating and integrating custom scalars into a GraphQL schema, turning theory into practice. Creating a Custom Scalar Defining a Custom Scalar in TypeScript: Creating a custom scalar in GraphQL with TypeScript involves defining its behavior through parsing, serialization, and validation functions. * Parsing: Transforms input data from the client into a server-understandable format. * Serializing: Converts server data back to a client-friendly format. * Validation: Ensures data adheres to the defined format or criteria. Example: A 'Color' Scalar in TypeScript The Color scalar will ensure that every color value adheres to a valid hexadecimal format, like #FFFFFF for white or #000000 for black: ` In this TypeScript implementation: * validateColors: a function that checks if the provided string matches the hexadecimal color format. * parseValue: a method function that converts the scalar’s value from the client into the server’s representation format - this method is called when a client provides the scalar as a variable. See parseValue docs for more information * serialize: a method function that converts the scalar’s server representation format to the client format, see serialize docs for more information * parseLiteral: similar to parseValue, this method function converts the scalar’s value from the client to the server’s representation format. Still, this method is called when the scalar is provided as a hard-coded argument (inline). See parseLiteral docs for more information In the upcoming section, we'll explore how to incorporate and validate these custom scalars within your schema, ensuring they function seamlessly in real-world scenarios. Integrating Custom Scalars into a Schema Incorporating the 'Color' Scalar After defining your custom Color scalar, the next crucial step is effectively integrating it into your GraphQL schema. This integration ensures that your GraphQL server recognizes and correctly utilizes the scalar. Step-by-Step Integration 1. Add the scalar to Type Definitions: Include the Color scalar in your GraphQL type definitions. This inclusion informs GraphQL about this new scalar type. 2. Resolver Mapping: Map your custom scalar type to its resolver. This connection is key for GraphQL to understand how to process this type during queries and mutations. ` 1. Use the scalar: Update your type to use the new custom scalar ` Testing the Integration With your custom Color scalar integrated, conducting thorough testing is vital. Ensure that your GraphQL server correctly handles the Color scalar, particularly in terms of accepting valid color formats and rejecting invalid ones. For demonstration purposes, I've adapted a creation mutation to include the primaryColor field. To keep this post focused and concise, I won't detail all the code changes here, but the following screenshots illustrate the successful implementation and error handling. Calling the mutation (createTechnology) successfully: Calling the mutation with forced fail (bad color hex): Conclusion The journey into the realm of custom GraphQL scalars reveals a world where data types are no longer confined to the basics. By creating and integrating scalars like the Color type, we unlock precision and specificity in our GraphQL schemas, which significantly enhance our applications' data handling capabilities. Custom scalars are more than just a technical addition; they testify to GraphQL's flexibility and power. They allow developers to express data meaningfully, ensuring that APIs are functional, intuitive, and robust. As we've seen, defining, integrating, and testing these scalars requires a blend of creativity and technical acumen. It encourages a deeper understanding of how data flows through your application and offers a chance to tailor that experience to your project's unique needs. So, as you embark on your GraphQL journey, consider the potential of custom scalars. Whether you're ensuring data integrity, enhancing API clarity, or simply making your schema a perfect fit for your application, the possibilities are as vast as they are exciting. Embrace the power of customization, and let your GraphQL schemas shine!...

Apr 10, 2024

5 mins

GraphQLJavaScript

How to Leverage Apollo Client Fetch Policies Like the Pros

Apollo Client provides a rich ecosystem and cache for interfacing with your GraphQL APIs. You write your query and leverage the useQuery hook to fetch your data. It provides you with some state context and eventually resolves your query. That data is stored in a local, normalized, in-memory cache, which allows Apollo Client to respond to most previously run requests near instantaneously. This has huge benefits for client performance and the feel of your apps. However, sometimes Apollo's default doesn't match the user experience you want to provide. They provide fetch policies to allow you to control this behavior on each query you execute. In this article, we'll explore the different fetch policies and how you should leverage them in your application. cache-first This is the default for Apollo Client. Apollo will execute your query against the cache. If the cache can fully fulfill the request, then that's it, and we return to the client. If it can only partially match your request or cannot find any of the related data, the query will be run against your GraphQL server. The response is cached for the next query and returned to the handler. This method prioritizes minimizing the number of requests sent to your server. However, it has an adverse effect on data that changes regularly. Think of your social media feeds - they typically contain constantly changing information as new posts are generated. Or a real-time dashboard app tracking data as it moves through a system. cache-first is probably not the best policy for these use cases as you won't fetch the latest data from the upstream source. You can lower the cache time of items for the dashboard to avoid the staleness issue and still minimize the requests being made, but this problem will persist for social media feeds. The cache-first policy should be considered for data that does not change often in your system or data that the current user fully controls. Data that doesn't change often is easily cached, and that's a recommended pattern. For data that the user controls, we need to consider how that data changes. If only the current user can change it, we have 2 options: Return the updated data in the response of any mutation which is used to update the cache Use cache invalidation methods like refetchQueries or onQueryUpdated These methods will ensure that our cache stays in sync with our server allowing the policy to work optimally. However, if other users in the system can make changes that impact the current user's view, then we can not invalidate the cache properly using these strategies which makes this policy unideal. network-only This policy skips the cache lookup and goes to the server to fetch the results. The results are stored in the cache for other operations to leverage. Going back to the example I gave in my explanation cache-first of a social media feed, the network-only policy would be a great way to implement the feed itself as it's ever-changing, and we'll likely even want to poll for changes every 10s or so. The following is an example of what this component could look like: ` Whenever this SocialFeed component is rendered, we always fetch the latest results from the GraphQL server ensuring we're looking at the current data. The results are put in the cache which we can leverage in some children components. cache-only cache-only only checks the cache for the requested data and never hits the server. It throws an error if the specified cache items cannot be found. At first glance, this cache policy may seem unhelpful because it's unclear if our cache is seeded with our data. However, in combination with the network-only policy above, this policy becomes helpful. This policy is meant for components down tree from network-only level query. This method is for you if you're a fan of React components' compatibility. We can modify the return of our previous example to be as follows: ` Notice we're not passing the full post object as a prop. This simplifies our Post component types and makes later refactors easier. The Post would like like the following: ` In this query, we're grabbing the data directly from our cache every time because our top-level query should have fetched it. Now, a small bug here makes maintainability a bit harder. Our top-level GetFeed query doesn't guarantee fetching the same fields. Notice how our Post component exports a fragment. Fragments are a feature Apollo supports to share query elements across operations. In our SocialFeed component, we can change our query to be: ` Now, as we change our Post to use new fields and display different data, the refactoring is restricted to just that component, and the upstream components will detect the changes and handle them for us making our codebase more maintainable. Because the upstream component is always fetching from the network, we can trust that the cache will have our data, making this component safe to render. With these examples, though, our users will likely have to see a loading spinner or state on every render unless we add some server rendering. cache-and-network This is where cache-and-network comes to play. With this policy, Apollo Client will run your query against your cache and your GraphQL server. This further simplifies our example above if we want to provide the last fetched results to the user but then update the feed immediately upon gathering the latest data. This is similar to what X/Twitter does when you reload the app. You'll see the last value that was in the cache then it'll render the network values when ready. This can cause a jarring user experience though, if the data is changing a lot over time, so I recommend using this methodology sparsely. However, if you wanted to update our existing example, we'd just change our SocialFeed component to use this policy, and that'll keep our client and server better in sync while still enabling 10s polling. no-cache This policy is very similar to the network-only policy, except it bypasses the local cache entirely. In our previous example, we wrote engagement as a sub-selector on a Post and stored fields there. These metrics can change in real time pretty drastically. Chat features, reactions, viewership numbers, etc., are all types of data that may change in real time. The no-cache policy is good when this type of data is active, such as during a live stream or within the first few hours of a post going out. You may typically want to use the cache-and-network policy eventually but during that active period, you'll probably want to use no-cache so your consumers can trust your data. I'd probably recommend changing your server to split these queries and run different policies for the operations for performance reasons. I haven't mentioned this yet, but you can make the fetch policy on a query dynamic, meaning you combine these different policies' pending states. This could look like the following: ` We pass whether the event is live to the component that then leverages that info to determine if we should cache or not when fetching the chat. That being said, we should consider using subscription operations for this type of feature as well, but that's an exercise for another blog post. standby This is the most uncommon fetch policy, but has a lot of use. This option runs like a cache-first query when it executes. However, by default, this query does not run and is treated like a "skip" until it is manually triggered by a refetch or updateQueries caller. You can achieve similar results by leveraging the useLazyQuery operator, but this maintains the same behavior as other useQuery operators so you'll have more consistency among your components. This method is primarily used for operations pending other queries to finish or when you want to trigger the caller on a mutation. Think about a dashboard with many filters that need to be applied before your query executes. The standby fetch policy can wait until the user hits the Apply or Submit button to execute the operation then calls a await client.refetchQueries({ include: ["DashboardQuery"] }), which will then allow your component to pull in the parameters for your operation and execute it. Again, you could achieve this with useLazyQuery so it's really up to you and your team how you want to approach this problem. To avoid learning 2 ways, though, I recommend picking just one path. Conclusion Apollo Client's fetch policies are a versatile and helpful tool for managing your application data and keeping it in sync with your GraphQL server. In general, you should use the defaults provided by the library, but think about the user experience you want to provide. This will help you determine which policy best meets your needs. Leveraging tools like fragments will enable you to manage your application and use composable patterns more effectively. With the rise of React Server Components and other similar patterns, you'll need to be wary of how that impacts your Apollo Client strategy. However, if you're on a legacy application that leverages traditional SSR patterns, Apollo allows you to pre-render queries on the server and their related cache. When you combine these technologies, you'll find that your apps perform great, and your users will be delighted....

May 17, 2024

7 mins

GraphQL

Export Your Data from Universal Analytics before you lose it

Introduction In 2023, Google announced the retirement of Universal Analytics and told everyone to migrate to Google Analytics 4, commonly known as GA4. Google Analytics has long been a staple for businesses and website owners seeking insights into their audience and performance metrics. However, with the introduction of Google Analytics 4, there comes a pivotal moment for users of the older Universal Analytics (UA) platform. Difference between UA and GA4 What data will you lose? You will lose ALL of the historical data! By July 1, 2024, Google will delete all UA data, and you will no longer have access to it after that. Also, Universal Analytics will no longer track new conversions, affecting ad campaign performance that relies on these conversions for Smart Bidding. Audience lists from Universal Analytics will also disappear, potentially impacting media activation and performance. API requests tied to Universal Analytics properties will fail, preventing data deletion requests and causing Looker Studio to stop displaying Universal Analytics data. Additionally, Attribution Projects (beta) in Google Analytics will be deleted. How to Export Your Data There are multiple ways to export your data from Google Analytics, such as using the Google Analytics add-on for Google Sheets or using BigQuery integration to export historical data from your Universal Analytics 360 property. This article will discuss the easiest way to export your data using the Google Analytics add-on from Google Workspace Marketplace. From “Extensions” > “Google Analytics” > “Create new report” The Google Analytics extension will display a Wizard that requires the following steps: * Set a name for your report * Then, you select the Google Analytics account * Then, the property you want to export * Also, there are some optional configuration that helps you filter the data you want to export Then click on “Create Report,” and it will export the report for you Here is an example of the template https://docs.google.com/spreadsheets/d/1zXEBEQQk6TPeGb7-Wm2J0uM0q0XnTdJqrfikCTmYs8c/copy Running the Report After setting up the report template with the necessary configurations, users can run the report to extract their UA data into Google Sheets. Depending on the size of the dataset, the report may take some time to generate, especially for larger datasets spanning multiple years. In such cases, splitting the data into smaller chunks may be necessary to ensure efficient processing. Conclusion As Google transitions from Universal Analytics to Google Analytics 4, businesses and website owners must proactively preserve their valuable historical data. With the impending deletion of all Universal Analytics data by July 1, 2024, failing to export your data in time could result in significant data loss, affecting ad campaign performance and analytics capabilities. By following the outlined methods to export data using tools like the Google Analytics add-on for Google Sheets, you can ensure that your historical data is safely archived and accessible for future analysis....

May 23, 2024

3 mins

ToolingTips

AI Leadership: Data-Driven Decision Making & Avoiding "Analysis Paralysis" with Jerry Reghunadh

In this episode of Leadership Exchange, host Rob Ocel chats with Jerry Reghunadh, Senior Director of Enterprise Architecture at Data Nimbus, about leadership, AI adoption, and data-driven decision-making. Jerry shares his career journey, insights on leveraging tools like ChatGPT and Copilot, and strategies for building effective data pipelines. They explore how companies can avoid "analysis paralysis," adopt AI strategically, and evaluate new technologies to solve real-world problems. Tune in for practical advice on aligning innovation with business goals and staying competitive in a rapidly evolving tech landscape. Chapters 00:00 – Introduction 00:31 – Jerry’s Leadership Journey 02:01 – Discussing AI in Leadership 03:59 – Experimenting with AI Tools 06:24 – Overcoming Analysis Paralysis 09:18 – Importance of Early AI Adoption 13:12 – AI’s Impact on Efficiency 14:30 – Sponsor Message 16:26 – Setting Realistic Goals for AI 19:41 – Data Management and AI 23:09 – Is AI Just a Fad? 27:22 – Testing New Technologies 29:56 – Final Thoughts on AI 30:25 – Connect with Jerry 30:44 – Closing Remarks 31:20 – Outro Follow Jerry on Social Media Linkedin: https://www.linkedin.com/in/jerrymannel/ Sponsored by This Dot: thisdot.co...

Nov 18, 2024

1 min

AIEngineering Leadership

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

How to Optimize GraphQL Performance with Redis

Intro

What is Redis?

Why GraphQL?

Why Redis with GraphQL?

Different caching strategies

What not to cache?

Conclusion

Mark Shenouda

You might also like

How to create and use custom GraphQL Scalars

How to Leverage Apollo Client Fetch Policies Like the Pros

Export Your Data from Universal Analytics before you lose it

AI Leadership: Data-Driven Decision Making & Avoiding "Analysis Paralysis" with Jerry Reghunadh

Let's innovate together!

You might also like

How to create and use custom GraphQL Scalars

How to Leverage Apollo Client Fetch Policies Like the Pros

Export Your Data from Universal Analytics before you lose it

AI Leadership: Data-Driven Decision Making & Avoiding "Analysis Paralysis" with Jerry Reghunadh