Developer Insights
Join millions of viewers! Our engineers craft human-written articles solving real-world problems weekly. Enjoy fresh technical content and numerous interviews featuring modern web advancements with industry leaders and open-source authors.
Web Scraping with Deno
TypeScript is a great choice for writing a web scraper, but using it and web APIs in Node is kind of a pain. But there's an alternative: web scraping with Deno!...
Mar 24, 2023
6 mins
Splitting Work: Multi-Threaded Programming in Deno
Deno is a new runtime for JavaScript/TypeScript built on top of the V8 JavaScript engine. It was created as an alternative for Node.js with a focus on security and modern language features. Here at This Dot, we've been working with Deno for a while, and we've even created a starter kit that you can use to scaffold your next backend Deno project. The starter kit uses many standard Deno modules, such as the Oak web server and the DenoDB ORM. One issue you may encounter, when scaling an application from this starter kit, is how to handle expensive or long-running asynchronous tasks and operations in Deno without blocking your server from handling more requests. Deno, just like Node.js, uses an event loop in order to process asynchronous tasks. This event loop is responsible for managing the flow of Deno applications and handling the execution of asynchronous tasks. The event loop is executed in a single thread. Therefore, if there is some CPU-intensive or long-running logic that needs to be executed, it needs to be offloaded from the main thread. This is where Deno workers come into play. Deno workers are built upon the Web Worker API specification, and provide a way to run JavaScript or TypeScript code in separate threads, allowing you to execute CPU-intensive or long-running tasks concurrently, without blocking the event loop. They communicate with the main process through a message-passing API. In this blog post, we will show you how to expand on our starter kit using Deno Workers. In our starter kit API, where we have CRUD operations for managing technologies, we'll modify the create endpoint to also read an image representing the technology and generate thumbnails for that image. Generating thumbnails Image processing is CPU-intensive. If the image being processed is large, it may require a significant amount of CPU resources to complete in a timely manner. When including image processing as part of an API, it's definitely a good idea to offload that processing to a separate thread if you want to keep your API responsive. Although there are many image processing libraries out there for the Node ecosystem, the Deno ecosystem does not have as many for now. Fortunately, for our use case, using a simple library like deno-image is good enough. With only a few lines of code, you can resize any image, as shown in the below example from deno-image's repository: ` Let's now create our own thumbnail generator. Create a new file called generate_thumbnails.ts in the src/worker folder of the starter kit: ` The function uses fetch to retrieve the image from a remote URL, and store it in a local buffer. Afterwards, it goes through a predefined list of thumbnail sizes, calling resize() for each, and then saving each image to the public/images folder, which is a public folder of the web server. Each image's filename is generated from the original image's URL, and appended with the thumbnail dimensions. Calling the web worker The web worker itself is a simple Deno module which defines an event handler for incoming messages from the main thread. Create worker.ts in the src/worker folder: ` The event's data property expects an object representing a message from the main thread. In our case, we only need an image URL to process an image, so event.data.imageUrl will contain the image URL to process. Then, we call the generateThumbnails function on that URL, and then we close the worker when done. Now, before calling the web worker to resize our image, let's modify the Technology type from the GraphQL schema in the starter kit to accept an image URL. This way, when we execute the mutation to create a new technology, we can execute the logic to read the image, and resize it in the web worker. ` After calling deno task generate-type-definition to generate new TypeScript files from the modified schema, we can now use the imageUrl field in our mutation handler, which creates a new instance of the technology. At the top of the mutation_handler.ts module, let's define our worker: ` This is only done once, so that Deno loads the worker on module initialization. Afterwards, we can send messages to our worker on every call of the mutation handler using postMessage: ` With this implementation, your API will remain responsive, because post-processing actions such as thumbnail generation are offloaded to a separate worker. The main thread and the worker thread communicate with a simple messaging system. Conclusion Overall, Deno is a powerful and efficient runtime for building server-side applications, small and large. Its combination of performance and ease-of-use make it an appealing choice for developers looking to build scalable and reliable systems. With its support for the Web Worker API spec, Deno is also well-suited for performing large-scale data processing tasks, as we've shown in this blog post. If you want to learn more about Deno, check out deno.framework.dev for a curated list of libraries and resources. If you are looking to start a new Deno project, check out our Deno starter kit resources at starter.dev....
Mar 21, 2023
4 mins
Hey Deno! Where is my package.json?
Disclaimer: This blog was written for Deno versions prior to 1.3.1. After 1.3.1, while Deno can handle projects that contain a package.json (to help facilitate migration), it is still recommended that you handle your dependencies as discussed in this article. Introduction Where is my package.json? That was one of my first questions when I started learning Deno. Coming from a NodeJS background, I am used to having a package manager (such as NPM) for managing dependencies, scripts and other configurations, and I’m used to using the package.json file to declare my dependencies and their versions. Deno has no concept of a package manager, as external modules are imported directly into local modules (e.g import { bold } from ‘https://deno.land/std@v0.32.0/fmt/colors.ts’). At first, this seems very convenient, but it got me wondering how I would be able to manage and update dependencies when they are imported into several different modules across a large project. And how about running scripts? Node allows you to define arbitrary scripts in the package.json that can be executed using npm run. How do we define and manage scripts in Deno? In this article, we will discuss the various ways to manage dependencies in Deno, and also how to manage all scripts needed for our project. Managing Dependencies Using deps.ts The standard practice for managing dependencies in Deno is to create a deps.ts file that imports, and then immediately re-exports all third-party code. ` In your local module, these methods, and classes can be referenced from the deps.ts file. ` You may be familiar with the concept of dev dependencies in NPM. You can define dev dependencies in Deno using a separate dev_deps.ts file, allowing for a clean separation between dev-only and production dependencies. With this approach, managing dependencies in Deno becomes much simpler. For example, to upgrade a dependency version, you make the change in the depts.ts file, and it propagates automatically to all modules in which it is referenced. When using this approach, one should consider integrity checking & lock files. This is basically Deno’s solution for avoiding production issues if the content in the remote url (e.g https://some.url/a.ts) is changed. This could lead to the production module running with different dependency code than your local module. Just like package-lock.json in NodeJS, Deno can store and check subresource integrity for modules using a small JSON file. To autogenerate a lock file, create a deno.json file at the root of your project and a deno.lock file will be autogenerated. You can also choose a different file name by updating the deno.json file like so: ` You can also disable automatically creating, and validating a lock file by specifying: ` We can manually create or update the lock file using the Deno cache command, and --lock and --lock-write flags like so: ` Then a new collaborator can clone the project on their machine and run: ` Using import_map.json Another way to manage dependencies in Deno is by using the import_map.json file. This method is useful if you want to use "bare-specifiers" (specifiers without an absolute or relative path to them, e.g import react from ‘react’). This file allows you to alias a specific import URL or file path, which can be useful if you want to use a custom alias for a dependency. To use the import_map.json file, you first need to create it in the root directory of your project. The file should contain a JSON object with a single key, "imports", which maps import aliases to fully-qualified module URLs or file paths. You can use the import_map.json file to map import paths to remote dependencies and even NPM specifiers. You can also use the import_map.json file to map aliases to local file paths. For example, if you have a local module in your project at ./src/lib/my_module.ts, you can map the import path "my_module" to this file. Here's an example of an import_map.json file: ` With this import_map.json file in place, you can now import the libraries using their aliases: ` Using the import_map.json file can be a convenient way to manage dependencies in your Deno projects, especially if you want to use custom aliases for your imports. Just be sure to include the --import-map flag when running your Deno application, like so: ` This will ensure that Deno uses the import map specified in the import_map.json file when resolving dependencies. Managing Command Scripts Like npm run, the Deno CLI also has a run command that is used to run scripts in files. Depending on the permission needed or the type of operation, there are certain flags that need to be passed to the run command. For example, if you want to run a web server that uses an env file and reads from it, your command will look like this: deno run --allow-net --allow-env --allow-read main.ts If we are reading and writing to a file, we need to add the --allow-write, or if we have an API that needs information about the operating system of the user, then we will also need to add --allow-sys. This can go on and on. We could decide to use --allow-all to accept all permissions, but this is not advisable. The good news is that we can manage all our command scripts without having to retype them every time. We can add these scripts to the deno.json file. The deno.json file is a configuration file for customizing basic deno CLI commands like fmt, lint, test etc. In addition, the file can also be used to define tasks using the "tasks" field. To define tasks, you can specify a mapping of task names to “tasks”. For example: ` You can then run these tasks using the Deno task command, and specifying the task name, like this: ` Conclusion In this article, we saw how to manage dependencies and command scripts in a Deno project. If you’re coming from a Node background, and were confused about where the package.json file was, we hope this clarified how to accomplish some of the same things using the depts.ts, dev_depts.ts, import_map.json, and deno.json files. We hope this article was helpful, and you were able to learn and be more comfortable with using Deno for your projects. If you want to learn more about Deno, check out deno.framework.dev for a list of libraries and resources. If you are looking to start a new Deno project, check out our starter kit resources at starter.dev....
Jan 3, 2023
5 mins
State of Deno: A Look at the Deno CLI, Node.js Compatibility and the Fresh Framework
In this State of Deno event, our panelists discussed the Deno CLI, Node.js compatibility for the npm ecosystem, and the Fresh framework. In this wrap-up, we will take a deeper look into these latest developments and explore what is on the horizon for Deno. You can watch the full State of Deno event on the This Dot Media YouTube Channel. Here is a complete list of the host and panelists that participated in this online event. Hosts: Tracy Lee, CEO, This Dot Labs, @ladyleet Panelists: Colin Ihrig, Software Engineer at Deno, @cjihrig Luca Casonato, Software Engineer at Deno, @lcasdev Bartek Iwańczuk, Software Engineer at Deno, @biwanczuk David Sherret, Software Engineer at Deno, @DavidSherret Table of Contents - Exploring the Deno CLI and its features - What is Deno? - Built in support for TypeScript - Built in toolchain - Deno install and upgrade commands - Deno permissions - Upcoming features - Deno products - Deno Deploy - Deno and Node.js compatibility - Future support for npm packages - The Deno to Node Transform library tool - Fresh framework - Conclusion - We look forward to seeing you at our next State of Deno! Exploring the Deno CLI and Its Features What is Deno? Deno is server-side runtime for JavaScript that also behaves similarly to a browser because it supports all of the same browser APIs on the server. This support provides access to existing knowledge, resources, and documentation for these browser APIs. The team at Deno works closely with browser vendors to make sure that new web APIs work well for both server-side runtime and browsers. Built In Support for TypeScript One of the advantages of Deno is that it ships with TypeScript by default. This removes the setup and configuration time, and reduces the barrier of entry for getting started with TypeScript. Deno also type checks your TypeScript code so you no longer have to use tsc. Built in Toolchain The Deno CLI comes with an entire toolchain which includes a formatter, Linter, package manager, vendor remote dependencies, editor integrations, and more. One of those tools is the Documentation Generator, which annotates library function comments, types, or interfaces with JSDocs comments to easily generate documentation. For a complete list of the Deno CLI tools, please visit their documentation page. Deno install and upgrade commands Deno install is another feature that allows you to take a script and install it using a global command. ` If there is a new version of Deno, you can just run the upgrade command and it will upgrade itself, which makes it a version manager for itself. ` Deno permissions Deno by default will not have file, network or environment access unless you grant those permissions by running a script with command line flags. ` Even with permissions granted, you can scope it to certain directories to allow it to only read and write in the directory of your choosing. If you run the program without permissions granted, the program will still prompt you for permission access before accessing a file. To learn more about Deno's permissions, please read through the documentation. Upcoming features Deno is currently working on improving performance in the areas of HTTP, FFI (Foreign Function Interface) and Node compatibility. There are also improvements being made on the Documentation Generator, to make sure that the docs provided are good and removing the need to maintain two separate docs. Deno Products Deno Deploy Deno deploy is a hosted offering that makes it easy to go from local development to production ready. This service integrates well with GitHub, and provides an option to pay only when users make requests to your services. Deno deploy has a dashboard that allows you to automate most of the processes and show you metrics, deployment statistics, CPU utilization, and network utilization. It also allows you to set up custom domains and provision certificates. Deno and Node.js compatibility Deno v1.15 will introduce a Node.js compatibility mode which will make it possible to run some of Node's programs in Deno. Node APIs like the http server will work in Deno as they would in Node.js. When it comes to the NPM ecosystem compatibility, the goal is to support the large number of packages built on Node.js. The Deno team is working on these compatibility issues, because it uses web APIs for most of their operations. All of these Web APIs were created after Node.js was conceived, meaning that Node implements a whole different set of APIs to do certain operations like performing network requests. This new Node.js API compatibility layer will be able to translate API calls to modern underlying APIs which is what the actual runtime implements. Future support for npm packages When it comes to supporting npm packages on Deno, the goal is to have a transpiler server that takes common.js code and translates that into ESM (ECMAScript module) code. The reason for this is because, just like browsers, Deno only supports ESM code. Deno uses the concept of npm specifiers to provide access to the npm package. Deno downloads the npm package and runs it from a global cache. common.js is also supported ,and it runs the code as it is. ` For npm packages, Deno will create a single copy of the downloaded package instead of multiple directories or sub-directories of modules. It is one global hash file, with no node_modules directory by default, and no need for a package.json by default. If a package requires a node_modules directory, then that directory can be created using a specifier flag. The following command will create a node_modules directory in the project directory, using symlink. ` The Deno to Node Transform library tool The Deno team built a tool to allow library authors to transform Deno packages to Node.js packages. Deno to Node Transform (DNT) takes the Deno code then builds it for Node and distributes it as an npm package. This allows library authors to take advantage of the Deno tool chain. This package can also be shipped on npm for Node.js users. Fresh framework Fresh is a new web framework for Deno, that makes use of the Deno toolchain ecosystem. Fresh uses JSX for templating, and it is similar to Next.js or Remix. A key difference between Fresh and Next.js or Remix, is that Fresh is built to be server-side rendered on the edge rather than server-side in a few locations. Another difference is that with Fresh, no JavaScript is shipped to the client by default, which makes it faster. Fresh handles the Server-side rendering, generates the HTML, sends the file to the client, and hydrates only the part of the page that requires JavaScript on the client by default. Here are some products that already use the Fresh framework: - Deno - merch.deno.com - Deno Deploy To learn more about how to build apps with the Fresh framework, please read through this helpful blog post. Conclusion The team at Deno is making great progress to bring more exciting features to the community to make the runtime engine easy to use for building or migrating existing libraries. If you have any questions about the State of Deno, be sure to ask here. What is it you find exciting about Deno? We will be happy to hear about it on Twitter! We look forward to seeing you at our next State of Deno!...
Dec 14, 2022
6 mins
Let's innovate together!
We're ready to be your trusted technical partners in your digital innovation journey.
Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.