Creating an AI gesture recognition game with next generation Angular and Tensorflow.js

Showcasing the Reactive Primitives

Angular 16 introduces a new reactive primitive called signals that was central to the control flow of the application. To showcase how the signals can be used in Angular, our team implemented the Snake game functionality twice. We created one version that utilizes RxJS, and another implementation that uses Angular's new signals API. To make both versions readily available, we built an abstract base Angular service with two concrete implementations, and two game board components that implement the functionality in the chosen method. This enables users to see both versions, so we exposed each implementation on their own route making all versions available to anyone using the app.

Integrating Video Feed Into App

To integrate the video feed from the web camera with the Tensorflow.js hand-detector, our team implemented wrappers around the browser's Media Capture and Streams API features. These implementations allowed us to quickly interact with the input from the player's hand gestures, and have the game respond in real time. This experience presented some challenges on mobile devices with camera access as mobile browsers maintain a different set of permission APIs. Due to our feature encapsulation, we were able to quickly identify the root cause of the issue and fix it for all related systems.

Meeting the Design Threshold In Time for the Event

Because the application would be demoed at Google I/O 2023, it had to meet a minimum design threshold. It had to be clean, attractive, and easy to use. Using only CSS, our team designed a classic monitor and controller interface that included other effects that gave the game a retro video game look and feel. We iterated over the design with the client, and our internal team to make the game's design accessible as some of the gradients and styles selected presented challenges for visibly impaired users. Ensuring the application could be used by the highest number of users was of the utmost importance.

Summary

In summary, This Dot Labs collaborated with the Angular Core Team at Google to develop an ML-integrated demo application for Google I/O 2023. They successfully delivered an interactive Snake Game that utilized hand gestures for control, leveraging TensorFlow.js. The application, built with Angular 16, showcased reactive primitives and integrated video feed from the web camera. The team's use of SCSS for styling, Firebase for deployment, and GitHub Actions for automated deployment ensured a seamless development process. The final product met the design threshold, offering a visually appealing and accessible experience for users.