VueJS

Detect Hand Sign Languages with Tensorflow

Published Jul 12, 2023

Updated Oct 12, 2023

7 min read

Interested in learning how to use Tensorflow to detect hand sign languages in your apps? By the end of this read, you will know how to implement Tensorflow in your application with very simple steps. In our example today, we will be using Vue.

What is Tensorflow?

Tensorflow is an end-to-end platform (meaning: delivering complex systems or services in functional form after developing it from beginning to end.) used for building Machine Learning applications, and it is also open-source. TensorFlow enables you to build dataflow graphs and structures to define how data moves through a graph by taking inputs as a multi-dimensional array called Tensor. You can read more on Tensorflow here.

What is a Model?

A model is a function with learnable parameters that maps an input to an output. A well-trained model will provide an accurate mapping from the input to the desired output.

Tensorflow Models

Tensorflow models are pre-trained models, and there are four defined categories of them:

Vision: Analyze features in images and videos.
Body: Detect key points and poses on the face, hands, and body with models from MediPipe
Text: Enable NLP in your web app using the power of BERT and other Transformer encoder architectures.
Audio: Classify audio to detect sounds.

If you want to go into more detail, check out Tensorflow Models.

All these models are broken down into subs and for our case, we will be making use of the Body Model which has the hand pose detection we need in order to detect the hand signs.

Hand Pose Detection

This model used a 2D and 3D multi-dimensional array which enables it to predict the keypoints of the hands.

Example of a 2D is [[1,2],[3,5],[7,8],[20,44]] and that of a 3D is [[1,2,5],[3,5,8],[7,8,6],[20,44,100]].

This hand pose detection is a model from the MediPipe as we established above, and it provides us with two model types which are lite and full. The accuracy of the prediction increases from lite to full while the inference speed reduces, i.e. the response time will be slower as the accuracy increases.

What do we need?

There are a few dependencies we need to get things working, and I also will be assuming that you have your project set up as well.

You will need to add these dependencies to the project


yarn add @tensorflow-models/hand-pose-detection

# Run the below commands if you want to use TF.js runtime.
yarn add @tensorflow/tfjs-core @tensorflow/tfjs-converter
yarn add @tensorflow/tfjs-backend-webgl

# Run the below commands if you want to use MediaPipe runtime.
yarn add @mediapipe/hands

yarn add fingerpose

Above, in the commands, you will notice we added a fingerpose. Let's talk a little about what we need the figerpose for.

Fingerpose

Fingerpose is a gesture classifier for hand landmarks detected by Mediapipe hand pose detection. It also allows you to add your own hand gesture, which means that a gesture that signifies the letter Z can signify Hello based on your fingerpose data. We will see an example of how the data looks in a bit. You can check out fingerpose for more details.

Get started

We are going to use Vue for this illustration. We will start by looking at the HTML first, and then we will cover the JavaScript.

Our Template will be a basic HTML that will have a video tag so we can show a video after getting access to our webcam.

Template

<template>
  <div class="wrapper">
    <video
      ref="videoCam"
      class="peer-video"
      preload="auto"
      autoPlay
      muted
      playsInline
    />
  </div>
</template>

The snippet above shows a div and a video tab. The video is used when we gain access to the webcam.

We will now be writing the JS required to initialize the webcam.

Script


<script  setup>
import { onMounted, ref } from "vue";

const videoCam = ref();
function openCam() {
  let all_mediaDevices = navigator.mediaDevices;

  if (!all_mediaDevices || !all_mediaDevices.getUserMedia) {
    console.log("getUserMedia() not supported.");
    return;
  }
  all_mediaDevices.getUserMedia({
    audio: true,
    video: true,

  })
    .then(function (vidStream) {
      if ("srcObject" in videoCam.value) {
        videoCam.value.srcObject = vidStream;
      } else {
        videoCam.value.src = window.URL.createObjectURL(vidStream);
      }
      videoCam.value.onloadedmetadata = function () {
        videoCam.value.play();
      };
    })
    .catch(function (e) {
      console.log(e.name + ": " + e.message);
    });
}
onMounted(() => {
  openCam();
});
</script>

We imported two methods from vue: onMounted and ref. The onMounted runs when the page is fully mounted while the ref is used to declare a reactive value to reference the video element. If you look at the video tag in the template, you will notice a ref property. You can check out Template ref and onMounted lifecycle hook.

In the openCam function, we first try to test if mediaDevices is available on your browser navigation.

The MediaDevices interface provides access to connected media input devices like cameras and microphones, as well as screen sharing. In essence, it lets you obtain access to any hardware source of media data.

This MediaDevice has a method getUserMedia which prompts the user for permission to use a media input. You can find all you need to know about getUserMedia here.

From the snippet, we can see that getUserMedia returns a promise, and with that, we can get the media stream as a response using then(). We check if the video element has srcObject or not. If it does we assign the media stream to the srcObject and if not, we convert the media stream to a URL and assign it to the src of the video element.

With this Snippet and with a few style, you should have your video showing your awesome face!

Introducing Tensorflow and Hand Detection

Now that we got our webcam working, we will update the Template and the script in order to detect, predict, and display the alphabet based on the hand sign prediction.

The updated HTML should now look like this:

<template>
  <div class="wrapper">
    <video
      ref="videoCam"
      class="peer-video"
      preload="auto"
      autoPlay
      muted
      playsInline
    />
    <div class="alphabet">{{ sign }}</div>
  </div>
</template>

The div with class name alphabet will display the alphabet based on the hand sign prediction.

We will be introducing two(2) new functions, createDetectionInstance and handleSignDetection.

Firstly, lets begin with the createDetectionInstance which is an integral part of the hand sign detection and then we will introduce handleSignDetection which predicts and displays the hand sign.


<script setup>
import { onMounted, ref } from "vue";
import * as handPoseDetection from "@tensorflow-models/hand-pose-detection";

let detector;
const videoCam = ref();

function openCam() {
   …
}

const createDetectionInstance = async () => {
  const model = handPoseDetection.SupportedModels.MediaPipeHands;
  const detectorConfig = {
    runtime: "mediapipe",
    modelType: "lite",
    solutionPath: "https://cdn.jsdelivr.net/npm/@mediapipe/hands/",
  } as const;
  detector = await handPoseDetection.createDetector(model, detectorConfig);
};

onMounted(async () => {
  openCam();
  await createDetectionInstance();
});
</script>

To be able to detect hand poses, we need to create an instance of the handpose detector, and here, we created a function createDetectionInstance which is an asynchronous function.

You can check out this Tensorflow blog to see more details.

Now that we have created an avenue to detect hand signs, let us start detecting the hand.

In that light, we will be adding a handleSignDetection function.


<script setup>
import { onMounted, ref } from "vue";
import * as handPoseDetection from "@tensorflow-models/hand-pose-detection";

let detector;
const videoCam = ref();

function openCam() {
  …
}

const createDetectionInstance = async () => {
  const model = handPoseDetection.SupportedModels.MediaPipeHands;
  const detectorConfig = {
    runtime: "mediapipe",
    modelType: "lite",
    solutionPath: "https://cdn.jsdelivr.net/npm/@mediapipe/hands/",
  } as const;
  detector = await handPoseDetection.createDetector(model, detectorConfig);
};

const handleSignDetection = () => {
  if (!videoCam.value || !detector) return;
  setInterval(async () => {
    const hands = await detector.estimateHands(videoCam.value);
    if (hands.length > 0) {
      console.log(hands)
    }
  }, 2000);
};
onMounted(async () => {
  openCam();
  await createDetectionInstance();
  handleSignDetection();
});
</script>

The handleSignDetection runs after creating the detection instance. We have a setInterval that runs every 2 seconds (PS: the 2 seconds timing is arbitrary and can be less or more) to check if there is any hand sign. We also have a conditional statement to ensure the video element exists, and the detection instance was created accordingly.

So, the detector calls a method estimateHands, which tries to predict the hand pose by getting keypoints with values that are either in 2D or 3D (Multi-dimensional Array).

If you check your console log, you will see an array of data if any hand pose is detected.

Now that we can detect hand poses, we will now add fingerpose that will help predict and display the alphabet based on the hand sign.


<script setup>
import { onMounted, ref } from "vue";
import * as handPoseDetection from "@tensorflow-models/hand-pose-detection";
import * as fp from "fingerpose";
import Handsigns from "@/utils/handsigns";

let detector;
const videoCam = ref();
let sign = ref(null);

function openCam() {
  …
}

const createDetectionInstance = async () => {
  const model = handPoseDetection.SupportedModels.MediaPipeHands;
  const detectorConfig = {
    runtime: "mediapipe",
    modelType: "lite",
    solutionPath: "https://cdn.jsdelivr.net/npm/@mediapipe/hands/",
  } as const;
  detector = await handPoseDetection.createDetector(model, detectorConfig);
};

const handleSignDetection = () => {
  if (!videoCam.value || !detector) return;
  setInterval(async () => {
    const hands = await detector.estimateHands(videoCam.value);
    if (hands.length > 0) {
      const GE = new fp.GestureEstimator([
        fp.Gestures.ThumbsUpGesture,
        Handsigns.aSign,
        Handsigns.bSign,
        Handsigns.cSign,
        …
        Handsigns.zSign,
      ]);

      const landmark = hands[0].keypoints3D.map(
        (value) => [
          value.x,
          value.y,
          value.z,
        ]
      );
      const estimatedGestures = await GE.estimate(landmark, 6.5);

      if (estimatedGestures.gestures && estimatedGestures.gestures.length > 0) {
        const confidence = estimatedGestures.gestures.map((p) => p.score);
        const maxConfidence = confidence.indexOf(
          Math.max.apply(undefined, confidence)
        );

        sign.value = estimatedGestures.gestures[maxConfidence].name
      }
    }
  }, 2000);
};
onMounted(async () => {
  openCam();
  await createDetectionInstance();
  handleSignDetection();
});
</script>

Assuming that our detector sensed a hand, it is time to match this value based on the hand signs we created with the fingerpose.

The landmark variable is a 3D array pulled from the hand result's keypoint3D key value. There is also a keypoint as well, which is a 2D value, and both will give the same result.

Now, using GE.estimate, we can generate a possible gesture that matches the sign, and a score/confidence is assigned to each gesture pending the amount of gesture predicted. So, the gesture with the highest score/confidence is selected since it is estimated to be the closest to the hand sign from the figerpose hand signs we created.

We also imported Handsigns and its content looks like this:

You can also get the handsigns folder from the 100-ms-vue repository. Looking at the screenshot, there is a GestureDescription instance that takes a string A which will represent what the hand sign will stand for. So, it could be anything you want the handsign to stand for.

onMounted is asynchronous because we need to ensure that our detection instance is created, which is required to detect the hand sign.

With the updated code, you should be able to display some letters.

Conclusion

Don't forget, you can see in detail how this was implemented in one of This Dot Labs' open source projects 100-ms-vue. Please note that what we did is just a basic implementation, and to have a production-ready version, it will need a bigger model, and a more complex detection to be able to identify hand sign language.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Jerry Hogan
Passionate developer, super fan of Vue, a singer, writer and musician.
@iamjerry_hogan @hdjerry

Image Text/Face recognition with AWS Rekognition👀

AWS Rekognition What is AWS Rekognition? Rekognition is a AWS service that provides deep learning visual analysis for your images. Rekognition is very easy to integrate into your application by providing an image or video to the AWS Rekognition API. The service will identify some following: objects, people, text, scenes, and activities. "Amazon Rekognition also provides highly accurate facial analysis and facial recognition. You can detect, analyze, and compare faces for a wide variety of use cases, including user verification, cataloging, people counting, and public safety." - *AWS Official Docs* Now let's start using AWS Rekognition Let's start with trying some of their demos to see how AWS Recognition works. 1) Go to the following link and play with the demos. Time to get our hands dirty Warning🚨 : 1) You need to have an AWS Management Console account. 2) It will ask you for your credit card info, but YOU won't be charged for what you use in this tutorial since it's part of the FREE TIER. Setting up our S3 Bucket 1) Go to Find services and look for S3 2) Click on CREATE A BUCKET 3) Enter the bucket name as thisdot-rk-YOUR_NAME 4) Click on NEXT twice 5) Uncheck all the boxes to grant public access to the bucket. Click NEXT *Note: I'm making this bucket public, because for the purpose of this tutorial, I'm not worried about security.* 6) Click on CREATE BUCKET. 7) Time to upload some images to S3. Click on the bucket you just created. 8) Download this image and save it as thisdot.png. https://thisdot-rk-pato.s3.amazonaws.com/thisdot.png 9) Click on the dropdown from Mange Permissions, and click on Grant public read access to this object(s). 10) Click NEXT 11) On the Storage Class, select Standard, and click NEXT 12) Click UPLOAD Setting up our Lambda Function 1) Go to Find services, and look for Lambda 2) Click on CREATE FUNCTION 3) For Lambda function name, enter thisdot-rk-YOUR_NAME 4) Under Runtime, click on the dropdown, and select Python 3.7. 5) Click on CREATE FUNCTION 6)Scroll down to where it says Function code. You should see something like this: 8)Erase everything you see in the editor, and paste the following code in there: *Note: Change the name of the bucket to the bucket name you created thisdot-rk-YOUR_NAME.* The following code is going to help us finding the text inside of the images. Using the *.detect_text* method. ` *Note: To Learn more about other AWS Boto Rekognition functions, visit this website.* 9) Scroll down to change the BASIC SETTINGS< of the lambda. 10) Change Memory to 512MB, and Timeout to 2min 30sec. This is to ensure your lambda doesn't timeout when processing images. 11) Scroll all the way to the top. In the upper right corner, you should see the SAVE button. Click on it. Setting up our Security Roles Using IAM 1) Search for the IAM Service (Services IAM) 2) On the left navigation bar, click on ROLES. 3) You can select any lambda you have created to give it a specific role. In this tutorial, we will select the following to give it access to AWS Rekognition. 4) Then click on ATTACH POLICIES 5) Search for rekognition 6) Select AmazonRekognitionFUllAccess 7) Click on ATTACH POLICY *Note: You can have multiple policies attached* Time to Test 1) Go back to your lambda function. 2) In the top right corner, select the dropdown that says "Select a test event" 3) Then select "Configure test events" 4) Give a name to your event 5) Then enter the following JSON object ` Where "thisdot.png" is the name of your image inside of your S3 bucket. 6) Click CREATE 7) On Your top right corner, you will see the TEST button. Click on it. 8) You should see a green box. Click on expand details. 9) Take a look at the response object. As you can see, it has found our text inside of the image. Awesome!!!, right? Now, let's jump to compare faces. Imagine you want to see if one person appears in the same picture. Rekognition can do this. Imagine all the possibilities! 1) Download these 2 images. Source Image https://thepracticaldev.s3.amazonaws.com/i/ktpt1lx1ubzt3ilupph7.jpg Target Image https://thepracticaldev.s3.amazonaws.com/i/p5j8z6hiey8z8rkspms6.jpg 2) Upload them to S3, following the steps previously mentioned. 3) Go back to your lambda, and create a new test, or edit the existing test. Your test will look like this: ` 4) Then, we are going to modify our lambda code to be able to compare faces ` 5) Look at the Execution results, and analyze the data. Tell me what you think about this tutorial on twitter or comment below!...

Oct 31, 2019

6 mins

Machine LearningAWS

ChatGPT can't solve these problems! with Chris Gardner

Chris Gardner talks neural networks, covering fundamental concepts, historical development, and practical applications. It’s important to understand the difference between artificial intelligence (AI) and machine learning, and the role of neural networks in solving intricate problems with vast datasets. This conversation centers around the intricacies of training neural networks, particularly as they become more complex with multiple layers. Chris and Rob touch on the fascinating yet daunting nature of neural networks, discussing their ability to learn beyond human comprehension. Turning to the practical side of using neural networks, Chris shares the existence of libraries that exist to simplify the process of building a network, enabling users to input data, specify parameters, and entrust the system with the training. Both caution about the biases inherent in the data and the responsibility associated with working on machine learning models. They address challenges related to ethics, highlighting the difficulties in identifying biases and emphasizing the delicate balance between excitement and caution in the evolving field of machine learning. Listen to the podcast here: https://modernweb.podbean.com/e/chris-gardner/...

Jan 16, 2024

1 min

AIMachine LearningChatGPT

Storybook: Can Your Next Project Benefit from It?

I will show you what Storybook is all about and hopefully help you decide if you need it in your next project. Don't worry if you have limited experience as an engineer because you don't need to have an advanced technical background to make use of Storybook. What is Storybook? Storybook is a tool for developing UIs that allows you to work on one component at a time. What is a component? A component is smallest part of a UI. In chemistry, we refer to it as an atom(_Smallest unit of a matter_). Most frameworks are now components based. Why Storybook? Storybook allows you to actually develop the entire UIs without starting the major application. But in order to cleary state why one would choose to use it, lets look at some benefits and challenges of Storybook. Benefits - Being able to build components in isolation,and not having to think about integrations or other systems running in your stack is a blessing - A clearer vision of your components styling and layout - Navigating between UI component as you get to see all component as a sidebar item - Communicating with other devs. It really allows you to show how your new component is supposed to be used and which features it boasts. - You can manipulate props value - Its is not framework agnostic i.e you can use for Vue, React, Angular, etc. - Great for documenting Challenges - If you’re integrating it into an existing project, there is some migration work to be done. - You rely on addons and decorators for handling data. - It can take up a lot of time to maintain and properly set up. - It takes team commitment to keep it updated and ensure everything still works. What are add-ons actually? Add-ons are plugins that can be added to make certain feature work. For example, by default scss doesn't work on Storybook, so you will have to install an add-on to make it work as expected. When to use Storybook In this section, we will the making use of This Dot Labs' Open source project starter.dev as a case study. starter.dev is built using a variety of technologies to demonstrate architectural patterns and best practices using these tools. This project uses these technologies like Vue, React, Nextjs, Angular, etc to show various ways Github can be implemented!. Lets Pick a component from starter.dev GitHub Showcases to demonstrate the features Storybook has. Every conponent has a control tab that allows us to manipulte the props if there be any. For example: Here, looking at the image with the props, you will notice some properties: - cardType: This is a dropdown/select option with two opions issue and pullrequest. - state: this defines the state of the card - author: the person who created it or a name, or whatever, you get the point. Lets change some of the props and see the result - cardType will be pullrequest - state will be closed Here, we can see there has been a change. This is so helpful, and tells the team members what the component can and can't do. Other features Looking at other interesting features like - Settings Here, we can toggle for feature like side nav, the controls (which, in the dropdown option, is call addons) - Responsiveness There are few options that help us test for responsivess. Below, you will find an image showing those options: Here we can select any option to see the responsiveness of that component. Large screen Mobile screen Hover state of the card - Inspection Here, we can see certain properties like padding, and current width and height Looking to get started To get started, all you need is to visit this sites based on your prefered technology - Storybook with Vue - Storybook with React - Storybook with Angular - Storybook with Web Components - Storybook with Web Svelte - Storybook with Web Preact - Storybook with Web Ember Conclusion It's up to you to decide if Storybook helps you achieve your goals faster, or if you think it's useful for your project setup. It's important to look at what it could offer and what it can't. You can decide to use it for just plain documentation or for components, or both....

Oct 28, 2022

6 mins

Storybook

Svelte 5 is Here!

Svelte 5 was finally released after a long time in development. Fortunately, we've been able to test it for some time, and now it has a stable release. Let's dig into its features and why this is such a significant change, even though Svelte 4 code is almost 100% compatible. Svelte syntax everywhere This is one of my favorite additions. Previously, Svelte syntax was limited to a component element. Refactoring code from a component and moving it to a JavaScript file worked differently. Now you can use the .svelte.js or .svelte.ts extension, which allows you to use the same syntax everywhere. It's important to note that it's a way to express that this is not just JS, and what you write may be compiled to something else, just like .svelte files are not just html even though they look very similar. Runes The introduction of runes is one of the most significant changes in Svelte. Many users felt attracted to variables being instantly reactive in previous versions. ` There was, however, a lot of magic underneath and a considerable amount of work from the compiler to make it behave reactively. $state In Svelte 5, a reactive variable has to be explicitly declared using the $state rune, which brings a clear distinction between what is reactive and what isn’t. ` In the previous code, $state is not an actual function being called; it's a hint for the compiler to do something special with this declaration. Rune names always start with a dollar sign ($) and do not need to be imported. However, there’s much more to the changes than just the way we declare reactive variables. Runes bring a lot of new features that make Svelte 5 a great improvement. In Svelte 5, objects or arrays use Proxies to allow for granular reactivity, meaning that individual properties in an object are reactive, even if they are nested objects or arrays. If you modify a property, it will only update that property and will not trigger an update on the whole object. It also supports triggering updates when methods like array.push are called. In previous versions, an assignment was required to trigger an update: ` To have a similar behavior of svelte 4 (no deep reactivity), use the $state.raw() rune. This syntax of .* is common for related features of a rune: $state.raw() will require an assignment to trigger reactivity. ` Because proxies are used, you may need to extract the underlying state instead of the proxy itself. In that case, $state.snapshot() should be used: ` $derived We will use $derived and $derived.by to declare a derived state. The two runes are essentially the same, except one allows us to use a function instead of an expression, allowing for more complex operations to calculate the derived values. ` $effect Effects are useful for running something triggered by a change in state. One of the things you'll note from the $effect rune is that dependencies don't need to be explicit. Those will be picked from reactive values read synchronously in the body of the effect function. ` Something to bear in mind is that these dependencies are picked each time the effect runs. The values read during the last run become the dependencies of the effect. ` Depending on the result of the random method, foo or bar will stop being a dependency of the effect. You should place them outside the condition so they can trigger reruns of the effect. *Variants* of the effect rune are $effect.pre, which runs before a DOM update, and $effect.tracking(), which checks for the context of the effect (true for an effect inside an effect or in the template). $props This rune replaces the export let keywords used in previous versions to define a component's props. To use the $props syntax, we can consider it to return an object with all the properties a component receives. We can use JavaScript syntax to destructure it or use the rest property if we want to retrieve every property not explicitly destructured.. ` $bindable If you want to mutate a prop so that the change flows back to the parent, you can use the $bindable prop to make it work in both directions. A parent component can pass a value to a child component, and the child component is able to modify it, returning the data back to the parent component. ` $inspect The $inspect rune only works in dev mode and will track dependencies deeply. By default, it will call console.log with the provided argument whenever it detects a change. To change the underlying function, use $inspect.with().with): ` $host The host rune gives access to the host element when compiling as a custom element: ` ` Other changes Another important change is that component events are just props in Svelte 5, so you can destructure them using the $props() rune: ` A special children property can be used to project content into a component instead of using slots. Inside the components, use the [@render[(https://svelte.dev/docs/svelte/@render) tag to place them. ` Optional chaining (?.) prevents from attempting to render it if no children are passed in. If you need to render different components in different places (named slots before Svelte 5, you can pass them as any other prop, and use @render with them. Snippets Snippets allow us to declare markup slices and render them conveniently using the render tag (@render). They can take any number of arguments. ` Snippets can also be passed as props to other components. ` ` Conclusion Some exciting changes to the Svelte syntax were introduced while, at the same time, maximum compatibility efforts were made. This was a massive rewrite with numerous improvements in terms of performance, bundle size, and DX. Besides that, a new CLI has been released, making the whole experience of starting a project or adding features delightful. If you haven't tried Svelte before, it's a great time to try it now....

Nov 27, 2024

4 mins

Svelte

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Detect Hand Sign Languages with Tensorflow

What is Tensorflow?

What is a Model?

Tensorflow Models

Hand Pose Detection

What do we need?

Fingerpose

Get started

Template

Script

Introducing Tensorflow and Hand Detection

Conclusion

Jerry Hogan

You might also like

Image Text/Face recognition with AWS Rekognition👀

ChatGPT can't solve these problems! with Chris Gardner

Storybook: Can Your Next Project Benefit from It?

Svelte 5 is Here!

Let's innovate together!

You might also like

Image Text/Face recognition with AWS Rekognition👀

ChatGPT can't solve these problems! with Chris Gardner

Storybook: Can Your Next Project Benefit from It?

Svelte 5 is Here!