Face and Emotion Detection using Microsoft Cognitive Services

binary-1327501_640

Microsoft Cognitive Services provide tools to help you include powerful, machine-based artificial intelligence in your applications. There are a range of APIs covering:

  • computer vision
  • speech recognition
  • language understanding
  • knowledge exploration
  • search

In this article, we’ll be looking at two APIs in the computer vision areas:

  • Face API
  • Emotion API

The Face API allows us to, for example:

  • detect faces in images
  • extract features such as gender and age
  • identify individuals

The Emotion API allows us to identify the type of emotion exhibited in a facial image.

Subscribing to the services

Before we can use any of the Cognitive Services APIs we need to subscribe to them. This involves signing up for a free account. There’s a generous free tier and, if you need more, you can pay based on usage.

Choose the services you want to use (at least the Face and Emotion APIs) and you’ll be supplied with an API key (in the form of a GUID) for each one.

Extracting facial features

We’ll use a C# console app in Visual Studio to try out the APIs. Usually you’d be developing a mobile app, web app or web service, but a console app lets us focus our attention on the cognitive services components.

If you’d prefer to use another programming language there are Python APIs, and the services can also be accessed via REST.

Once you’ve created your console app, import the following package using NuGet.

Install-Package Microsoft.ProjectOxford.Face

Then create an instance of the face service client, passing it your Face API subscription key.

IFaceServiceClient faceServiceClient = 
    new FaceServiceClient("c92cb126fe1a433f8b68a4060b2d3b0f");

Let’s attempt to detect a face and extract all the available features. We’ll use a picture of Hillary Clinton expressing joy at winning in Pennsylvania.

IEnumerable<FaceAttributeType> requestedFaceAttributes = 
    new List<FaceAttributeType>         
    {
        FaceAttributeType.Age,
        FaceAttributeType.Gender,
        FaceAttributeType.HeadPose,
        FaceAttributeType.Smile,
        FaceAttributeType.FacialHair,
        FaceAttributeType.Glasses
    };

Face[] faces = await faceServiceClient.DetectAsync("http://bit.ly/295NCb1", true, 
    true, requestedFaceAttributes);

This returns data for each face in the image. We know there’s only one so we can get that using

Face face = faces.Single();

We can show all the facial attributes as follows.

Console.WriteLine($"Age: {face.FaceAttributes.Age}");
Console.WriteLine(
    $"Gender: {face.FaceAttributes.Gender.ToLower()}");
Console.WriteLine(
    $"Smiling: {(face.FaceAttributes.Smile >= 0.5 ? "yes" : "no")}");
Console.WriteLine(
    $"Beard: {(face.FaceAttributes.FacialHair.Beard >= 0.5 ? "yes" : "no")}");
Console.WriteLine(
    $"Moustache: {(face.FaceAttributes.FacialHair.Moustache >= 0.5 ? "yes" : "no")}");
Console.WriteLine(
    $"Sideburns: {(face.FaceAttributes.FacialHair.Sideburns >= 0.5 ? "yes" : "no")}");

Console.Write("Glasses: ");

switch (face.FaceAttributes.Glasses)
{
    case Glasses.ReadingGlasses:
        Console.WriteLine("reading");

        break;

    case Glasses.Sunglasses:
        Console.WriteLine("sunglasses");

        break;

    case Glasses.SwimmingGoggles:
        Console.WriteLine("swimming goggles");

        break;

    default:
        Console.WriteLine("none");

        break;
}

I’m sure Hillary will be pleased to know what she’s been identified as a 48-year-old female with no facial hair. Obviously, public figures at public events are going to be better turned out than the rest of us—and tend to adopt “unnatural” poses—which might explain why it knocked a couple of decades off her age. Machine learning is weak when presented with outliers.

Cropping the image

When attempting to analyze what emotion(s) Hillary is expressing, we’ll get better, faster results if we focus attention on her face. Let’s crop the image to get rid of extraneous data.

To do this, we’ll use the ImageProcessor package

Install-Package ImageProcessor

The Face object that was returned from the Face API’s DetectAsync method contains the location of Hillary’s face within the image. We’ll crop to that region.

using (var webClient = new WebClient())
{
    using (Stream stream = webClient.OpenRead("http://bit.ly/295NCb1"))
    {
        if (stream == null)
        {
            throw new Exception("Failed to open image URL for reading");
        }

        using (var imageFactory = new ImageFactory())
        {
            Rectangle faceRectangle = new Rectangle(
                face.FaceRectangle.Left, 
                face.FaceRectangle.Top, 
                face.FaceRectangle.Width, 
                face.FaceRectangle.Height);

            imageFactory
                .Load(stream)
                .Crop(faceRectangle)
                .Format(new JpegFormat())
                .Save(@"C:\Dropbox\Public\face-image.jpg));
        }
    }
}

As analyzing URLs with the Emotion API is easier than working with local images I saved the cropped JPEG file to my public Dropbox folder.

Analyzing Emotions

Import the Emotion API SDK package.

Install-Package Microsoft.ProjectOxford.Emotion

As with the Face API, we need to create a client and pass it your Emotion API subscription key.

EmotionServiceClient emotionServiceClient = 
    new EmotionServiceClient("580d30404e5a46d3a4c2035555342412");

Now we just call the RecognizeAsync method on our cropped image.

Emotion[] emotions = await emotionServiceClient.RecognizeAsync(
    https://dl.dropboxusercontent.com/u/xxxxxxx/face-image-jpg");

Again, we only have one face in the image, so let’s get the emotion object for that face.

Emotion emotion = emotions.Single();

Now we can examine the results.

Console.WriteLine($"Anger: {emotion.Scores.Anger}");
Console.WriteLine($"Contempt: {emotion.Scores.Contempt}");
Console.WriteLine($"Disgust: {emotion.Scores.Disgust}");
Console.WriteLine($"Fear: {emotion.Scores.Fear}");
Console.WriteLine($"Happiness: {emotion.Scores.Happiness}");
Console.WriteLine($"Neutral: {emotion.Scores.Neutral}");
Console.WriteLine($"Sadness: {emotion.Scores.Sadness}");
Console.WriteLine($"Surprise: {emotion.Scores.Surprise}");

In my tests, Hillary’s “happiness” score was 0.82, her “surprise” score was 0.18 and the rest were all under 0.005. Seems reasonable.

There’s a Gist of the entire code available to peruse.

There’s much, much more to the Cognitive Services suite. What I hope to have conveyed is how easy it is to add sophisticated machine learning capabilities to your own apps. One of my clients is considering using this service to produce alerts when unauthorized people enter a secure area, for example.

If you are interested in machine learning or .NET development, Learning Tree provides a range of courses designed to expand your skills. Machine learning courses can be found in the big data curriculum, while there’s a range of C# courses in the Visual Studio and .NET curriculum.

Type to search blog.learningtree.com

Do you mean "" ?

Sorry, no results were found for your query.

Please check your spelling and try your search again.