Co-authored by Jason Sosa of Immersive Labs and Dr. Stephen Moore
Face detection is a computer-vision technology that determines the location and size of a human face in an arbitrary (digital) image. The facial features in the image are detected, and any other objects like trees, buildings, bodies, etc. are ignored. The human face is a rich source of information — by looking at the person’s face, we can immediately identify whether the person is male or female, the person’s approximate age, facial expression, and so on. Face detection can be regarded as a more “general” case of face localization. In face localization, the task is to find the locations and sizes of a known number of faces.
Face detection is one of the visual tasks which humans can do effortlessly. However, in computer vision terms, this task is not easy. A general statement of the problem can be defined as follows: Given a still or video image, detect and localize an unknown number (if any) of faces. The solution to the problem involves segmentation, extraction, and verification of faces, and possibly facial features, from an uncontrolled background. As a visual front-end processor, a face detection system should also be able to achieve the task regardless of illumination, orientation, or camera distance. There are different approaches to detect a face in a given image. Below are just some of the techniques used for computer-vision face detection.
Finding faces by color
This is the approach where a face is detected using skin color. Once we have access to color images, it is possible to use the typical skin color to find face segments. But in this approach, there is a drawback. Skin color varies from race to race, and this does not work well with all skin colors. In addition, this approach is not very robust under varying lighting conditions.
Finding faces by motion
Faces are usually moving in real-time videos. Calculating the moving area will capture the face segment. However, other objects in the video can also be moving, and may affect the results. A specific type of facial motion is blinking. Detecting a blinking pattern in an image sequence can detect the presence of a face. Eyes usually blink together and are symmetrically positioned, which eliminates similar motions in the video. Each image is subtracted from the previous image. The difference image will show boundaries of moved pixels. If the eyes happen to be blinking, there will be a small boundary within the face. A face model can contain the appearance, shape, and motion of faces. There are several shapes of faces. Some common shapes are oval, rectangle, round, square, heart, and triangle. Motions include, but are not limited to, blinking, raised eyebrows, flared nostrils, wrinkled forehead, and opened mouth. The face models will not be able to represent any person making any expression, but the technique does result in an acceptable degree of accuracy. The models are passed over the image to find faces; however, this technique works better with face tracking. Once the face is detected, the model is laid over the face and the system is able to track face movements.
Finding faces in image with controlled background
This is the easiest of all the approaches. In this approach, images are used with a plain mono color background, or images with a predefined static background. Removing the background gives only the face boundaries, assuming the image contains only a frontal face. Intel OpenCV is an open source library which adopts this method to detect human frontal faces in a given image. The OpenCV library makes it fairly easy to detect a frontal face in an image by using its Haar Cascade Face Detector (also known as the Viola-Jones method). OpenCV comes with several different classifiers for frontal face detection, as well as some profile faces (side view), eye detection, nose detection, mouth detection, whole body detection, etc. Once it detects a human face in a given image, it will mark a rectangular box around it.
Differences between Face Detection and Face Recognition
As computers become more ubiquitous, concerns over privacy have come to the forefront of discussions in the media and online. One area in particular which has raised concerns is the area of face detection and face recognition. Media outlets commonly interchange the terms face detection and face recognition. This has also led to misconception about the technology. Simply put, face detection detects human faces. While face recognition searches for faces similar to ones stored in a database. Face recognition can identify and remember your face even years after it was first recorded. With face detection, no faces or identity information is learnt or stored, thus if a person leaves the camera view and re-appears later that face is not remembered. Face Detection software uses some characterizes of the face to classify demographics (age and gender) and are immediately discarded. In fact, face detection software is not capable of recognizing individuals.
In contrast to face detection is the problem of facial recognition. Recognition software concentrates on making a positive identification of the individual against a database that archives identity information. Face recognition requires the face image or at the very least, information that can identify the face as unique be stored and compared to other such identity information in a database. Such films as Minority Report depict retinal scanning and provides a glimpse into the future where advertisers record identity information and market products based on previous purchases.
Real-Time vs Server Side Face Detection/Recognition
Various technologies exist within face detection/recognition to tackle specific problems. Companies such as Face.com provide a REST API which allow third party developers to submit images for analysis and tagging. This is done by sending an image file of a face to their servers. The face data is then returned with tagged attributes such as identity, age and gender. This server side technique provides a high level of recognition accuracy but does not allow for real-time face detection and tracking due to the latency of the network connection. Real-time face detection runs locally on the client side and allows for real-world detection of multiple faces simultaneously. The processing time required for analysis is less than 100 milliseconds.
Benefits of Face Detection
Detection of human faces is becoming a very important task in various applications. Using face detection as the foundation, many applications can be based on top of it.
- Face detection is most widely used as a pre-processor in face recognition systems. Face recognition systems have many potential applications in computer vision communication and automatic access control systems.
- Face detection is also being researched for use in the area of energy conservation. Televisions and computers can save energy by reducing screen brightness. People tend to watch TV while doing other tasks, and are not always focused 100% on the screen. The TV brightness stays at the same level unless the user lowers it manually. The system can recognize the face direction of the TV viewer. When the viewer is not looking at the screen, the TV brightness is lowered. When the face returns to the screen, the brightness is increased. Programming and advertising can also be tailored based on Face recognition.
- Gender Recognition: From a given image, we can detect whether the person in the image is male or female. This is particularly useful for advertisers and retailers interested in audience measurement in physical spaces.
- Detection of facial expression (happy, surprised, angry, sad, etc).