Improvement of Haar Feature Based Face Detection in OpenCV Incorporating Human Skin Color Characteristic

detection method detects facial features and ignores anything else, such as building, trees and also bodies. However, many face detection method fail to detect correct faces from images. David, Kriegman, and Ahuja presented a survey of face detection and presented the trends of researches in face detection [4]. In the survey authors categorized and evaluated different face detection algorithms. Some limitations of those algorithms were also discussed in a brief. A common problem of the existing methods is that they treat non facial area as a facial area. The popular Haar like feature based face detection [9, 12] also suffers from the same problem.


Introduction
Face detection is a very important and challenging matter in the field of image processing. It is also a crucial step of face recognition. Face recognition has distinct advantages over other biometrics systems (e.g., finger print, palm print etc.) because of its non-contact process. Face images can be captured from a distance without touching the person who is being identified and the identification does not require interacting with the person. In addition, face recognition serves the crime deterrent purpose because face images that have been recorded and archived can later help to identify a person.
Face detection is one of the tasks that can be done easily by human vision but very complex for a machine. Several studies are available on face detection which discusses different techniques like feature based face detection, example based face detection, geometric based face detection, neural network based face detection and many more [1][2][3]. Now a day's almost every smart phone uses face detection for automatic focusing on the face area when capturing a photo.
The main goal of any face detection system is to achieve a very high detection rate along with low computational cost. A face features or pattern throughout the whole training images. Then the classifier can be applied to the region of interest to detect that particular object. In the detecting process the classifier finds those features throughout the region of interest and returns the coordinates of that particular object in the input region. Otherwise it does not return any coordinate. OpenCV contains many pre-trained classifier for detecting face, eye, upper body, smile, nose etc. Following subsections briefly explains HAAR feature selection and detecting process of cascade classifier those are used by OpenCV for face detection.

HAAR Feature Selection
HAAR features are very popular for face detection. Figure 1 shows such features whose are pixel based rectangular patterns. Human face can be represented with these rectangular features. A rectangular feature is like a rectangular area that represents a part of an object. Figure 2 shows only three features representing eye area, nose area and mouth area of a human face.
The task of classifier is to find HAAR features throughout the input image. As the face size varies with image size, classifies uses a window that can be scaled at any size and can be moved to the any position. This scaling is achieved by the detector itself rather than scaling images. And this window is moved left to right and upper to down throughout the whole images. The classifier searches the features across this window. These features are also scaled to any multiply. For an example, for 24x24 resolution window the land for edge feature 1(b) (shown in Figure 1) scaled with 2x2 pixels, the classifier will find this 2x2 pixels feature throughout the whole window. Then this features size will be increased to 4x4 pixels and again classifier will find this 4x4 pixels feature to the whole window. This process is done for several scales. Thus the combination of the features, scaling and positions become very large (more than 160,000); thus incurs a very high computational cost. To overcome this situation, Adboost is used to select some important features those explain a face very well from the large number of features. Such selection discards a large amount of unnecessary computation during the training process. The first two important features selected by AdaBoost are shown in Figure 3.
Finally, to detect a rectangular feature (F) from an image, the sum of the pixels which lie within the shaded rectangles ( P s ) were subtracted from the sum of pixels in the white rectangles ( P w ). (1) Figure 4 shows this rectangular feature selection. If rectangular feature value is positive or above some threshold value, it was considered as a facial feature. Otherwise it is not considered as a facial feature.
The set of features was quite large and computing these feature value one by one is very inefficient for a large image. To overcome this problem, a technique called "Integral Image" is used through which rectangle features could be computed very easily. It is an intermediate representation of an image in which a location (x, y) contains the sum of the pixels above and to the left    of (x, y), inclusive: Where points A, B, C, D belong to the integral image I, as shown in the Figure 5. To find the rectangular feature value in the dotted area in Figure 4 with the help of integral image, calculations are as follows.
Area X = Sum of pixels in the rectangular area from the lefttop corner to pixel X.

Face Detection through Cascade Classifier
Though Adaboost has narrowed down the number of features to a few hundred, matching those few hundred to each window is time consuming. To overcome this problem HFFD uses a 38 layer cascaded classifier to detect frontal upright faces. In the cascaded structure 38 classifiers are placed in the hierarchical manner. The first some classifiers are trained with most important features. The number of features in the first five layers of the detector is 1, 10, 25, 25 and 50 features respectively. Finally, to detect a window containing face from an image, classifiers were applied on that window one by one. If a window fails at the first stage, it is discarded and no more classifier is applied on it. If a window passes through a classifier then next classifier is applied on it. If all the facial features are found by the cascaded classifiers then output is positive (i.e., there is a face in the image) otherwise output is negative. The advantages of the cascaded classifier are that it can discard an irrelevant window without matching with all of the features. Figure 6 shows the structure of cascade classifier.

Face Detection Incorporating Human Skin Color Analysis in HFFD
A common problem is encountered in traditional HFFD of OpenCV is that it detects some objects or areas as facial areas but those are not human faces at all. The reason behind is that HAAR feature may matched with other objects which is not face. To overcome this problem we have analyzed human skin color and incorporated in HFFD. This section first explains properties of human skin color and then incorporates it in HFFD.

Human Skin Color Analysis
In this study, we have analyzed human skin color in different color spaces like RGB, HSV, HSL, CMYK etc. and we have found a very interesting characteristic of human skin color in RGB color space. Figure 7 shows different skin color tones. The common property of human skin color is observed that the red component is always greater than the green component (i.e., R>G) and the   To show how this characteristic behaves on real human image, an analysis on images of five people having different skin colors is presented in Figure 8. Images have been selected carefully for proper understanding of the matter and the people of images have distinct skin colors of dark, darkish, light dark, white and very white. Facial area of an image was divided into 100 blocks with some consecutive pixels. Average RGB value of each block is measured and plotted in line charts presented in the right side of corresponding images. In the line chart, X axis represents the direction of block numbers and Y axis depicts the average values of red, green and blue components of corresponding block individually. From the line charts, it is clearly observed that R>G>B conditions are matched in almost all blocks for any image. Few exceptions are acceptable for eye, mustache or beard region blocks. Table 1 shows summary of RBG counts of 100 blocks for each image of Figure 8. From the table it is found that R>G>B is true for 90 or above for all the faces. Finally, it is obvious that R>G>B is true for average RBG values of whole facial area.

Proposed Face Detection incorporating Human Skin Color Analysis in HFFD
In the proposed method RGB characteristics of HSC is incorporated to make face detection process more accurate with HFFD. HFFD method replies several non-facial regions as faces. Such non-facial region will not satisfy the condition of R>G>B of HSC. As computing the RGB value of each pixel is much time consuming, we have developed a model which selects pixels at 5 pixel interval of detected face area by Rectangular Haar-like feature process. Then RGB value of each selected pixel is calculated and if majority of the selected pixels satisfy the proposed relation then it is considered as a face otherwise not. Figure 9 shows flow chart of proposed face detection method.

Experimental Results and Analysis
Used Software and Library

Testing with Different Individual Pictures
To investigate the effectiveness human skin color analysis incorporation in face detection, the proposed method was tested on large number images. After adding skin color feature much wrong detection were reduced. Consider four images shown in Figures 10, 11, 12 and 13. Red rectangles are the result of standard HFFD and green rectangles are the result of proposed HFFD-HSC. Based on this four sample images a performance measurement is shown in Table 2. For sample Image 1 number of actual faces in the image was four. But HFFD detected five faces with one Wrong Detection (WD). On the other hand, proposed method detected faces properly eliminating wrong detection made by HFFD. For Image 2, 3 and 4 the detection process also improved eliminating several wrong face detections.
Sample Image 2 has complex background and performance of HFFD is very poor for the image. Although proposed HFFD-HSC reduced wrong detection, still six WD is available for the image. For better understanding, Table 3 presents status of HSC incorporation on all 13 faces detected by HFFD. Table presents correctness ratio with Number of Pixels Satisfy HSC Condition (NPSH) and Number of Pixels Violet HSC Condition (NPVH). In the table we can see that Faces 3, 4, 6, 8 and 10 have their correctness ratio less than 50%. Therefore, those five faces were eliminated by our proposed method as the wrong detections.

Testing with Benchmark Picture Dataset
We have applied our proposed method and compared with

Improvement of Haar Feature Based Face Detection in OpenCV Incorporating Human Skin Color Characteristic
Copyright: © 2016 Akhand et al.        Table 4 also shows the required time in seconds for individual data set by HFFD and proposed HFFD-HSC. It is worthy notable that proposed HFFD-HSC verifies the outcome of HFFD with human skin color characteristics and require additional computational time. However, additional time required by HFFD-HSC is minimal. As an example, required time for data set 1 were 1561 seconds and 1600 seconds by HFFD and HFFD-HSC, respectively. Therefore, additional time required by HFFD-HSC is only 39 seconds (=1600-1561) for 1022 images of the data set.

Different Light Effect on the Performance of the Proposed Method
The proposed HFFD-HSC works on RGB skin color property (i.e., R>G>B) and therefore this section investigates the performance on images with different light effect. Figure 14 presents face detected by HFFD and HFFD-HSC on nine different images with different lighting effects. The light effects on the images are flash light, spot light, sidelight, backlight, red light, green, blue light etc. From the figure it is interesting to observe that proposed HFFD-HSC only failed to recognize for the cases of green light (h) and blue light (i). In such cases, 'R>G>B' condition violated due to increase of green and blue components. On the other hand, red light is found positive effect on the proposed HFFD-HSC in Figure 14(g).
For better understanding, Table 5 shows individual face wise analysis of each image of Figure 14. Among nine images, three contains (i.e., a, c and e) more than one faces. From the table it is observed that for first image (i.e., 14(a)) the true face is verified by proposed HFFD-HSC and WD of face 1 by HFFD in bottom left corner is properly identified with correctness ratio 0%. Similar effect also found for Figure 14(c) and WD of face 1 by HFFD in middle left position is properly identified with correctness ratio 9%. On the other hand, due to green and blue light effects, proposed HFFD-HSC failed to verify the true faces identified by HFFD in Figure 14(h) and Figure 14(i). Finally, only extreme images with green and blue light effects the proposed method failed to improve performance.