Journal of Computer Science Applications and
Information Technology

Face detection from a digital image or video stream is used often for various purposes. But sometimes a system detects an object or area as a face where there is no face at all. This paper presents a technique to reduce such wrong detection rate introducing human skin color (HSC) characteristic. The general property of human skin in RGB color space is that it possess R>G>B (i.e., red values are higher than green value and green value is higher than blue). In this study, such HSC property has been incorporated with the popular Haar Feature Based Face Detection (HFFD) in OpenCV, to reduce wrong detection of faces. Proposed HFFD with HSC (HFFD-HSC) has been tested and compared with standard HFFD rigorously on large number images with single and multiple faces. Experimental results identified the effectiveness of HSC incorporation in HFFD to improve its performance reducing wrong detection of faces.

Keywords: Haar feature based face detection; OpenCV; skin color analysis; RGB;

Face detection is a very important and challenging matter in the field of image processing. It is also a crucial step of face recognition. Face recognition has distinct advantages over other biometrics systems (e.g., finger print, palm print etc.) because of its non-contact process. Face images can be captured from a distance without touching the person who is being identified and the identification does not require interacting with the person. In addition, face recognition serves the crime deterrent purpose because face images that have been recorded and archived can later help to identify a person.

Face detection is one of the tasks that can be done easily by human vision but very complex for a machine. Several studies are available on face detection which discusses different techniques like feature based face detection, example based face detection, geometric based face detection, neural network based face detection and many more [1-3]. Now a day's almost every smart phone uses face detection for automatic focusing on the face area when capturing a photo.

The main goal of any face detection system is to achieve a very high detection rate along with low computational cost. A face detection method detects facial features and ignores anything else, such as building, trees and also bodies. However, many face detection method fail to detect correct faces from images. David, Kriegman, and Ahuja presented a survey of face detection and presented the trends of researches in face detection [4]. In the survey authors categorized and evaluated different face detection algorithms. Some limitations of those algorithms were also discussed in a brief. A common problem of the existing methods is that they treat non facial area as a facial area. The popular Haar like feature based face detection [9, 12] also suffers from the same problem.

This paper presents a technique to improve feature based face detection introducing human skin color (HSC) characteristic. A number studies are available on human skin color based face detections [5-8, 13-25]. The methods analyzed different color spaces (e.g., RGB, YCbCr, HIS, TSL, HSV) and their main focus was to generate a rule with the help of these color spaces which can determine whether a color is similar to human skin color or not. Different studies have also shown different techniques to model human skin color. In this study, HSC property has been incorporated with the popular Haar Feature Based Face Detection (HFFD) in OpenCV, and found to improve its performance.

The rest of the paper is organized as follows. Section II explains HAAR feature based face detection method. Section III presents the proposed face detection technique incorporating of human skin color analysis in HAAR. Section IV presents experimental results to identify the proficiency of the proposed method. Section V concludes the paper with a brief summary.

OpenCV is a very popular tool for object detection. Any types of objects including human faces can be detected by it. Currently OpenCV is using Haar feature based cascaded classifier for face detection [10]. At first the classifier is trained with a lot of positive images (the images containing particular object like car or face we are interested to detect) scaled to same size say 20x20 resolution. And then the classifier is trained with some negative images (arbitrary images that does not contain that particular object like car or face) of same size. After completion of the training process the classifier capture frequently happening features or pattern throughout the whole training images. Then the classifier can be applied to the region of interest to detect that particular object. In the detecting process the classifier finds those features throughout the region of interest and returns the coordinates of that particular object in the input region. Otherwise it does not return any coordinate. OpenCV contains many pre-trained classifier for detecting face, eye, upper body, smile, nose etc. Following subsections briefly explains HAAR feature selection and detecting process of cascade classifier those are used by OpenCV for face detection.

HAAR features are very popular for face detection. Figure 1 shows such features whose are pixel based rectangular patterns. Human face can be represented with these rectangular features. A rectangular feature is like a rectangular area that represents a part of an object. Figure 2 shows only three features representing eye area, nose area and mouth area of a human face.

The task of classifier is to find HAAR features throughout the input image. As the face size varies with image size, classifies uses a window that can be scaled at any size and can be moved to the any position. This scaling is achieved by the detector itself rather than scaling images. And this window is moved left to right and upper to down throughout the whole images. The classifier searches the features across this window. These features are also scaled to any multiply. For an example, for 24x24 resolution window the land for edge feature 1(b) (shown in Figure 1) scaled with 2x2 pixels, the classifier will find this 2x2 pixels feature throughout the whole window. Then this features size will be increased to 4x4 pixels and again classifier will find this 4x4 pixels feature to the whole window. This process is done for several scales. Thus the combination of the features, scaling and positions become very large (more than 160,000); thus incurs a very high computational cost. To overcome this situation, Adboost is used to select some important features those explain a face very well from the large number of features. Such selection discards a large amount of unnecessary computation during the training process. The first two important features selected by AdaBoost are shown in Figure 3.

Finally, to detect a rectangular feature (F) from an image, the sum of the pixels which lie within the shaded rectangles ( Ps ) were subtracted from the sum of pixels in the white rectangles ( P_w ).

Figure 4 shows this rectangular feature selection. If rectangular feature value is positive or above some threshold value, it was considered as a facial feature. Otherwise it is not considered as a facial feature.

The set of features was quite large and computing these feature value one by one is very inefficient for a large image. To overcome this problem, a technique called "Integral Image" is used through which rectangle features could be computed very easily. It is an intermediate representation of an image in which a location (x, y) contains the sum of the pixels above and to the left

of (x, y), inclusive:
$$I{{\displaystyle \sum }}^{\text{}}\left(\text{x},\text{ y}\right){{\displaystyle =\sum }}^{\text{}}\text{ i}(\text{x' },\text{y'}) \left(2\right)$$ $$\begin{array}{l}x\text{'} \le x\\ y\text{'} \le y\\ Sum = I \left(C\right)+ I \left(A\right)– I \left(B\right)– I \left(D\right) \left(3\right)\end{array}$$

Where points A, B, C, D belong to the integral image I, as shown in the Figure 5. To find the rectangular feature value in the dotted area in Figure 4 with the help of integral image, calculations are as follows.

Area X = Sum of pixels in the rectangular area from the lefttop corner to pixel X.

Area A = 0
Area B= 0+2+7 = 9
Area C = 0+3 = 3
Area P= 0+2+7+3+3+1 = 16
Area D= 0+3+4 = 7
Area Q= 0+2+7+3+3+1+4+8+7 = 35

Number of pixels in white area in doted area = (Area Q + Area C - Area D – Area P) = 35 + 3 - 7 - 16 = 15
Number of pixels in shaded area in doted area = (Area A + Area P - Area B – Area C) = 0 + 16 - 9 - 3 = 4
Rectangular feature value in doted area f = 15 - 4 = 11

Though Adaboost has narrowed down the number of features to a few hundred, matching those few hundred to each window is time consuming. To overcome this problem HFFD uses a 38 layer cascaded classifier to detect frontal upright faces. In the cascaded structure 38 classifiers are placed in the hierarchical manner. The first some classifiers are trained with most important features. The number of features in the first five layers of the detector is 1, 10, 25, 25 and 50 features respectively. Finally, to detect a window containing face from an image, classifiers were applied on that window one by one. If a window fails at the first stage, it is discarded and no more classifier is applied on it. If a window passes through a classifier then next classifier is applied on it. If all the facial features are found by the cascaded classifiers then output is positive (i.e., there is a face in the image) otherwise output is negative. The advantages of the cascaded classifier are that it can discard an irrelevant window without matching with all of the features. Figure 6 shows the structure of cascade classifier.

A common problem is encountered in traditional HFFD of OpenCV is that it detects some objects or areas as facial areas but those are not human faces at all. The reason behind is that HAAR feature may matched with other objects which is not face. To overcome this problem we have analyzed human skin color and incorporated in HFFD. This section first explains properties of human skin color and then incorporates it in HFFD.

In this study, we have analyzed human skin color in different color spaces like RGB, HSV, HSL, CMYK etc. and we have found a very interesting characteristic of human skin color in RGB color space. Figure 7 shows different skin color tones. The common property of human skin color is observed that the red component is always greater than the green component (i.e., R>G) and the

green component is always greater than the blue component (i.e., G>B).

To show how this characteristic behaves on real human image, an analysis on images of five people having different skin colors is presented in Figure 8. Images have been selected carefully for proper understanding of the matter and the people of images have distinct skin colors of dark, darkish, light dark, white and very white. Facial area of an image was divided into 100 blocks with some consecutive pixels. Average RGB value of each block is measured and plotted in line charts presented in the right side of corresponding images. In the line chart, X axis represents the direction of block numbers and Y axis depicts the average values of red, green and blue components of corresponding block individually. From the line charts, it is clearly observed that R>G>B conditions are matched in almost all blocks for any image. Few exceptions are acceptable for eye, mustache or beard region blocks. Table 1 shows summary of RBG counts of 100 blocks for each image of Figure 8. From the table it is found that R>G>B is true for 90 or above for all the faces. Finally, it is obvious that R>G>B is true for average RBG values of whole facial area.

In the proposed method RGB characteristics of HSC is incorporated to make face detection process more accurate with HFFD. HFFD method replies several non-facial regions as faces. Such non-facial region will not satisfy the condition of R>G>B of HSC. As computing the RGB value of each pixel is

much time consuming, we have developed a model which selects pixels at 5 pixel interval of detected face area by Rectangular Haar-like feature process. Then RGB value of each selected pixel is calculated and if majority of the selected pixels satisfy the proposed relation then it is considered as a face otherwise not. Figure 9 shows flow chart of proposed face detection method.

The algorithm was implemented by Java programming on NetBeans 7.4 IDE. Experiments have been conducted on a single machine (Fujitsu LH531, Intel (R) Core i3 CPU 2.1 GHz, 4 GB RAM) with Windows 8 Professional OS. We have used OpenCv library to implement HFFD. For drawing line chart we have used Gnuplot 5.0.

To investigate the effectiveness human skin color analysis incorporation in face detection, the proposed method was tested on large number images. After adding skin color feature much wrong detection were reduced. Consider four images shown in Figures 10, 11, 12 and 13. Red rectangles are the result of standard HFFD and green rectangles are the result of proposed HFFD-HSC. Based on this four sample images a performance measurement is shown in Table 2. For sample Image 1 number of actual faces in the image was four. But HFFD detected five faces with one Wrong Detection (WD). On the other hand, proposed method detected faces properly eliminating wrong detection made by HFFD. For Image 2, 3 and 4 the detection process also improved eliminating several wrong face detections.

Sample Image 2 has complex background and performance of HFFD is very poor for the image. Although proposed HFFDHSC reduced wrong detection, still six WD is available for the image. For better understanding, Table 3 presents status of HSC incorporation on all 13 faces detected by HFFD. Table presents correctness ratio with Number of Pixels Satisfy HSC Condition (NPSH) and Number of Pixels Violet HSC Condition (NPVH). In the table we can see that Faces 3, 4, 6, 8 and 10 have their correctness ratio less than 50%. Therefore, those five faces were eliminated by our proposed method as the wrong detections.

We have applied our proposed method and compared with

standard HFFD on first 10 datasets of put face database [11]. There are total 9971 images of single face of different people. Table 4 compares the performance between HFFD and proposed HFFD-HSC on the images. It is clearly observed from the result that proposed HFFD-HSC removed much Wrong Detection (WD) of HFFD. As an example, dataset 1 has 1022 images with single face in each image. Standard HFFD replied total 1034 faces with total 12 wrong identification. For the same dataset, proposed HFFD-HSC replied 1024 faces with only two wrong identification.

For datasets 3, 6, 8, 10 WD is zero for the proposed method and showed 100% accuracy. From the table it is clearly revealed that proposed HSC incorporation is an effective technique to improve performance of HFFD.

Table 4 also shows the required time in seconds for individual data set by HFFD and proposed HFFD-HSC. It is worthy notable that proposed HFFD-HSC verifies the outcome of HFFD with human skin color characteristics and require additional computational time. However, additional time required by HFFD-HSC is minimal. As an example, required time for data set 1 were 1561 seconds and 1600 seconds by HFFD and HFFD-HSC, respectively. Therefore, additional time required by HFFD-HSC is only 39 seconds (=1600-1561) for 1022 images of the data set.

The proposed HFFD-HSC works on RGB skin color property (i.e., R>G>B) and therefore this section investigates the performance on images with different light effect. Figure 14 presents face detected by HFFD and HFFD-HSC on nine different images with different lighting effects. The light effects on the images are flash light, spot light, sidelight, backlight, red light, green, blue light etc. From the figure it is interesting to observe that proposed HFFD-HSC only failed to recognize for the cases of green light (h) and blue light (i). In such cases, 'R>G>B' condition violated due to increase of green and blue components. On the other hand, red light is found positive effect on the proposed HFFD-HSC in Figure 14(g).

For better understanding, Table 5 shows individual face wise analysis of each image of Figure 14. Among nine images, three contains (i.e., a, c and e) more than one faces. From the table it is observed that for first image (i.e., 14(a)) the true face is verified by proposed HFFD-HSC and WD of face 1 by HFFD in bottom left corner is properly identified with correctness ratio 0%. Similar effect also found for Figure 14(c) and WD of face 1 by HFFD in middle left position is properly identified with correctness ratio 9%. On the other hand, due to green and blue light effects, proposed HFFD-HSC failed to verify the true faces identified by HFFD in Figure 14(h) and Figure 14(i). Finally, only extreme images with green and blue light effects the proposed method failed to improve performance.

This paper shows a technique to reduce wrong detections by HFFD incorporating human skin color characteristic measure. The proposed method improved performance decreasing wrong detection rate. Experimental results on large number of images of put face database [11] and some complex images revealed that human skin incorporation is an effective technique to improve performance of HFFD.

1. Chauhan M, Sakle M. Study & Analysis of Different Face Detection Techniques. International Journal of Computer Science and Information Technologies (IJCSIT). 2014;5(2):1615-1618.
2. Sung K, Poggio T. Example-based learning for view-based human face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1998;20(1):39-51.
3. Atharifard A, Ghofrani S. Robust Component-Based Face Detection Using Color Feature. Proceedings of the World Congress on Engineering. 2011;II.
4. David MY, Kriegman J, Ahuja N. Detecting Faces in images: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2002;24(1):34-58.
5. Kovac J, Peer P, Solina F. Human Skin Colour Clustering for Face Detection. EUROCON 2003. Computer as a Tool. 2003.
6. Singh SKr, Chauhan DS, Vatsa M, Singh R. A Robust Skin Color Based Face Detection Algorithm. Tamkang Journal of Science and Engineering. 2003;6(4):227-234.
7. Rahman NAA, Wei KC, See J. RGB-H-CbCr Skin Color Model for Human Face Detection.  MMU International Symposium on Information & Communications Technologies (M2USIC). 2006.
8. Zangana HM, Al-Shaikhli IF. A New Algorithm for Human Face Detection Using Skin Color Tone. IOSR Journal of Computer Engineering. 2013;11(6):31-38.
9. Viola P, Jones M. Rapid Object Detection using a Boosted Cascade of Simple Features. In Prof. of the 2001 IEEE Computer Vision and Pattern Recognition (CVRP). 2001.
10. OpenCV face detection.  Available from:  http://docs.opencv.org/trunk/d7/d8b/tutorial_py_face_detection.html
11. Put face database. Available from:  https://biometrics.cie.put.poznan.pl/index.php?view=article&id=4
12. Papageorgiou CP, Oren M, Poggio T. A General Framework for Object Detection. International Conference on Computer Vision. 1998.
13. Chandrappa DN, Ravishankar M, Babu DRR. Face Detection in Color Images Using Skin Color Model Algorithm Based on Skin Color Information. In Proc. of the 3^rd International Conference on Electronics Computer Technology (ICECT). 2011;1:254-258.
14. Phung SL, Bouzerdoum A, Chai D.  A Novel Skin Color Model in YCbCr Color Space and its Applications to Human Face Detection. In Proc. of the International Conference on Image Processing. 2002:289-292.
15. Li Z, Xue L, Tan F. Face Detection in Complex Background Based on Skin Color Features and Improved Adaboost Algorithm. In Proc. of the IEEE International Conference on Progress in Informatics and Computing. 2010:723-727.
16. Hsu RL, Abdel-Mottaleb M, Jain AK. Face detection in color images. IEEE Trans. PAMI. 2002;24(5):696-707.
17. Phung SL, Chai D, Bouzerdoum A. A universal and robust human skin color model using neural networks. Proc.  IJCNN'01.2001:2844-2849.
18. Gundimada S, Tao L, Asari V. Face Detection Technique Based on Intensity and Skin Color Distribution. ICIP2004. 2004:1413-1416.
19. Wang Y, Yuan B. A Novel Approach for Human Face Detection from Color Images under Complex Background. Pattern Recognition. 2001;34(10):1983-1992.
20. Hsu RL, Abdel-Mottaleb M, Jain AK. Face Detection in Color Images. IEEE Transaction on Pattern Analysis and Machine Intelligence. 2002;24(5):696-706.
21. Vezhnevets V, Sazonov V, Andreeva A. A survey on pixel based skin color detection techniques. In: Proc. Graphicon. 2003:85-92.
22. Vijay Lakshmi HC, Patil Kulakarni S. Segmentation  Algorithm for Multiple Face Detection in Color Images with  Skin Tone Regions using Color Spaces and Edge Detection  Techniques. In International Journal of Computer Theory and Engineering. 2010;2(4);1793-8201.
23. Vijay Lakshmi HC, Patil Kulakarni S. Face Localization and Detection Algorithm for Colour Images using Wavelet Approximations. International Conference ACVIT-09. 2009.
24. Garcia C, Tziritas G. Face Detection Using Quantized Skin Color Regions Merging and Wavelet Packet Analysis. IEEE Trans. Multimedia. 1999;1(3):264-277.
25. Jones MJ, Rehg JM. Statistical color models with application to skin detection. Proc. CVPR'99. 1999:274-280.