Generic placeholder image

Recent Patents on Engineering

Author(s): Guoqiang Chen*, Bingxin Bai, Hongpeng Zhou, Mengchao Liu and Huailong Yi

DOI: 10.2174/1872212115666210827100258

DownloadDownload PDF Flyer Cite As
Facemask Detection Based on Double Convolutional Neural Networks

Article ID: e270821195862 Pages: 12

  • * (Excluding Mailing and Handling)

Explore Articles
About Journal
For Authors
For Editors
For Reviewers

Abstract

Background: The study on facemask detection is of great significance because facemask detection is difficult, and the workload is heavy in places with a large number of people during the COVID-19 outbreak.

Objective: The study aims to explore new deep learning networks that can accurately detect facemasks and improve the network's ability to extract multi-level features and contextual information. In addition, the proposed network effectively avoids the interference of objects like masks. The new network could eventually detect masks wearers in the crowd.

Methods: A Multi-stage Feature Fusion Block (MFFB) and a Detector Cascade Block (DCB) are proposed and connected to the deep learning network for facemask detection. The network's ability to obtain information improves. The network proposed in the study is Double Convolutional Neural Networks (CNN) called DCNN, which can fuse mask features and face position information. During facemask detection, the network extracts the featural information of the object and then inputs it into the data fusion layer.

Results: The experiment results show that the proposed network can detect masks and faces in a complex environment and dense crowd. The detection accuracy of the network improves effectively. At the same time, the real-time performance of the detection model is excellent.

Conclusion: The two branch networks of the DCNN can effectively obtain the feature and position information of facemasks. The network overcomes the disadvantage that a single CNN is susceptible to the interference of the suspected mask objects. The verification shows that the MFFB and the DCB can improve the network's ability to obtain object information, and the proposed DCNN can achieve excellent detection performance.

Keywords: Face mask detection, MFFB, DCB, DCNN, data fusion, deep learning.

Graphical Abstract

[1]
Y.C. Chang, H.T. Chen, and J.H. Chuang, "Pedestrian detection in aerial images using vanishing point transformation and deep learning", 25th IEEE International Conference on Image Processing, Athens, Greece, 2018, pp. 1917-1921.
[http://dx.doi.org/10.1109/ICIP.2018.8451144]
[2]
Y.Y. Qu, L. Jiang, and X.P. Guo, "Moving vehicle detection with convolutional networks in UAV videos", 2nd International Conference on Control, Automation and Robotics, Hong Kong, China, 2016, pp. 225-229.
[http://dx.doi.org/10.1109/ICCAR.2016.7486730]
[3]
L. Wang, and Z.J. Zhang, "Automatic detection of wind turbine blade surface cracks based on UAV-taken images", IEEE Trans. Ind. Electron., vol. 64, no. 9, pp. 7293-7303, 2017.
[http://dx.doi.org/10.1109/TIE.2017.2682037]
[4]
D.H. Xu, C.B. Wen, and J.H. Liu, "Wind turbine blade surface inspection based on deep learning and UAV-taken images", J. Renew. Sustain. Energy, vol. 11, no. 5, p. 053305, 2019.
[http://dx.doi.org/10.1063/1.5113532]
[5]
J.C. Ma, Y.X. Li, and K. Du, "Segmenting ears of winter wheat at flowering stage using digital images and deep learning", Comput. Electron. Agric., vol. 168, p. 105159, 2020.
[http://dx.doi.org/10.1016/j.compag.2019.105159]
[6]
Q. Yang, L.S. Shi, and J.Y. Han, "A near real-time deep learning approach for detecting rice phenology based on UAV images", Agric. For. Meteorol., vol. 287, no. 107938, 2020.
[http://dx.doi.org/10.1016/j.agrformet.2020.107938]
[7]
Y.L. Pi, N.D. Nath, and A.H. Behzadan, "Convolutional neural networks for object detection in aerial imagery for disaster response and recovery", Adv. Eng. Inform., vol. 43, p. 101009, 2020.
[http://dx.doi.org/10.1016/j.aei.2019.101009]
[8]
R. Thomazella, J.E. Castanho, and F.R.L. Dotto, "Environmental Monitoring Using Drone Images and Convolutional Neural Networks", IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, 2018, pp. 8941-8944.
[http://dx.doi.org/10.1109/IGARSS.2018.8518581]
[9]
S. Sudhakar, V. Vijayakumar, and C.S. Kumar, "Unmanned Aerial Vehicle (UAV) based forest fire detection and monitoring for reducing false alarms in forest-fires", Comput. Commun., vol. 149, pp. 1-16, 2020.
[http://dx.doi.org/10.1016/j.comcom.2019.10.007]
[10]
Y. Zhou, T. Rui, and Y.R. Li, "A UAV patrol system using panoramic stitching and object detection", Comput. Electr. Eng., vol. 80, no. 106473, 2019.
[http://dx.doi.org/10.1016/j.compeleceng.2019.106473]
[11]
X.M. Mai, Z.Y. Pan, and J.J. Qian, "Anomaly Detection of Power Transmission Line Using Stereo Vision-Based Multi-Rotor UAV", 2018 Chinese Automation Congress, Xi'an, China, 2018, pp. 2440-2445.
[http://dx.doi.org/10.1109/CAC.2018.8623277]
[12]
Z.Y. Jiao, G.Z. Jia, and Y.J. Cai, "A new approach to oil spill detection that combines deep learning with unmanned aerial vehicles", Comput. Ind. Eng., vol. 135, pp. 1300-1311, 2019.
[http://dx.doi.org/10.1016/j.cie.2018.11.008]
[13]
J.Y. Ruan, and Z.L. Wang, "An Improved Algorithm for Dense Object Detection Based on YOLO", International Conference on Computer Engineering and Networks, 2018pp. 592-599
[14]
X. Zeng, Y.P. Wu, and S.Z. Hu, "DSPNet: Deep scale purifier network for dense crowd counting", Expert Syst. Appl., vol. 141, no. 112977, 2020.
[http://dx.doi.org/10.1016/j.eswa.2019.112977]
[15]
Y.P. Guo, Y. Xu, and S.L. Li, "Dense construction vehicle detection based on orientation-aware feature fusion convolutional neural network", Autom. Construct., vol. 112, no. 103124, 2020.
[http://dx.doi.org/10.1016/j.autcon.2020.103124]
[16]
A. Fagette, N. Courty, and D. Racoceanu, "Unsupervised dense crowd detection by multiscale texture analysis", Pattern Recognit. Lett., vol. 44, pp. 126-133, 2014.
[http://dx.doi.org/10.1016/j.patrec.2013.09.020]
[17]
B. Huang, H. Xiong, and J.Q. Wang, "Detection-level fusion for multi-object perception in dense traffic environment", 2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Daegu, Korea (South), 2017, pp. 411-416.
[http://dx.doi.org/10.1109/MFI.2017.8170355]
[18]
Z. Ji, Q.K. Kong, and H.R. Wang, "Small and dense commodity object detection with multi-scale receptive field attention", Proceedings of the 27th ACM international conference on multimedia, nice, France, 2019, pp. 1349-1357.
[http://dx.doi.org/10.1145/3343031.3351064]
[19]
J. Bai, J. Zhu, and R. Zhao, "Area-based non-maximum suppression algorithm for multi-object fault detection", Front Optoelectron., pp. 1-8, 2020.
[20]
H. Liang, H. Song, and H. Li, "Vehicle counting system using deep learning and multi-object tracking methods", Transp. Res. Rec., 2020.
[http://dx.doi.org/10.1177/0361198120912742]
[21]
J. Li, Q. Hou, and J. Xing, "SSD Object Detection Model Based on Multi-Frequency Feature Theory", IEEE Access, vol. 8, pp. 82294-82305, 2020.
[http://dx.doi.org/10.1109/ACCESS.2020.2990477]
[22]
F. Liu, W. Jia, and Z. Yang, "A multi-object tracking method based on bounding box and features", international conference on computer science, engineering and education applications, 2019, pp. 217-227.
[23]
Q. Ji, H. Yu, and X. Wu, "Hierarchical-matching-based online and real-time multi-object tracking with deep appearance features", Algorithms, vol. 13, no. 80, 2020.
[http://dx.doi.org/10.3390/a13040080]
[24]
X. Liu, L. Wang, and Q. Zhang, "Joint multi-object detection and segmentation from an untrimmed video", IFIP International Conference on Artificial Intelligence Applications and Innovations, 2020, pp. 317-329.
[http://dx.doi.org/10.1007/978-3-030-49161-1_27]
[25]
A. Mauri, R. Khemmar, B. Decoux, N. Ragot, R. Rossi, R. Trabelsi, R. Boutteau, J.Y. Ertaud, and X. Savatier, "Deep learning for real-time 3d multi-object detection, localisation, and tracking: application to smart mobility", Sensors (Basel), vol. 20, no. 2, p. e532, 2020.
[http://dx.doi.org/10.3390/s20020532] [PMID: 31963641]
[26]
W. Tian, M. Lauer, and L. Chen, "Online multi-object tracking using joint domain information in traffic scenarios", IEEE Trans. Intell. Transp. Syst., vol. 21, pp. 374-384, 2019.
[http://dx.doi.org/10.1109/TITS.2019.2892413]
[27]
A. Jadhav, P. Mukherjee, and V. Kaushik, "Aerial multi-object tracking by detection using deep association networks", 2020 National Conference on Communications (NCC), Kharagpur, India, 2020, pp. 1-6.
[http://dx.doi.org/10.1109/NCC48643.2020.9056035]
[28]
S. Wu, X.H. Zhou, and J. Lu, "Multi Object Tracking Based on Detection with Deep Learning and Hierarchical Clustering", IEEE 4th International Conference on Image, Vision and Computing (ICIVC), Xiamen, China, 2019, pp. 367-370.
[http://dx.doi.org/10.1109/ICIVC47709.2019.8981026]
[29]
B. Lee, E. Erdenee, and S. Jin, "Multi-class multi-object tracking using changing point detection", European Conference on Computer Vision, 2016, pp. 68-83.
[http://dx.doi.org/10.1007/978-3-319-48881-3_6]
[30]
D. Zhao, H. Fu, L. Xiao, T. Wu, and B. Dai, "Multi-object tracking with correlation filter for autonomous vehicle", Sensors (Basel), vol. 18, no. 7, p. 2004, 2018.
[http://dx.doi.org/10.3390/s18072004] [PMID: 29932136]
[31]
Y. Xia, S. Qu, and S. Goudos, "Multi-object tracking by mutual supervision of CNN and particle filter", Pers. Ubiquitous Comput., pp. 1-10, 2019.
[http://dx.doi.org/10.1007/s00779-019-01278-1]
[32]
C. Xu, and Y. Zhou, "Consistent online multi-object tracking with part-based deep network", Chinese Conference on Pattern Recognition and Computer Vision (PRCV), 2018, pp. 180-192.
[http://dx.doi.org/10.1007/978-3-030-03335-4_16]
[33]
J. Son, M. Baek, and M. Cho, "Multi-object tracking with quadruplet convolutional neural networks", Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 2017, pp. 5620-5629.
[http://dx.doi.org/10.1109/CVPR.2017.403]
[34]
X. Ke, J.P. Li, and W.Z. Guo, "Dense small face detection based on regional cascade multi-scale method", IET Image Process., vol. 13, pp. 2796-2804, 2019.
[http://dx.doi.org/10.1049/iet-ipr.2018.6571]
[35]
M. Sabokrou, M. Fayyaz, M. Fathy, and R. Klette, "Deep-cascade: Cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes", IEEE Trans. Image Process., vol. 26, no. 4, pp. 1992-2004, 2017.
[http://dx.doi.org/10.1109/TIP.2017.2670780] [PMID: 28221995]
[36]
M. Sabokrou, M. Fayyaz, and M. Fathy, "Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes", Comput. Vis. Image Underst., vol. 172, pp. 88-97, 2018.
[http://dx.doi.org/10.1016/j.cviu.2018.02.006]
[37]
K. Singh, S. Rajora, and D.K. Vishwakarma, "Crowd anomaly detection using Aggregation of Ensembles of fine-tuned ConvNets", Neurocomputing, vol. 371, pp. 188-198, 2020.
[http://dx.doi.org/10.1016/j.neucom.2019.08.059]
[38]
H.N. Xie, T. Wang, and M.N. Qiao, "Robust object detection for tiny and dense targets in VHR aerial images", 2017 Chinese Automation Congress, Jinan, China, 2017, pp. 6397-6401.
[http://dx.doi.org/10.1109/CAC.2017.8243930]
[39]
S.C. Xie, X.H. Zhang, and J. Cai, "Video crowd detection and abnormal behavior model detection based on machine learning method", Neural Comput. Appl., vol. 31, pp. 175-184, 2019.
[http://dx.doi.org/10.1007/s00521-018-3692-x]
[40]
M. Yang, S. Rajasegarar, and S.M. Erfani, "Deep Learning and One-class SVM based Anomalous Crowd Detection", 2019 International Joint Conference on Neural Networks, Budapest, Hungary, 2019, pp. 1-8.
[http://dx.doi.org/10.1109/IJCNN.2019.8852256]
[41]
Y.X. Chen, W.R. Ding, and H.G. Li, "Arbitrary-Oriented Dense Object Detection in Remote Sensing Imagery", IEEE 9th International Conference on Software Engineering and Service Science, Beijing, China, 2018, pp. 436-440.
[42]
Z. Ou, F. Xiao, and B. Xiong, "FAMN: Feature Aggregation Multipath Network for Small Traffic Sign Detectio", IEEE Access, vol. 7, pp. 178798-178810, 2019.
[http://dx.doi.org/10.1109/ACCESS.2019.2959015]
[43]
L.T. Guan, Y. Wu, and J.Q. Zhao, "Scan: semantic context aware network for accurate small object detection", Int. J. Comput. Intel. Sys., vol. 11, pp. 951-961, 2018.
[http://dx.doi.org/10.2991/ijcis.11.1.72]
[44]
C.Q. Cao, B. Wang, and W.R. Zhang, "An improved faster r-cnn for small object detection", IEEE Access, vol. 7, pp. 106838-106846, 2019.
[http://dx.doi.org/10.1109/ACCESS.2019.2932731]
[45]
C. Chen, J.D. Zhong, and Y. Tan, "Multiple-oriented and small object detection with convolutional neural networks for aerial image", Remote Sens., vol. 11, no. 2176, 2019.
[http://dx.doi.org/10.3390/rs11182176]
[46]
Z.X. Du, J.Y. Yin, and J. Yang, "Expanding receptive field yolo for small object detection", J. Phys. Conf. Ser., vol. 1314, no. 012202, 2019.
[http://dx.doi.org/10.1088/1742-6596/1314/1/012202]
[47]
P.C. Fang, and Y.J. Shi, "Small object detection using context information fusion in faster r-cnn", IEEE 4th International Conference on Computer and Communications, Chengdu, China, 2018, pp. 1537-1540.
[http://dx.doi.org/10.1109/CompComm.2018.8780579]
[48]
S.R. Klomp, D.W.J.M. van de Wouw, and P.H.N. de With, "Real-time small-object change detection from ground vehicles using a siamese convolutional neural network", J. Imaging Sci. Technol., vol. 63, pp. 1062-3701, 2019.
[http://dx.doi.org/10.2352/J.ImagingSci.Technol.2019.63.6.060402]
[49]
H. Krishna, and C.V. Jawahar, "Improving small object detection", 2017 4th IAPR Asian Conference on Pattern Recognition, Nanjing, China, 2017, pp. 340-345.
[50]
H.T. Li, K.Z. Lin, and J.X. Bai, "Small object detection algorithm based on feature pyramid-enhanced fusion ssd", Complexity, vol. 2019, 2019.7297960
[http://dx.doi.org/10.1155/2019/7297960]
[51]
Y. Zhang, and T. Shen, "Small object detection with multiple receptive fields", E&ES, vol. 440, p. 032093, 2020.
[http://dx.doi.org/10.1088/1755-1315/440/3/032093]
[52]
B.A. Mudassar, and S. Mukhopadhyay, "Rethinking convolutional feature extraction for small object detection", British Machine Vision Conference (BMVC), 2019, p. 234.
[53]
N.D. Nguyen, T. Do, and T.D. Ngo, "An evaluation of deep learning methods for small object detection", J. Electr. Comput. Eng., p. 3189691, 2020.2020
[http://dx.doi.org/10.1155/2020/3189691]
[54]
B. Bosquet, M. Mucientes, and V.M. Brea, "STDnet: Exploiting high resolution feature maps for small object detection", Eng. Appl. Artif. Intell., vol. 91, no. 103615, 2020.
[http://dx.doi.org/10.1016/j.engappai.2020.103615]
[55]
M. Liu, X. Wang, A. Zhou, X. Fu, Y. Ma, and C. Piao, "UAV-YOLO: Small Object Detection on Unmanned Aerial Vehicle Perspective", Sensors (Basel), vol. 20, no. 8, p. E2238, 2020.
[http://dx.doi.org/10.3390/s20082238] [PMID: 32326573]
[56]
Y. Li, H. Dong, and H. Li, "Multi-block ssd based small object detection for uav railway scene surveillance", Chin. J. Aeronauti., vol. 33, no. 6, pp. 1747-1755, 2020.
[http://dx.doi.org/10.1016/j.cja.2020.02.024]
[57]
C. Yu, K. Liu, and W. Zou, "A method of small object detection based on improved deep learning", Opt. Mem. Neural. Networks, vol. 29, pp. 69-76, 2020.
[http://dx.doi.org/10.3103/S1060992X2002006X]
[58]
J. Liu, S.J. Yang, and L. Tian, "Multi-component fusion network for small object detection in remote sensing images", IEEE Access, vol. 7, pp. 128339-128352, 2019.
[http://dx.doi.org/10.1109/ACCESS.2019.2939488]
[59]
H.W. Luo, C.S. Zhang, and F.C. Pan, "Contextual-yolov3: implement better small object detection based deep learning", 2019 International Conference on Machine Learning, Big Data and Business Intelligence, Taiyuan, China, 2019, pp. 134-141.
[http://dx.doi.org/10.1109/MLBDBI48998.2019.00032]
[60]
Z. Yang, Y.L. Liu, and L.R. Liu, "Detecting small objects in urban settings using slimnet model", IEEE Trans. Geosci. Remote Sens., vol. 57, pp. 8445-8457, 2019.
[http://dx.doi.org/10.1109/TGRS.2019.2921111]
[61]
F. Pérez-Hernández, S. Tabik, and A. Lamas, "Object detection binary classifiers methodology based on deep learning to identify small objects handled similarly: Application in video surveillance", Knowl. Base. Syst., p. 105590, 2020.
[http://dx.doi.org/10.1016/j.knosys.2020.105590]
[62]
M. Ahmad, M. Abdullah, and D. Han, "Small Object Detection in Aerial Imagery using RetinaNet with Anchor Optimization", 2020 International Conference on Electronics, Information, and Communication (ICEIC), Barcelona, Spain, 2020, pp. 1-3.
[63]
J. Zhou, B. Chen, and J. Zhang, "Multi-scales feature integration single shot multi-box detector on small object detection", MIPPR 2019: Pattern Recognition and Computer Vision International Society for Optics and Photonics, vol. 11430, 2020.
[http://dx.doi.org/10.1117/12.2538020]
[64]
X. Zhang, J. Wu, and Z. Peng, "SODNet: small object detection using deconvolutional neural network", IET Image Process., vol. 14, pp. 1662-1669, 2020.
[http://dx.doi.org/10.1049/iet-ipr.2019.0833]
[65]
J. Qu, C. Su, and Z. Zhang, "Dilated convolution and feature fusion ssd network for small object detection in remote sensing images", IEEE Access, vol. 8, pp. 82832-82843, 2020.
[http://dx.doi.org/10.1109/ACCESS.2020.2991439]
[66]
S. Jiang, W. Yao, and M.S. Wong, "An optimized deep neural network detecting small and narrow rectangular objects in google earth images", IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 13, pp. 1068-1081, 2020.
[http://dx.doi.org/10.1109/JSTARS.2020.2975606]
[67]
D.W. Ma, X.J. Wu, and H. Yang, "Efficient small object detection with an improved region proposal networks", IOP Conference Series: Materials Science and Engineering IOP Publishing, vol. 533. 2019, p. 012062.
[http://dx.doi.org/10.1088/1757-899X/533/1/012062]
[68]
M. Tan, R. Pang, and Q.V. Le, "Efficientdet: Scalable and efficient object detection", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10781-10790.
[69]
S.F. Zhang, L.Y. Wen, and X. Bian, "Single-Shot Refinement Neural Network for Object Detection", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 4203-4212.
[http://dx.doi.org/10.1109/CVPR.2018.00442]
[70]
W. Liu, D. Anguelov, and D. Erhan, "SSD: Single Shot MultiBox Detector", european conference on computer vision, pp. 21-37, 2016.
[http://dx.doi.org/10.1007/978-3-319-46448-0_2]
[71]
T.Y. Lin, P. Dollar, and R. Girshick, "Feature pyramid networks for object detection", Comput. Vis. Patt. Recognit, pp. 936-944, 2017.
[http://dx.doi.org/10.1109/CVPR.2017.106]
[72]
K. Simonyana, and A. Zisserman, "Very deep convolutional networks for large-scale image recognition", Comput. Sci., 2014.
[73]
K.M. He, X.Y. Zhang, and S.Q. Ren, "Deep residual learning for image recognition", Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.