Generic placeholder image

Recent Patents on Engineering

Author(s): Pooja Jain*, Kavita Taneja and Harmunish Taneja

DOI: 10.2174/1872212115666210715163919

DownloadDownload PDF Flyer Cite As
Convolutional Neural Network Based Intelligent Advertisement Search Framework for Online English Newspapers

Article ID: e150721194823 Pages: 19

  • * (Excluding Mailing and Handling)

Abstract

Background: Instant access to desired information is the key element for building an intelligent environment creating value for people and steering towards society 5.0. Online newspapers are one such example which provide instant access to information anywhere and anytime on our mobiles, tablets, laptops, desktops, etc. But when it comes to searching for a specific advertisement in newspapers, online newspapers do not provide easy advertisement search options. Also, there are no specialized search portals which can provide for keyword-based advertisement search across multiple online newspapers. As a result, to find a specific advertisement in multiple newspapers, a sequential manual search is required across a range of online newspapers.

Objective: This research paper proposes a keyword-based advertisement search framework to provide an instant access to the relevant advertisements from online English newspapers in a category of reader’s choice.

Methods: First, an image extraction algorithm is proposed which can identify and extract the images from online newspapers without using any rules on advertisement placement and/or size. It is followed by a proposed deep learning Convolutional Neural Network (CNN) model named ‘Adv_Recognizer’ which is used to separate the advertisement images from non-advertisement images. Another CNN Model, ‘Adv_Classifier’, is proposed, which classifies the advertisement images into four pre-defined categories. Finally, Optical Character Recognition (OCR) technique is used to perform keyword-based advertisement searches in various categories across multiple newspapers.

Results: The proposed image extraction algorithm can easily extract all types of well-bounded images from different online newspapers and this algorithm is used to create ‘English newspaper image dataset’ of 11,000 images, including advertisements and non-advertisements. The proposed ‘Adv_Recognizer’ model separates advertisement and non-advertisement images with an accuracy of around 97.8%. and the proposed ‘Adv_Classifier’ model classifies the advertisements in four predefined categories exhibiting an accuracy of around 73.5%.

Conclusion: The proposed framework will help newspaper readers in performing exhaustive advertisement searches across a range of online English newspapers in a category of their own interest. It will also help in carrying out advertisement analysis and studies.

Keywords: Advertisement image classification, convolutional neural networks (CNN), newspaper advertisements, newspaper layout segmentation, optical character recognition (OCR), residual networks (ResNet), transfer learning.

Graphical Abstract

[1]
T. Cover, and P. Hart, "Nearest neighbor pattern classifica-tion", IEEE Trans. Inf. Theory, vol. 13, no. 1, pp. 21-27, 1967.
[http://dx.doi.org/10.1109/TIT.1967.1053964]
[2]
S.K. Murthy, "Automatic construction of decision trees from data: A multi-disciplinary survey", Data Min. Knowl. Discov., vol. 2, no. 4, pp. 345-389, 1998.
[http://dx.doi.org/10.1023/A:1009744630224]
[3]
I. Rish, "“An empirical study of the naive Bayes classifier”, IJCAI 2001 Workshop Empir", Methods Artif. Intell., vol. 3, no. 22, pp. 41-46, 2001.
[4]
D.D. Lewis, Naive (Bayes) at forty: The independence as-sumption in information retrievalEuropean conference on machine learning: ECML-98, Berlin, Heidelberg, 1998, pp. 4-15.
[http://dx.doi.org/10.1007/BFb0026666]
[5]
C. Cortes, and V. Vapnik, "Support-vector networks", Mach. Learn., vol. 20, no. 3, pp. 273-297, 1995.
[http://dx.doi.org/10.1007/BF00994018]
[6]
J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, and T. Chen, "Recent advances in convolutional neural networks", Pattern Recognit., vol. 77, pp. 354-377, 2018.
[http://dx.doi.org/10.1016/j.patcog.2017.10.013]
[7]
Y. LeCun, K. Kavukcuoglu, and C. Farabet, Convolutional networks and applications in visionProceedings of 2010 IEEE International Symposium on Circuits and Systems Paris, France, 2010, pp. 253-256.
[http://dx.doi.org/10.1109/ISCAS.2010.5537907]
[8]
A. Krizhevsky, I. Sutskever, and G.E. Hinton, "ImageNet classification with deep convolutional neural networks", Commun. ACM, vol. 60, no. 6, pp. 84-90, 2017.
[http://dx.doi.org/10.1145/3065386]
[9]
Y. Guo, Y. Liu, A. Oerlemans, S. Lao, S. Wu, and M.S. Lew, "Deep learning for visual understanding: A review", Neurocomputing, vol. 187, pp. 27-48, 2016.
[http://dx.doi.org/10.1016/j.neucom.2015.09.116]
[10]
A.S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, CNN features off-the-shelf: An astounding baseline for recognition2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops Columbus, OH, USA, 2014, pp. 512-519.
[http://dx.doi.org/10.1109/CVPRW.2014.131]
[11]
M. Bokser, "Omnidocument technologies", Proc. IEEE, vol. 80, no. 7, pp. 1066-1078, 1992.
[http://dx.doi.org/10.1109/5.156470]
[12]
S. Mori, C.Y. Suen, and K. Yamamoto, "Historical review of OCR research and development", Proc. IEEE, vol. 80, no. 7, pp. 1029-1058, 1992.
[http://dx.doi.org/10.1109/5.156468]
[13]
P. Jain, K. Taneja, and H. Taneja, "Which OCR toolset is good and why? A comparative study", Kuwait J. Sci., vol. 48, no. 2, 2021.
[http://dx.doi.org/10.48129/kjs.v48i2.9589]
[14]
R.P. Kaur, and M.K. Jindal, "A survey on newspaper image segmentation techniques", Int J Adv Res Sci Eng, vol. 6, no. 10, pp. 1789-1797, 2017.
[15]
B. Gatos, S.L. Mantzaris, K.V. Chandrinos, A. Tsigris, and S.J. Perantonis, Integrated algorithms for newspaper page de-composition and article trackingProceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR ’99 (Cat. No.PR00318), Bangalore, India, 1999, pp. 559-562.
[http://dx.doi.org/10.1109/ICDAR.1999.791849]
[16]
F. Liu, Y. Luo, M. Yoshikawa, and D. Hu, A new compo-nent based algorithm for newspaper layout analysisProceedings of Sixth International Conference on Document Analysis and Recognition, Seattle, WA, USA,, 2001, pp. 1176-1180.
[17]
P.E. Mitchell, and H. Yan, Newspaper document analysis featuring connected line segmentationProceedings of Sixth International Conference on Document Analysis and Recogni-tion Seattle, WA, USA, 2001, pp. 1181-1185.
[http://dx.doi.org/10.1109/ICDAR.2001.953971]
[18]
P.E. Mitchell, and H. Yan, "Newspaper layout analysis incor-porating connected component separation", Image Vis. Comput., vol. 22, no. 4, pp. 307-317, 2004.
[http://dx.doi.org/10.1016/j.imavis.2003.11.001]
[19]
P.E. Mitchell, and H. Yan, Connected pattern segmentation and title grouping in newspaper imagesProceedings of the 17th International Conference on Pattern Recognition, Cam-bridge England, UK, vol. 1. 2004, pp. 397-400.
[http://dx.doi.org/10.1109/ICPR.2004.1334135]
[20]
R. Furmaniak, Unsupervised newspaper segmentation using language contextNinth International Conference on Document Analysis and Recognition (ICDAR 2007) Curitiba, Parana, Brazil, vol. 2. 2007, pp. 1263-1267.
[http://dx.doi.org/10.1109/ICDAR.2007.4377118]
[21]
K. Chaudhury, A. Jain, S. Thirthala, V. Sahasranaman, S. Saxena, and S. Mahalingam, Google newspaper search-image processing and analysis pipeline2009 10th International Conference on Document Analysis and Recognition, Barcelona, Spain, 2009, pp. 621-625.
[http://dx.doi.org/10.1109/ICDAR.2009.272]
[22]
T. Palfray, D. Hebert, S. Nicolas, P. Tranouez, and T. Paquet, Logical segmentation for article extraction in digitized old newspapersProceedings of the 2012 ACM symposium on Document engineering - DocEng ’12, Paris, France, 2012, p. 129.
[http://dx.doi.org/10.1145/2361354.2361383]
[23]
A. Antonacopoulos, C. Clausner, C. Papadopoulos, and S. Pletschacher, ICDAR 2013 Competition on Historical News-paper Layout Analysis (HNLA 2013)12th International Conference on Document Analysis and Recognition, Washing-ton DC, USA, 2013, pp. 1454-1458.
[http://dx.doi.org/10.1109/ICDAR.2013.293]
[24]
A. Bansal, S. Chaudhury, S.D. Roy, and J.B. Srivastava, Newspaper article extraction using hierarchical fixed point model2014 11th IAPR International Workshop on Document Analysis Systems, Tours, France, 2014, pp. 257-261.
[http://dx.doi.org/10.1109/DAS.2014.42]
[25]
Q. Li, J. Wang, D. Wipf, and Z. Tu, Fixed-point model for structured labelingInternational conference on machine learning, 2013, pp. 214-221.
[26]
W-T. Chu, and H-Y. Chang, Advertisement detection, seg-mentation, and classification for newspaper images and web-site snapshots2016 International Computer Symposium (ICS) Chiayi, Taiwan, 2016, pp. 396-401.
[http://dx.doi.org/10.1109/ICS.2016.0086]
[27]
B. Meier, T. Stadelmann, J. Stampfli, M. Arnold, and M. Cieliebak, “Fully convolutional neural networks for newspa-per article segmentation”, 2017 14th IAPR International Con-ference on Document Analysis and Recognition. (ICDAR) Kyoto, Japan, 2017, pp. 414-419.
[http://dx.doi.org/10.1109/ICDAR.2017.75]
[28]
A. Almutairi, and M. Almashan, Instance segmentation of newspaper elements using mask R-CNN2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA) Boca Raton, FL, USA, 2019, pp. 1371-1375.
[http://dx.doi.org/10.1109/ICMLA.2019.00223]
[29]
S.B. Kotsiantis, I. Zaharakis, and P. Pintelas, "Supervised machine learning: A review of classification techniques", Emerg. Artif. Intell. Appl. Comput. Eng., vol. 160, no. 1, pp. 3-24, 2007.
[30]
T.N. Phyu, "Survey of classification techniques in data min-ing", Proceedings of the 2009 International Multi Conference of Engineers and Computer Scientists, Hong Kong, 2009.
[31]
P. Kamavisdar, S. Saluja, and S. Agrawal, "A survey on image classification approaches and techniques", Int. J. Adv. Res. Comput. Commun. Eng., vol. 2, no. 1, pp. 1005-1009, 2013.
[32]
S. Kaur, and S. Kalra, Disease prediction using hybrid K-means and support vector machine2016 1st India International Conference on Information Processing (IICIP) Delhi, India, 2016, pp. 1-6.
[http://dx.doi.org/10.1109/IICIP.2016.7975367]
[33]
F. Shaheen, B. Verma, and Md. Asafuddoula, Impact of automatic feature extraction in deep learning architecture2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA) Gold Coast, Australia, 2016, pp. 1-8.
[http://dx.doi.org/10.1109/DICTA.2016.7797053]
[34]
P. Jain, K. Taneja, and H. Taneja, "Convolutional neural net-work based advertisement classification models for online english newspapers", Turk. J. Comput. Math. Educ. TURCOMAT, vol. 12, no. 2, pp. 1687-1698, 2021.
[http://dx.doi.org/10.17762/turcomat.v12i2.1505]
[35]
R.A. Peleato, J-C. Chappelier, and M. Rajman, Using infor-mation extraction to classify newspapers advertisementsProceedings of the 5th International Conference on the Statis-tical Analysis of Textual Data Lausanne, Switzerland, 2000, pp. 28-30.
[36]
L-Y. Duan, J. Wang, Y. Zheng, J.S. Jin, H. Lu, and C. Xu, Segmentation, categorization, and identification of commer-cial clips from TV streams using multimodal analysisProceedings of the 14th annual ACM international conference on Multimedia - MULTIMEDIA ’06, Santa Barbara CA, USA, 2006, p. 201.
[http://dx.doi.org/10.1145/1180639.1180697]
[37]
L. Zhang, Z. Zhu, and Y. Zhao, Robust commercial detection systemMultimedia and Expo, 2007 IEEE International Conference on Beijing, China, 2007, pp. 587-590.
[http://dx.doi.org/10.1109/ICME.2007.4284718]
[38]
D. Li, B. Wang, Z. Li, N. Yu, and M. Li, On detection of advertising imagesMultimedia and Expo, 2007 IEEE International Conference on Beijing, China, 2007, pp. 1758-1761.
[http://dx.doi.org/10.1109/ICME.2007.4285011]
[39]
Y. Freund, and R.E. Schapire, "A decision-theoretic generali-zation of on-line learning and an application to boosting", J. Comput. Syst. Sci., vol. 55, no. 1, pp. 119-139, 1997.
[http://dx.doi.org/10.1006/jcss.1997.1504]
[40]
C. Gong, and F. Zhu, On detection of contextual advertise-ments2010 2nd International Asia Conference on Informatics in Control, Automation and Robotics (CAR 2010) Wuhan, China, 2010, pp. 29-32.
[http://dx.doi.org/10.1109/CAR.2010.5456544]
[41]
A. Ouji, Y. Leydier, and F. Lebourgeois, Advertisement detection in digitized press images2011 IEEE International Conference on Multimedia and Expo 2011 Barcelona, Spain, pp. 1-6.
[42]
J-Y. Jung, "Vocabulary expansion technique for advertisement classification", KSII Trans. Internet Inf. Syst., vol. 6, no. 5, pp. 1373-1387, 2012.
[43]
B. Banerjee, Machine learning models for political video ad-vertisement classification, Capstones., Iowa State University, 2017.
[44]
A.T. Vo, H.S. Tran, and T.H. Le, Advertisement image clas-sification using convolutional neural network2017 9th International Conference on Knowledge and Systems Engineering (KSE) Hue, Vietnam, 2017, pp. 197-202.
[http://dx.doi.org/10.1109/KSE.2017.8119458]
[45]
K. Almgren, M. Krishnan, F. Aljanobi, and J. Lee, "AD or Non-AD: A deep learning approach to detect advertisements from magazines", Entropy (Basel), vol. 20, no. 12, p. 982, 2018.
[http://dx.doi.org/10.3390/e20120982] [PMID: 33266705]
[46]
S. Dhiman, and A.J. Singh, "Tesseract Vs Gocr A Comparative Study", Int. J. Recent Technol. Eng., vol. 2, no. 4, p. 80, 2013.
[47]
A. Gabasio, Comparison of optical character recognition (OCR) software Master’s Thesis, Lund University, LTH, 2013.
[48]
C. Patel, D. Shah, and A. Patel, "Automatic Number Plate Recognition System (ANPR): A Survey", Int. J. Comput. Appl., vol. 69, no. 9, pp. 21-33, 2013.
[http://dx.doi.org/10.5120/11871-7665]
[49]
M. Tomaschek, Evaluation of off-the-shelf OCR technolo-gies Ph.D Thesis, Masaryk University, 2018.
[50]
A.P. Tafti, A. Baghaie, M. Assefi, H.R. Arabnia, Z. Yu, and P. Peissig, OCR as a service: An Experimental Evaluation of Google docs ocr, tesseract, abbyy finereader, and transym., Advances in Visual Computing, pp. 735-746, 2016.
[http://dx.doi.org/10.1007/978-3-319-50835-1_66]
[51]
S. Vijayarani, and A. Sakila, "Performance Comparison of OCR Tools", Int. J. Ubi Comp, vol. 6, no. 3, pp. 19-30, 2015.
[http://dx.doi.org/10.5121/iju.2015.6303]
[52]
F. Asad, A. Ul-Hasan, F. Shafait, and A. Dengel, High per-formance ocr for camera-captured blurred documents with lstm networks2016 12th IAPR Workshop on Document Analysis Systems (DAS) Santorini, Greece, 2016, pp. 7-12.
[http://dx.doi.org/10.1109/DAS.2016.69]
[53]
C. Reul, M. Dittrich, and M. Gruner, Case study of a highly automated layout analysis and ocr of an incunabulum: ‘Der Heiligen Leben’ (1488)Proceedings of the 2nd Internation-al Conference on Digital Access to Textual Cultural Heritage, 2017, pp. 155-160.
[http://dx.doi.org/10.1145/3078081.3078098]
[54]
C. Reul, D. Christ, A. Hartelt, N. Balbach, M. Wehner, U. Springmann, C. Wick, C. Grundig, A. Büttner, and F. Puppe, "OCR4all-An Open-Source Tool Providing a (Semi-) Auto-matic OCR workflow for historical printings", Appl. Sci. (Basel), vol. 9, no. 22, p. 4853, 2019.
[http://dx.doi.org/10.3390/app9224853]
[55]
F. Borisyuk, A. Gordo, and V. Sivakumar, Rosetta: Large scale system for text detection and recognition in imagesProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining London United Kingdom, 2018, pp. 71-79.
[http://dx.doi.org/10.1145/3219819.3219861]
[56]
M. Namysl, and I. Konya, Efficient, lexicon-free ocr using deep learning2019 International Conference on Document Analysis and Recognition (ICDAR) Sydney, Australia, 2019, pp. 295-301.
[http://dx.doi.org/10.1109/ICDAR.2019.00055]
[57]
I. Muhammad, and Z. Yan, "Supervised machine learning approaches: A survey", ICTACT J. Soft Comput., vol. 05, no. 03, pp. 946-952, 2015.
[http://dx.doi.org/10.21917/ijsc.2015.0133]
[58]
C. Goutte, and E. Gaussier, A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evalua-tionEuropean Conference on Information Retrieval Berlin, Heidelberg, 2005, pp. 345-359.
[http://dx.doi.org/10.1007/978-3-540-31865-1_25]
[59]
P. Roy, S. Dutta, N. Dey, G. Dey, S. Chakraborty, and R. Ray, Adaptive thresholding: A comparative study2014 International Conference on Control, Instrumentation, Communication and Computational Technologies (ICCICCT) Kanyakumari, India, 2014, pp. 1182-1186.
[http://dx.doi.org/10.1109/ICCICCT.2014.6993140]
[60]
M. Basu, "Gaussian-based edge-detection methods-a survey", IEEE Trans. Syst. Man Cybern. C, vol. 32, no. 3, pp. 252-260, 2002.
[http://dx.doi.org/10.1109/TSMCC.2002.804448]
[61]
R.M. Haralick, S.R. Sternberg, and X. Zhuang, "Image analy-sis using mathematical morphology", IEEE Trans. Pattern Anal. Mach. Intell., vol. 9, no. 4, pp. 532-550, 1987.
[http://dx.doi.org/10.1109/TPAMI.1987.4767941] [PMID: 21869411]
[62]
N. Ketkar, Introduction to pytorch. Deep Learning with Py-thon. Apress: Berkeley, CA, 2017, pp. 195-208.
[http://dx.doi.org/10.1007/978-1-4842-2766-4_12]
[63]
V. Subramanian, Deep learning with PyTorch: A practical approach to building neural network models using PyTorch , 2018.
[64]
PyTorch:, An imperative style, high-performance deep learn-ing library.Adv. Neural Inf. Process. Syst., vol. 32. 2019, pp. 8026-8037.
[65]
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, S.J. Pan, and Q. Yang, "A survey on transfer learning", IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345-1359, 2010.
[http://dx.doi.org/10.1109/TKDE.2009.191]
[66]
L. Torrey, and J. Shavlik, Transfer learning. Handbook of research on machine learning applications and trends: algo-rithms, methods, and techniques., IGI global, 2010, pp. 242-264.
[http://dx.doi.org/10.4018/978-1-60566-766-9.ch011]
[67]
K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Las Vegas, NV, USA, 2016, pp. 770-778.
[http://dx.doi.org/10.1109/CVPR.2016.90]
[68]
V. Nair, and G.E. Hinton, Rectified linear units improve re-stricted boltzmann machines.CML’10: Proceedings of the 27th International Conference on International Conference on Machine Learning, 2010, pp. 807-814.
[69]
J.S. Bridle, "Probabilistic interpretation of feedforward classi-fication network outputs, with relationships to statistical pat-tern recognition", Neurocomputing Berlin, Heidelberg, pp. 227-236, 1990.
[http://dx.doi.org/10.1007/978-3-642-76153-9_28]
[70]
C. Nwankpa, W. Ijomah, A. Gachagan, and S. Marshall, “Activation functions: Comparison of trends in practice and research for deep learning”ArXiv181103378 Cs., 2020. Available from: http://arxiv.org/abs/1811.03378
[71]
S. Sharma, S. Sharma, and A. Athaiya, "Activation functions in neural networks", Int. J. Eng. Appl. Sci. Technol., vol. 4, no. 12, pp. 310-316, 2020.
[72]
Ö. Erkan, "B. Işık, A. Çiçek, and F. Kara, “Prediction of dam-age factor in end milling of glass fibre reinforced plastic com-posites using artificial neural network”", Appl. Compos. Mater., vol. 20, no. 4, pp. 517-536, 2013.
[http://dx.doi.org/10.1007/s10443-012-9286-3]
[73]
S. Vani, and T.V.M. Rao, An experimental approach towards the performance assessment of various optimizers on convo-lutional neural network2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI) Tirunelveli, India, 2019, pp. 331-336.
[http://dx.doi.org/10.1109/ICOEI.2019.8862686]
[74]
G.E. Nasr, E.A. Badr, and C. Joun, Cross entropy error func-tion in neural networks: Forecasting gasoline demandFLAIRS conference, 2002, pp. 381-384.
[75]
D.P. Kingma, and J. Ba, "Adam: A method for stochastic optimization", ArXiv14126980 Cs., 2021. Available from: http://arxiv.org/abs/1412.6980
[76]
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting", J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929-1958, 2014.