Improving Fine-grained Opinion Mining Approach with a Deep Constituency Tree-long Short Term Memory Network and Word Embedding

Dalila      Bouras; Mohamed      Amroune; Hakim      Bendjenna; Issam      Bendib

Abstract

Objective: One key task of fine-grained opinion mining on product review is to extract product aspects and their corresponding opinion expressed by users. Previous work has demonstrated that precise modeling of opinion targets within the surrounding context can improve performances. However, how to effectively and efficiently learn hidden word semantics and better represent targets and the context still needs to be further studied. Recent years have seen a revival of the Long Short- Term Memory (LSTM), with its effectiveness being demonstrated on a wide range of problems. However, LSTM based approaches are still limited to linear data processing since it processes the information sequentially. As a result, they may perform poorly on user-generated texts, such as product reviews, tweets, etc., whose syntactic structure is not precise.

Methods: In this research paper, we propose a constituency tree long short term memory neural network- based approach. We compare our model with state-of-the-art baselines on SemEval 2014 datasets.

Results: Experiment results show that our models obtain competitive performances compared to various supervised LSTM architectures.

Conclusion: Our work contributes to the improvement of state-of-the-art aspect-level opinion mining methods and offers a new approach to support the human decision-making process based on opinion mining results.

Keywords: Opinion mining, sentiment analysis, LSTM, deep learning, word embedding, constituency tree-LSTM, neural network.

Graphical Abstract

[1] 
S. Poria, "A review of affective computing: From unimodal analysis to multimodal fusion", Inf. Fusion, vol. 37, pp. 98-125, 2017.
[http://dx.doi.org/10.1016/j.inffus.2017.02.003] 
[2] 
E. Cambria, D. Das, S. Bandyopadhyay, A. Feraco, Eds., A practical guide to sentiment analysis., 1st ed Basel, Switzerland: Springer International Publishing , 2017.
[http://dx.doi.org/10.1007/978-3-319-55394-8] 
[3] 
Y. Ma, "Sentic LSTM: A hybrid network for targeted aspect-based sentiment analysis", Cognit. Comput., vol. 10, no. 4, pp. 639-650, 2018.
[http://dx.doi.org/10.1007/s12559-018-9549-x] 
[4] 
S. Poria, "Context-dependent sentiment analysis in user-generated videos", Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017pp. 873-883 
[http://dx.doi.org/10.18653/v1/P17-1081] 
[5] 
I. Chaturvedi, "Bayesian network based extreme learning machine for subjectivity detection", J. Franklin Inst., vol. 355, no. 4, pp. 1780-1797, 2018.
[http://dx.doi.org/10.1016/j.jfranklin.2017.06.007] 
[6] 
S.R. Das,  and M.Y. Chen, "Yahoo! for Amazon: Sentiment extraction from small talk on the web", Manage. Sci., vol. 53, no. 9, pp. 1375-1388, 2007.
[http://dx.doi.org/10.1287/mnsc.1070.0704] 
[7] 
S. Morinaga, "Mining product reputations on the web", Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 2002 
[http://dx.doi.org/10.1145/775047.775098] 
[8] 
M. Hu,  and B. Liu, "Mining and summarizing customer reviews", Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004pp. 168-177 
[9] 
M. Hu,  and B. Liu, Mining opinion features in customer reviews., AAAI, 2004.
[10] 
A-M. Popescu, B. Nguyen,  and O. Etzioni, "OPINE: Extracting product features and opinions from reviews", Proceedings of HLT/EMNLP 2005 Interactive Demonstrations, 2005pp. 32-33 
[http://dx.doi.org/10.3115/1225733.1225750] 
[11] 
L. Zhuang, F. Jing,  and X-Y. Zhu, "Movie review mining and summarization", Proceedings of the 15th ACM international conference on Information and knowledge management, 2006pp. 43-50 
[12] 
L. Zhang, Extracting and ranking product features in opinion documentsProceedings of the 23rd international conference on computational linguistics: Posters, 2010. Association for Computational Linguistics, 2010.
[13] 
G. Qiu, "Opinion word expansion and target extraction through double propagation", Comput. Linguist., vol. 37, no. 1, pp. 9-27, 2011.
[http://dx.doi.org/10.1162/coli_a_00034] 
[14] 
F. Li, "Structure-aware review mining and summarization", Proceedings of the 23rd international conference on computational linguistics, 2010pp. 653-661 
[15] 
D. Bouras, M. Amroune,  and H. Bendjenna, A review of recent aspect extraction techniques for opinion mining systemsin 2018 2nd International Conference on Natural Language and Speech	Processing (ICNLSP), 2018.
[16] 
M. Chernyshevich, "Ihs r&d belarus: Cross-domain extraction of product features using crf", Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), 2014pp. 309-313 
[http://dx.doi.org/10.3115/v1/S14-2051] 
[17] 
N. Jakob,  and I. Gurevych, "Extracting opinion targets in a single-and cross-domain setting with conditional random fields", Proceedings of the 2010 conference on empirical methods in natural language processing, 2010pp. 1035-1045 
[18] 
L. Shu, H. Xu,  and B. Liu, "Lifelong learning crf for supervised aspect extraction", arXiv [cs.CL], 2017.
[19] 
R. He, W.S. Lee, H.T. Ng,  and D. Dahlmeier, "An unsupervised neural attention model for aspect extraction", in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. (Volume 1: Long Papers), 2017
[http://dx.doi.org/10.18653/v1/P17-1036] 
[20] 
Q. Mei, "Topic sentiment mixture: Modeling facets and opinions in weblogs", in Proceedings of the 16th international conference on World Wide Web - WWW ’07, 2007 
[http://dx.doi.org/10.1145/1242572.1242596] 
[21] 
Y. Yin, "Unsupervised word and dependency path embeddings for aspect term extraction",  Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). arXiv [cs.CL], 2016 
[22] 
M. Taboada, "Lexicon-based methods for sentiment analysis", Comput. Linguist., vol. 37, no. 2, pp. 267-307, 2011.
[http://dx.doi.org/10.1162/COLI_a_00049] 
[23] 
X. Ding, B. Liu,  and P.S. Yu, A holistic lexicon-based approach to opinion miningin proceedings of the 2008 international conference on web search and data mining, 2008, pp. 231-240.
[http://dx.doi.org/10.1145/1341531.1341561] 
[24] 
V. Perez-Rosas, C. Banea,  and R. Mihalcea, Learning Sentiment Lexicons in Spanish in LREC., vol. 12, p. 73, 2012.
[25] 
A. Agarwal, "Sentiment analysis of twitter data", Proceedings of the Workshop on Language in Social Media (LSM 2011), 2011pp. 30-38 
[26] 
A. Sharma,  and S. Dey, "A comparative study of feature selection and machine learning techniques for sentiment analysis", in Proceedings of the 2012 ACM Research in Applied Computation Symposium on - RACS ’12, 2012 
[http://dx.doi.org/10.1145/2401603.2401605] 
[27] 
T. Wilson, J. Wiebe,  and P. Hoffmann, "Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis", Comput. Linguist., vol. 35, no. 3, pp. 399-433, 2009.
[http://dx.doi.org/10.1162/coli.08-012-R1-06-90] 
[28] 
D. Bahdanau, K. Cho,  and Y. Bengio, "Neural machine translation by jointly learning to align and translate", arXiv preprint, 2014.
[29] 
I. Sutskever, O. Vinyals,  and Q.V. Le, "Sequence to sequence learning with neural networks", Adv. Neural Inf. Process. Syst., vol. 2, pp. 3104-3112, 2014.
[30] 
A. Graves, N. Jaitly,  and A-R. Mohamed, "Hybrid speech recognition with deep bidirectional LSTM", in 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, 2013 
[http://dx.doi.org/10.1109/ASRU.2013.6707742] 
[31] 
O. Vinyals, "Show and tell: A neural image caption generator", Proceedings of the IEEE conference on computer vision and pattern recognition, 2015pp. 3156-3164 
[http://dx.doi.org/10.1109/CVPR.2015.7298935] 
[32] 
W. Zaremba,  and I. Sutskever, "Learning to execute", arXiv preprint, 2014.
[33] 
R. Collobert, "J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “Natural language processing (almost) from scratch", J. Mach. Learn. Res., vol. 12, pp. 2493-2537, 2011.
[34] 
S. Hochreiter,  and J. Schmidhuber, "Long short-term memory", Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
[http://dx.doi.org/10.1162/neco.1997.9.8.1735] [PMID: 9377276] 
[35] 
N. Chomsky, Syntactic Structures, Mouton and Co., The Hague.(1965) Aspects of the Theory of Syntax, The MIT Press, 1965.
[36] 
K.S. Tai, R. Socher,  and C.D. Manning, "Improved semantic representations from tree-structured long short-term memory networks", arXiv preprint, 2015.
[37] 
A. Blumer, "Occam’s razor", Inf. Process. Lett., vol. 24, no. 6, pp. 377-380, 1987.
[http://dx.doi.org/10.1016/0020-0190(87)90114-1] 
[38] 
H. Xu, "Double embeddings and cnn-based sequence labeling for aspect extraction", arXiv preprint, 2018.
[39] 
D. Tang, "Effective LSTMs for target-dependent sentiment classification", arXiv [cs.CL], 2015.
[40] 
Y. Wang, M. Huang,  and L. Zhao, "Attention-based LSTM for aspect-level sentiment classification", in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016 
[http://dx.doi.org/10.18653/v1/D16-1058] 
[41] 
P. Chen, "Recurrent attention network on memory for aspect sentiment analysis", in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017 
[http://dx.doi.org/10.18653/v1/D17-1047] 
[42] 
D. Ma, "Interactive attention networks for aspect-level sentiment classification", arXiv preprint, 2017.
[43] 
D. Hazarika, "Modeling inter-aspect dependencies for aspect-based sentiment analysis", Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 2, 2018pp. 266-270 
[http://dx.doi.org/10.18653/v1/N18-2043] 
[44] 
A. Graves, "Generating sequences with recurrent neural networks", arXiv preprint, 2013.
[45] 
D. Klein,  and C.D. Manning, "Corpus-based induction of syntactic structure: Models of dependency and constituency", in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017 
[http://dx.doi.org/10.3115/1218955.1219016] 
[46] 
S. Poria, "A rule-based approach to aspect extraction from product reviews", in Proceedings of the Second Workshop on Natural Language Processing for Social Media (SocialNLP), 2014 
[http://dx.doi.org/10.3115/v1/W14-5905] 
[47] 
S. Huang, "Fine-grained product features extraction and categorization in reviews opinion mining", In 2012 IEEE 12th International Conference on Data Mining Workshops, 2012 
[48] 
L. Jiang, "Target-dependent twitter sentiment classification", Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1, 2011pp. 151-160 
[49] 
S. Kiritchenko, "NRC-Canada-2014: Detecting aspects and sentiment in customer reviews", Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), 2014pp. 437-442 
[http://dx.doi.org/10.3115/v1/S14-2076] 
[50] 
N. Kaji,  and M. Kitsuregawa, "Building lexicon for sentiment analysis from massive collection of HTML documents", Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007pp. 1075-1083 
[51] 
D. Rao,  and D. Ravichandran, "Semi-supervised polarity lexicon induction", Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, 2009pp. 675-682 
[http://dx.doi.org/10.3115/1609067.1609142] 
[52] 
S.M. Mohammad, S. Kiritchenko,  and X. Zhu, "NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets", arXiv	preprint, 2013.
[53] 
M. Stojčić, A. Stjepanović,  and Đ. Stjepanović, "ANFIS model for the prediction of generated electricity of photovoltaic modules", DOAJ, vol. 2, no. 1, pp. 35-48, 2019.
[54] 
D. Pamucar,  and G. Ćirović, "Vehicle route selection with an adaptive neuro fuzzy inference system in uncertainty conditions", DOAJ, vol. 1, no. 1, pp. 13-37, 2018.
[http://dx.doi.org/10.31181/dmame180113p] 
[55] 
Y. Bodyanskiy, S. Popov,  and T. Rybalchenko, Multilayer neuro-fuzzy network for short term electric load forecasting.in Computer Science – Theory and Applications., Berlin Heidelberg Springer: Berlin, Heidelberg, 2008, pp. 339-348.
[http://dx.doi.org/10.1007/978-3-540-79709-8_34] 
[56] 
S. Sremac, "ANFIS model for determining the economic order quantity", DOAJ, vol. 1, no. 2, pp. 81-92, 2018.
[http://dx.doi.org/10.31181/dmame1802079s] 
[57] 
J. de Jesús Rubio, "USNFIS: Uniform stable neuro fuzzy inference system", Neurocomputing, vol. 262, pp. 57-66, 2017.
[http://dx.doi.org/10.1016/j.neucom.2016.08.150] 
[58] 
B.V. Krishna, A.K. Pandey,  and A.S. Kumar, Feature based opinion mining and sentiment analysis using fuzzy logic.in Cognitive Science and Artificial Intelligence., Singapore Springer Singapore, 2018, pp. 79-89.
[59] 
M. Afzaal, "Fuzzy aspect based opinion classification system for mining tourist reviews", Adv. Fuzzy Syst., vol. 2016, pp. 1-14, 2016.
[http://dx.doi.org/10.1155/2016/6965725] 
[60] 
Y. Kim, "Convolutional neural networks for sentence classification", arXiv preprint, 2014.
[61] 
R. Socher, "Recursive deep models for semantic compositionality over a sentiment treebank", Proceedings of the 2013 conference on empirical methods in natural language processing, 2013pp. 1631-1642 
[62] 
D. Bouras, "“Techniques and trends for fine-grained opinion mining and sentiment analysis: Recent survey”, Recent Adv. Comput", Sci. a Commun., vol. 13, no. 2, pp. 234-246, 2020.
[http://dx.doi.org/10.2174/2213275912666181227144256] 
[63] 
S. Poria, E. Cambria,  and A. Gelbukh, "Aspect extraction for opinion mining with a deep convolutional neural network", Knowl. Base. Syst., vol. 108, pp. 42-49, 2016.
[http://dx.doi.org/10.1016/j.knosys.2016.06.009] 
[64] 
L. Dong, "Adaptive recursive neural network for target-dependent twitter sentiment classification", Proceedings of the 52nd annual meeting of the association for computational linguistics, 2014pp. 49-54 
[http://dx.doi.org/10.3115/v1/P14-2009] 
[65] 
P. Chen, "Clause sentiment identification based on convolutional neural network with context embedding", in 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), 2016 
[66] 
W. Xue,  and T. Li, "Aspect based sentiment analysis with gated convolutional networks", arXiv preprint, 2018.
[67] 
M. Jabreel, F. Hassan,  and A. Moreno, Target-dependent sentiment analysis of tweets using bidirectional gated recurrent neural networks.in Advances in Hybridization of Intelligent Methods., Cham: Springer International Publishing, 2018, pp. 39-55.
[http://dx.doi.org/10.1007/978-3-319-66790-4_3] 
[68] 
S. Ruder, P. Ghaffari,  and J.G. Breslin, "A hierarchical model of reviews for aspect-based sentiment analysis", arXiv preprint, 2016.
[69] 
B. Xing, "Earlier attention? Aspect-aware LSTM for aspect Sentiment Analysis", arXiv preprint, 2019.
[70] 
Y. Tay, L.A. Tuan,  and S.C. Hui, "Learning to attend via word-aspect associative fusion for aspect-based sentiment analysis", arXiv [cs.CL], 2017.
[71] 
J. Wang, "Aspect sentiment classification with both Word-level and clause-level attention networks", IJCAI, 2018.
[72] 
Y. Bengio, P. Simard,  and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult", IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157-166, 1994.
[http://dx.doi.org/10.1109/72.279181] [PMID: 18267787] 
[73] 
R. Pascanu, T. Mikolov,  and Y. Bengio, "On the difficulty of training recurrent neural networks", arXiv [cs.LG], 2012.
[74] 
D. Chen,  and C. Manning, "A fast and accurate dependency parser using neural networks", Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014pp. 740-750 
[http://dx.doi.org/10.3115/v1/D14-1082] 
[75] 
P. Maria, "Semeval-2014 task 4: Aspect based sentiment analysis", Proceedings of the 10th international workshop on semantic evaluation (SemEval-2014), 2014pp. 27-35 
[76] 
J. Duchi, E. Hazan,  and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization", J. Mach. Learn. Res., vol. 12, pp. 2121-2159, 2011.
[77] 
N. Srivastava, "Dropout: A simple way to prevent neural networks from overfitting", J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929-1958, 2014.
[78] 
S. Semeniuta, A. Severyn,  and E. Barth, "Recurrent dropout without memory loss", arXiv preprint, 2016.
[79] 
S. Sarawagi,  and W.W. Cohen, "Semi-markov conditional random fields for information extraction", Adv. Neural Inf. Process. Syst., pp. 1185-1192, 2005.
[80] 
N.V. Cuong, "Conditional random field with high-order dependencies for sequence labeling and segmentation", J. Mach. Learn. Res., vol. 15, no. 1, pp. 981-1009, 2014.
[81] 
Z. Toh,  and W. Wang, "Dlirec: Aspect term extraction and term polarity classification system", Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), 2014pp. 235-240 
[http://dx.doi.org/10.3115/v1/S14-2038] 
[82] 
M. Pontiki, "Semeval-2016 task 5: Aspect based sentiment analysis", Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), 2016pp. 19-30 
[http://dx.doi.org/10.18653/v1/S16-1002] 
[83] 
P. Liu, S. Joty,  and H. Meng, "Fine-grained opinion mining with recurrent neural networks and word embeddings", Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015pp. 1433-1443 
[http://dx.doi.org/10.18653/v1/D15-1168] 
[84] 
X. Li,  and W. Lam, "Deep multi-task learning for aspect term extraction with memory interaction", Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
[http://dx.doi.org/10.18653/v1/D17-1310] 
[85] 
W. Wang, "Recursive neural conditional random fields for aspect-based sentiment analysis", Proceedings of EMNLP, 2016pp. 616-626 
[http://dx.doi.org/10.18653/v1/D16-1059] 
[86] 
N. Reimers,  and I. Gurevych, "Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging", arXiv preprint, 2017.
[87] 
D. Tang, "Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging", arXiv preprint, 2015.

Cite As

Recent Advances in Computer Science and Communications

Improving Fine-grained Opinion Mining Approach with a Deep Constituency Tree-long Short Term Memory Network and Word Embedding

Abstract

Graphical Abstract