publications

2022

1. LNAI

   Impact Evaluation of Multimodal Information on Sentiment Analysis

   Luis N. Zúñiga-Morales, Jorge Ángel González-Ordiano, J.Emilio Quiroz-Ibarra, and Steven J. Simske

   In Advances in Computational Intelligence 2022

   Abs Bib

   Text-based sentiment analysis is a popular application of artificial intelligence that has benefited in the past decade from the growth of digital social networks and its almost unlimited amount of data. Currently, social network users can combine different types of information in a single post, such as images, videos, GIFs, and live streams. As a result, they can express more complex thoughts and opinions. The goal of our study is to analyze the impact that incorporating different types of multimodal information may have on social media sentiment analysis. In particular, we give special attention to the interaction between text messages and images with and without text captions. To study this interaction we first create a new dataset in Spanish that contains tweets with images. Afterwards, we manually label several sentiments for each tweet, as follows: the overall tweet sentiment, the sentiment of the text, the sentiment of the individual images, the sentiment of the caption, if present, and—in cases where a single tweet has several images—the aggregate sentiment of all images present in the tweet. We conclude that incorporating visual information into text-based sentiment analysis raises the performance of the classifiers that determine the overall sentiment of a tweet by an average of 25.5%.

   @inproceedings{Zuniga2022,
     author = {Z{\'u}{\~{n}}iga-Morales, Luis N. and Gonz{\'a}lez-Ordiano, Jorge {\'A}ngel and Quiroz-Ibarra, J.Emilio and Simske, Steven J.},
     editor = {Pichardo Lagunas, Obdulia and Mart{\'i}nez-Miranda, Juan and Mart{\'i}nez Seis, Bella},
     title = {Impact Evaluation of Multimodal Information on Sentiment Analysis},
     booktitle = {Advances in Computational Intelligence},
     year = {2022},
     publisher = {Springer Nature Switzerland},
     address = {Cham},
     pages = {18--29},
     isbn = {978-3-031-19496-2}
   }