Sentiment analysis has become a new trend in social media monitoring, brand monitoring, product analytics, market research. Like most areas that use artificial intelligence, sentiment analysis (also known as Opinion Mining) is an interdisciplinary field spanning computer science, psychology, social sciences, linguistics, and cognitive science. The goal of sentiment analysis is to identify and extract attitudes, emotions, opinions within the text. Or in other words, sentiment analysis is a mining of subjective impressions, but not facts, from users’ tweets, reviews, posts, etc. Why do we need to do an opinion mining? The reasons are obvious:
- 80% of the world’s data is unstructured. People increasingly tend to share their opinions on forums, blogs, social networks, chats. It is very time-consuming to analyze and sort all these texts manually. So, the cost-effective way is to automate this process by sentiment analysis systems.
- The second reason is that it is possible to provide real-time sentiment analysis. This property allows to react quickly on specific situations in real-time, like an angry customer that is about to churn.
There are three types of sentiment analysis algorithms: rule-based, automatic and hybrid, that combines the first two approaches. In rule-based system, there are two lists of polarized words (keywords) or short phrases, like “bad, horrible, ugly, etc.” and “good, beautiful, amazing, high quality, etc.” Then, classic NLP techniques are used to find and count the numbers of positive and negative words in the text. The comparison of these two numbers give us a positive, or negative, or neutral sentiment.
Automatic approaches use machine learning techniques. In this approach, it is important to prepare correct database (or use existing one) with, e.g., users’ reviews that are already labeled as positive, negative, and neutral. Then the classification model should be trained using this labeled dataset. The resulting model is able to predict tags (positive, negative, neutral) for unseen text.
In general, the precision of sentiment analysis algorithms is about 60-80%. What makes sentiment analysis so inaccurate? Well, there are a lot of reasons, here is a list of some challenges in sentiment analysis:
- Complex sentence structure. People often express opinions in complex ways. For example, the sentences “I don’t like the movie that you love” has both positive and negative sentiment towards the movie. Most of the systems are unable to recognize the sentiment that was expressed in the sentence.
- Multiple topics. Review could contain several topics, e.g. user can mention few products in one review or few aspects of the product. For example, a review may contain sentence “iPhone X is too expensive” with negative opinion about price and another one “The camera of iPhone X performs really well” with positive opinion about cameras. The model that is able to manage reviews with several topics in one document is in high demand in sentiment analysis.
- Implicit sentiment and sentiment ambiguity. A sentence with a positive or negative phrase does not necessarily express any sentiment and vice versa, a sentence without opinion marker may have some polarity. For example, the sentence “Can you recommend a good movie I could watch?” hasn’t any sentiment or opinion but has positive sentiment word “good”. Another sentence “I went to eat, but the restaurant was closed” expresses negative polarity without sentiment word markers.
- Sarcasm, irony and metaphors. Sarcasm and irony are often used in the reviews. Readers can easily understand the product quality from such reviews. Unfortunately, that cannot be said about machines. It is often impossible to identify a sentence containing sarcasm or irony without knowing the context.
Applications of the Sentiment Analysis
The use of sentiment analysis is wide. There are its applications in e-commerce, marketing, sociology and psychology research, politics. Below, we will mention a few.
- Customer feedback. The analysis of customers’ reviews can prevent a crisis in a company. The first negative experience with a company service or product is a reason for a big part of customers to stop relationship with a company. Constant monitoring of reviews and outer social media is the key to understanding, e.g., whether users like the new version of the company app or not. Sentiment analysis provides quick reaction and nipping of possible problem.
- Market research and analysis. Tracking customer sentiment helps to understand overall customer satisfaction, his loyalty, and how likely are customers to engage again in a day/week/month. The knowledge in these metrics could give insights for the marketing strategy.
- Sociology. Sociology is interested in the idea propagation through groups. Opinions and reactions to ideas in a society are relevant to adoption of new ideas. The insights to this process can be taken from sentiment analysis of reactions on blogs, tweets, etc. For example, sentiment analysis can give understanding behaviors across cultures and inter-group identity meanings through analysis of the news media during the Arab Spring. Another application of the sentiment analysis is an exploration of the influence spreading of communities and individuals in the blogosphere.