Maria Ogneva is the Director of Social Media at Biz360, a social media monitoring, measurement and engagement platform. You can follow her on Twitter at @themaria or @biz360, or find her musings on the company blog and her personal blog.
Automated sentiment analysis has recently been the focus of an intense debate in the blogosphere. How accurate is it? What is the methodology? In what context is it useful for a business or a brand?
Sentiment analysis can be very useful for business if employed correctly. In this article, I will attempt to demystify the process, provide context, and offer some concrete examples of how businesses can utilize it.
What is Automated Sentiment Analysis?
Wikipedia (
) defines sentiment analysis as the process that “aims to determine the attitude of a speaker or a writer with respect to some topic.” Automated sentiment analysis is the process of training a computer to identify sentiment within content through Natural Language Processing (NLP). Various sentiment measurement platforms employ different techniques and statistical methodologies to evaluate sentiment across the web. Some rely 100% on automated sentiment, some employ humans to analyze sentiment, and some use a hybrid system.
Each automated platform has to first be trained to identify sentiment correctly, and methods for doing so vary widely. At Biz360, we use machine learning to train our system, based on annotations created via Amazon’s Mechanical Turk, which basically means that every item in the training data set is annotated by humans first.
Automated sentiment analysis will never be as accurate as human analysis, because it doesn’t account for the subtleties of sarcasm or body language. However, according to our experience with Mechanical Turk, humans only agree 79% of the time. That means even when the raw accuracy of automated sentiment analysis is well below perfect, statistically, it can be thought of as more accurate when compared to human analysis. In other words, automated analysis can be almost as good as human analysis (or, “as good as it gets”).
When Automated Sentiment is Meaningful
If you have 1 to 10 articles, the most effective way to measure sentiment is to simply read them. But what happens if you have 50,000? This is where automated sentiment can provide some directional insight and set the tone for further analysis.
Sentiment is a useful metric when taken in concert with others, but you would be ill-advised to base a strategy on sentiment alone. As with any metric, context is important, and so are the human insights that drive strategy.
For example, we built a project at Biz360 called IdolStats, where we predict the weekly results of American Idol voting based on sentiment towards each contestant as well as the volume of social media mentions. Because we look at both volume and sentiment (as well as trend comparisons to previous weeks), we are able to make a more accurate judgment than we would with sentiment alone.
What is also very telling about the social media health of a brand is how it’s public sentiment compares to that of its competitors. If your sentiment is 20% negative, is that bad? The answer is, it depends. However, if you see your competitors with a roughly 50% positive and 10% negative sentiment, while yours is 20% negative, that probably merits more discovery to understand the drivers of these opinions.
You should aim to uncover important phrases like “bad quality,” “crashes too often,” and “won’t start.” This can and should affect your product design, customer service, marketing messaging and social media outreach. Can you discover if there are particular influencers driving the negative conversation and do anything to reach out to them?
It may also be worthwhile to capture sentiment on a trend line so that you can identify sudden positive or negative spikes. At the end of the day, sentiment is a great jumping off point that can help structure further analysis. As Nathan Gilliatt says, “Insight isn’t automated; what you do next depends on what you find.”
What to Look For
If you decide that automated sentiment is a sensible addition to your social media measurement and monitoring toolbox, here is a quick rundown of things to think about when deciding on a platform and figuring out how sentiment fits into your workflow.
- Demystifying accuracy: As I mentioned above, machines will never be able to measure sentiment as well as humans, and even humans don’t agree 100% of the time. The number of sentiment types is also part of the equation. Some platforms offer three sentiments, some offer four, and some offer more than five. The more you increase the number of sentiment types, the less accurate (but more information rich) your results become.
- Isolating content types: A lot of social media mentions are neutral in nature (the Biz360 engineering team estimates it to be around 60%), and some social media sources tend to skew higher on the neutral scale. For example, a higher percentage of updates are neutral on Twitter () than any other medium (consider these common examples: “I just had a cup of coffee,” or “craving tacos for lunch – who’s in?” or “Apple launches the iPad tablet”). Depending on the source you are looking at, your sentiment results will differ — this should be expected. Make sure your sentiment platform allows you to isolate results by content type.
- Sentiment override: Because automated sentiment is not going to be 100% accurate, you, the user, need to have some kind of override control. When picking a tool, ensure that it allows you to override sentiment, and toss irrelevant results.
- Entity level vs. article level sentiment: Until recently, the industry default has been able to measure sentiment at the level of the article. Over time, some platforms have developed ways to measure sentiment on the level of the entity (entity level analysis can measure the sentiment of an entity or multiple entities within an article even if the overall sentiment of the article is different).In my opinion, entity-level sentiment is far more useful for a brand. For example, if I am a community manager at Gowalla (
), and I am reading an article about geo-location services, it may very well contain three different sentiments: Gowalla, Foursquare () and the tone of the overall article. If I’m Gowalla, I would probably want to know all three, but care most about sentiment towards Gowalla.
What’s Next for Sentiment Measurement?
One of the most important areas for sentiment analysis, and social media monitoring in general, is bridging the gap between insight and action. It’s one thing to retrieve a sentiment pie chart. It’s another to masterfully place it within the context of your brand’s social media performance.
The key to successful engagement is sentiment prioritization:
1. Influence: Because social media mentions are plentiful, priorotization tools must continue evolving. Of the 10,000 tweets and blog posts about your brand, how do you pick the top 50 to focus on?
If you need to neutralize the mentions that hurt your brand the most, you should drill down into negative mentions, identify the content coming from the most influential people in your industry, understand how far each tweet traveled, and how many people were impacted by this content. If someone blogs and tweets the same negative mention about you, how do you account for that? How do you quantify the multiplied effect of cross-platform communication originating from the same person? Including influencer analytics alongside sentiment measurement is becoming a standard of the social media monitoring industry.
2. Reputation: Taking the influencer concept a step further, each notable user should have a social media reputation profile. If someone’s negative sentiment indexes higher than average (i.e. that person hates everything equally), then that person’s negative sentiment should be somewhat discounted — in statistics, we toss outliers like these out of the consideration set. Moreover, your reputation and influence on one channel should carry over into other channels.
3. Intensity: As far as sentiment algorithms are concerned, part of a successful prioritization process is going to be identifying the intensity of each mention. “I really hate product X and will never buy it” is quite different from “Product X is running a little slow today.” Ability to cross-reference intensity, influence, trajectory, velocity and sentiment of each social media mention will drive us towards a reliable priority system.
Social CRM Ties it All Together
The true usability of a sentiment system is revealed when you are able to drill into each “piece of the sentiment pie” or a point on the sentiment trend graph. Prioritize results by relative importance, reach out when necessary, and have each interaction become part of that conversation record. This is the definition of Social CRM (Customer Relationship Management), and its goal is to enrich each conversation and deepen engagement with the same person across multiple channels. The Holy Grail of Social CRM is the ability to close the analytical loop between publicly expressed sentiment, engagement action, subsequent purchase intent, and ultimately, product purchase.
To illustrate, imagine the following scenario:
- You wake up to a large spike in social media mentions.
- You drill down to understand sentiment and discover that it’s mostly negative.
- Prioritize the negative mentions and portion off 50 most important pieces of content to cover.
- Discover that a flurry of negative tweets originated from an influential social media personality and spread like wildfire.
- You read this person’s blog, discover a performance issue that this person had with your product.
- Leave an honest and humble comment on the blog, committing in public to fixing the issue.
- Reach out to the blogger privately, gain an understanding of what would fix the situation, and then do so.
- All of the above is automatically logged into the system under the blogger’s name, which allows you to track the progress of that relationship over time.
- Because you now have a history on this person, and your system cross-references his/her social media profiles across several platforms, you are able to track this person’s affinity and sentiment towards your brand over time, monitor their purchase intent, and note their influence on others’ purchase decisions.
How are you measuring sentiment in your business? What do you find useful? How does sentiment analysis help drive your strategy and further research?