Slide deck from a talk given at the BreakingNews.ie Measurement Conference on 10th September 2014.
It's an updated version of a talk I gave (only once) in 2011. My understanding (and the technology) have developed since then.
2. TWITTER VOLUME
15
10
5
0
1 Jun 2014
1 Jul 2014
Thousands
“facebook” AND (“newsfeed” OR “news feed”) June 1 – July 31 2014
Data: Netbase
3. WHAT HAPPENED HERE?
aND HOW DOES THAT MAKE YOU
FEEL?
Topic Analysis
Emotions
Data: Brandwatch, Netbase
4. DELOITTE: BALANCE OF SENTIMENT AFFECTS SALES
The balance of sentiment in Tweets is a
more powerful driver of sales than
reach (or volume) alone, with positive
Tweets having generally a higher impact
than negative Tweets. Therefore, to
gain the most out of the online word-of-
mouth embodied by Tweets,
companies would be best served by
addressing the balance of sentiment
about their games through increasing the
number of positive Tweets.
6.10%
3.30%
1.60%
30% more positive Tweets
30% fewer negative Tweets
30% more non-Twitter advertising
Deloitte, “Tweets for Sales, Gaming” (2013)
5. “SENTIMENT” HAS BECOME A MARKETING OBJECTIVE
We want to increase the
positive buzz around Your
brand
6. “SENTIMENT” HAS BECOME A MARKETING OBJECTIVE
Real case study.
Name obscured to protect the innocent.
7. THIS IS HONEST ABE
Public sentiment is
everything.
With public
sentiment, nothing
can fail.
Without it, nothing
can succeed.
8. I’M GOING TO BE EVEN MORE HONEST
SENTIMENT
ANALYSIS
IS
SHITE!
DTEOLNL'T U HSO WLDH ABTA CYKO.U. .REALLY THINK!
9. THE PROMISE OF
SOCIAL INTELLIGENCE
AN ALMOST INFINITE
SOURCE OF QUAL QUANT
DATA!
FINALLY THEY WILL REVEAL
WHAT THEY REALLY THINK!
11. WEIGHTED SENTIMENT, MRS BROWN’S BOYS, JULY 2014
14.8%
12.2%
11.4%
7.3%
-4.1%
-5.2%
-6.3%
-8.5%
POSITIVE
NEGATIVE
4 DIFFERENT TOOLS GIVE 4 DIFFERENT MEASURES OF SAME DATA
17. FACEBOOK USED LEXICAL APPROACH
“Posts were determined to
be positive or negative if they
contained at least one
positive or negative word”
Visit: http://www.pnas.org/content/111/24/8788.full
LIWC: http://www.liwc.net/
15
10
5
0
Thousands
18. CLASSIFIERS SUPERVISED LEARNING…
TRAIN MODEL
TAGGED
TEST MODEL
DATA
TRAINING
DATA
TEST
DATA
POSITIVE
NEUTRAL
NEGATIVE
19. THIS PRESENTATION IS GOOD
visit: text-processing.com/demo/sentiment
CLASSIFIER SAYS “POSITIVE”
20. THIS PRESENTATION IS BAD
CLASSIFIER SAYS “NEGATIVE”
visit: text-processing.com/demo/sentiment
21. THIS PRESENTATION IS NOT GOOD
CLASSIFIER SAYS “NEGATIVE”
visit: text-processing.com/demo/sentiment
25. DOES IT UNDERSTAND WHAT IT’S READING?
2.
1.
Take the original text
Randomise the word order
3.
Re-test
recipe: http://stackoverflow.com/questions/17825945/generating-a-list-of-random-words-in-excel-but-no-duplicates
32. EXPERIMENT: IS GUINNESS GOOD FOR YOU?
Selected 50 positive and 50 negative tweets
as scored by classifier.
Passed these tweets to human markers.
Each tweet scored 3 times (5 point scale)
Average score compared to classifier.
34. RESULTS: IS GUINNESS GOOD FOR YOU?
NEG
NEUT
POS
CLASSIFIER
50
0
50
MANUAL
23
30
47
AGREEMENT
40%
0%
72%
(Agreement based on #tweets with matched judgments)
35. HUMANS DON’T ALWAYS AGREE…
Advert used to say
#Guinness is good for you
but I think it is not acceptable
to say that these days, but in
moderation I thrive on it at
70
It's called
a rotten apple
#twobeersonecup
#Guinness #angryorchard
#delicious
http://t.co/GUFxrYoF6i
37. BIG ENOUGH NUMBERS
If the sample is large enough, won’t these problems get ironed out?
38. SAMPLE BIAS (1936 US PRESIDENTIAL ELECTION)
SURVEY SIZE
ROOSEVELT
LITERARY DIGEST
2,400,000
43%
GALLUP
50,000
54%
ACTUAL
61%
See: Tim Harford, “Big Data: are we making a big mistake?” (FT Magazine, 28 March 2008)
I THINK YOU'll find
iT'S 48 times bigger
ALF LANDON
39. WIN 30 MINS OF FREE CONSULTANCY (VALUE £750)
I'm loving
#breakingnewsconf,
@mediaczar.
IT’S FAR TOO EASY TO GAME “POSITIVE SENTIMENT” METRICS
40. RECOMMENDATIONS
LIFE ISN’T A POPULARITY CONTEST
DON’T LET SENTIMENT BECOME A KPI
MAKE SENTIMENT A TOOL FOR MORE COMPLEX RESEARCH
PUT GREAT TECH TO MORE MEANINGFUL USE
41. THANK YOU!
PLEASE DON’T ASK ME ANY TRICKY QUESTIONS THAT WILL MAKE ME LOOK STUPID
I’M @MEDIACZAR ON TWITTER
FEEL FREE TO COME AND TALK TO ME AFTERWARDS
Editor's Notes
"respondents who returned their questionnaires represented only that subset of the population with a relatively intense interest in the subject at hand, and as such constitute in no sense a random sample... it seems clear that the minority of anti-Roosevelt voters felt more strongly about the election than did the pro-Roosevelt majority.”
50x bigger sample. Wrong answer.