Now that even Bridgewater has joined the Twitter craze and is using user-generated content for real-time economic modelling, and who knows what else, the scramble to determine who has the most market-moving, and actionable, Twitter stream is on. Because with HFT algos having camped out at all the usual newswire sources: Bloomberg, Reuters, Dow Jones, etc. the scramble to find a "content edge" for market moving information has never been higher. However, that opens up a far trickier question: whose information on the fastest growing social network, one which many say may surpass Bloomberg in terms of news propagation and functionality, is credible and by implication: whose is not? Indeed, that is the $64K question. Luckily, there is an algo for that.

In a note by Castillo et al from Yahoo Research in Spain and Chile, the authors focus on automatic methods for assessing the credibility of a given set of tweets. Specifically, they analyze microblog postings related to “trending” topics, and classify them as credible or not credible, based on features extracted from them. Our results shows that there are measurable differences in the way messages propagate, that can be used to classify them automatically as credible or not credible, with precision and recall in the range of 70% to 80%.

Needless to say, the topic of social media credibility is a critical one, in part due to the voluntary anonymity of the majority of sources , the frequent error rate of named sources, the painfully subjective attributes involved in determining good and bad information, and one where discerning the credible sources has become a very lucrative business. Further from the authors:

In a recent user study, it was found that providing information to users about the estimated credibility of online content was very useful and valuable to them. In absence of this external information, perceptions of credibility online are strongly influenced by style-related attributes, including visual design, which are not directly related to the content itself. Users also may change their perception of credibility of a blog posting depending on the (supposed) gender of the author. In this light the results of the experiment described are not surprising. In the experiment, the headline of a news item was presented to users in different ways, i.e. as posted in a traditional media website, as a blog, and as a post on Twitter. Users found the same news headline significantly less credible when presented on Twitter.

 

This distrust may not be completely ungrounded. Major search engines are starting to prominently display search results from the “real-time web” (blog and microblog postings), particularly for trending topics. This has attracted spammers that use Twitter to attract visitors to (typically) web pages offering products or services. It has also increased the potential impact of orchestrated attacks that spread lies and misinformation. Twitter is currently being used as a tool for political propaganda. Misinformation can also be spread unwillingly. For instance, on November 2010 the Twitter account of the presidential adviser for disaster management of Indonesia was hacked. The hacker then used the account to post a false tsunami warning. On January 2011 rumors of a shooting in the Oxford Circus in London, spread rapidly through Twitter. A large collection of screenshots of those tweets can be found online.

 

Recently, the Truthy service from researchers at Indiana University, has started to collect, analyze and visualize the spread of tweets belonging to “trending topics”. Features collected from the tweets are used to compute a truthiness score for a set of tweets. Those sets with low truthiness score are more likely to be part of a campaign to deceive users. Instead, in our work we do not focus specifically on detecting willful deception, but look for factors that can be used to automatically approximate users’ perceptions of credibility.

The study's conclusion: "we have shown that for messages about time-sensitive topics, we can separate automatically newsworthy topics from other types of conversations. Among several other features, newsworthy topics tend to include URLs and to have deep propagation trees. We also show that we can assess automatically the level of social media credibility of newsworthy topics. Among several other features, credible news are propagated through authors that have previously written a large number of messages, originate at a single or a few users in the network, and have many re-posts."

All of the above is largely known. What isn't, however, is the mostly generic matrix used by various electronic and algorithmic sources to determine who is real and who isn't, and thus who is market moving and who, well, ins't. Once again, courtesy of Castillo, one can determine how the filtering algo operates, (and thus reverse engineer it). So without further ado, here is the set of features used by Twitter truth-seekers everywhere.

Those are the variables. And as for the decision tree that leads an algo to conclude if a source's data can be trusted and thus acted upon, here it is in its entirety. First, verbally:

As the decision tree shows, the top features for this task were the following:

• Topic-based features: the fraction of tweets having an URL is the root of the tree. Sentiment-based features like fraction of negative sentiment or fraction of tweets with an exclamation mark correspond to the following relevant features, very close to the root. In particular we can observe two very simple classification rules, tweets which do not include URLs tend to be related to non-credible news. On the other hand, tweets which include negative sentiment terms are related to credible news. Something similar occurs when people use positive sentiment terms: a low fraction of tweets with positive sentiment terms tend to be related to noncredible news.
• User-based features: these collection of features is very relevant for this task. Notice that low credible news are mostly propagated by users who have not written many messages in the past. The number of friends is also a feature that is very close to the root.
• Propagation-based features: the maximum level size of the RT tree is also a relevant feature for this task. Tweets with many re-tweets are related to credible news.

These results show that textual information is very relevant for this task. Opinions or subjective expressions describe people’s sentiments or perceptions about a given topic or event. Opinions are also important for this task that allow to detect the community perception about the credibility of an event. On the other hand, user-based features are indicators of the reputation of the users. Messages propagated trough credible users (active users with a significant number of connections) are seen as highly credible. Thus, those users tend to propagate credible news suggesting that the Twitter community works like a social filter.

And visually:

Get to the very bottom of the tree without spooking too many algos, and you too can have a Carl Icahn-like impact on the stock of your choosing.

Source: "Information Credibility on Twitter"