Twitter recently announced they would be making changes to the way in which companies could access data from the network.
From now on, companies will only be able to access the Twitter firehose— the stream of 500 million tweets per day that comes in from Twitter users, which companies can pull valuable insights from—by going directly through Twitter (with GNIP, the company that Twitter acquired in 2014).
Previously Datasift and another company, NTT, which dealt with tweets in Japanese, licensed Twitter data directly from the network. Those companies created a structure for the data and sold access to it.
More than a few people have cried foul at Twitter’s move and how suddenly it was apparently made (This guy might be overstating things a bit though). It does perhaps leave some companies that partnered with Datasift in the lurch, at least in the short term.
— Chris Barchak (@cbarchak) 12 Avril 2015
This story is coming on the heels of the announcement that Facebook is partnering solely with Datasift for its new Topic Data—the information about topics people are discussing on Facebook that will be available to marketers (and which replaces the previously available listening data).
I work with all of the companies involved here, and from here these moves make sense based on what Facebook and Twitter aim to do with their respective data sets.
What’s Twitter’s reasoning?
When Datasift worked with Twitter, they pulled data from that network and other sources, and served it to both end users and agencies, analytics companies and platforms that in turn worked with end users.
The nature of Twitter means that most of the data the network generates is public. Twitter and GNIP have sold to people who can inject the firehose (like Falcon Social) and drive product innovation and build value for clients by providing actionable insights.
Twitter, as well as end users, are getting more ambitious in terms of the uses of this data. Listening to what people are talking about in a city could, for example, help a company determine how much demand there will be for their product, and they could ramp up their production accordingly.
Twitter is cutting off Datasift, which both resold and repackaged data into a format that their clients could use more or less as is.
Twitter says, basically, that this is what needs to happen in order to help them and their customers meet those ambitions. As Twitter’s Zach Hofer-Shall put it, they want to “develop a deeper understanding of customer needs, get direct feedback for the product roadmap and work more closely with data customers.”
Facebook makes its own data moves
This comes alongside the news that Facebook will change their offering so that their data will be only accessible through Datasift, Twitter’s ex-partner.
It’s not, as it might seem, a case of retaliation. (Twitter didn’t stop working with Datasift because of the Facebook partnership.) It’s been part of their plan since they acquired GNIP.
Facebook’s new program is called topic data—it will be keyword based, anonymized demographic data and insights from all posts and status updates on the network, except those that are only viewable by the poster.
The news is a change for Facebook, which has generally allowed less access to data about what people were saying because so many posts on the network are private. Their public search API and monetization of that data has not always been their top priority.
Datasift will be the intermediary between Facebook and anyone that wants to access that data on the things people are mentioning on the network. Partners will work with Datasift to integrate the data via their Pylon infrastructure and API.
I’ve seen the tools for Facebook data that will be available via Datasift and the possible applications for it and they are providing a strong product to the marked and looking forward to see the application that the marked will bring
Facebook in giving data to datasift will lay down certain ground rules—not only will it be anonymized, queries will have to have at least 100 matches to return any results, certain queries will not be allowed. I see this as a smart strategy for these privacy first platforms and datasift is well-positioned to function in a framework like this.
I could see other networks that need to control information, like, say, Sina Weibo or WeChat going down a similar route.
Different data directions
The two approaches are a reflection of the use cases for the two networks. You could also call them two different directions for how people deal with data. As more and more information is generated by social networks, there’s naturally going to be a divergence in terms of data treatment.
Twitter wants companies to innovate with its public data, and thinks that the best way to do so is to serve it in its undigested form to companies that can create different infrastructures around it in order to better interpret it.
Facebook, on the other hand, needs to keep a somewhat tighter rein on its data; posts are still, by default, private. Their choosing a single provider to work with on Topic Data will presumably allow a greater amount of control over the data and how people use it.
A move towards consolidation
In social media in general, there’s been a lot of hand wringing about Twitter’s announcement, and a lot talk about the relative wisdom of counting on another company for a significant part of your business.
It’s not all wrong, but it tends to underestimate businesses like Datasift, and ignore what, for me, has been Twitter’s strategy for a while now.
Major social networks have to figure out how to balance plenty of interests—their users, themselves, third parties and others.
Partners like Datasift are smart enough to realize that there needs to be a good reason for a social network to work with them. In this case, Twitter decided that wasn’t happening, but for Facebook it was.
As the space matures, there’ll be some general consolidation in the market—instead of having tons of different companies doing similar things, there will be a smaller number that bring value to networks and end users in each other these type of data sets.
In this consolidation, for vendors, advantage comes down to how smart you are about where the data market is moving and having the ability to build infrastructure and innovative products—those who can’t will find it increasingly harder to compete in the next stages.