July 1, 2008

Gnip launches and Plaxo’s pulse is racing

A new service called Gnip just launched that supplies a sorely needed piece of backend infrastructure for the burgeoning Social Web–making it quick and efficient for user-generated content created on a rapidly growing list of social sites (e.g. Digg or Flickr) to show up in tools like Plaxo Pulse that “aggregate” this data from across the web on behalf of users. Those familiar with Plaxo’s eagerness to lead by example by integrating new technologies that help open up the Social Web ecosystem should not be surprised to learn that we’re a launch partner for Gnip, and have already integrated their infrastructure into Pulse.
Here’s a bit of info on how Gnip works, why it’s good for Plaxo users, and why it’s good for the Social Web:
In a nutshell, Gnip acts as a middleman that notifies aggregators of social media, like Plaxo, when user-generated content sites, like Digg or Flickr, have new content they need to distribute. So, rather than us having to repeatedly ask our content site partners, “got anything new for any of our users? how about now? now?”, Gnip notifies us when there’s new stuff from any of our members, so we can immediately pick up that content and show it to the people who are supposed to see it in Pulse.
For Plaxo users, the benefit is simple: when you digg a story or bookmark a link with del.icio.us, etc. you should see that activity show up in Pulse a lot quicker–often within 60 seconds, whereas before integrating with Gnip, it might have taken an hour or more. Starting today, Digg and del.icio.us should be very quick to update, with Flickr and Twitter hopefully following shortly. And any publisher can easily send data thru Gnip using their API, so if Pulse pulls feeds from your site and you’d like that content to show up faster, we’ll then make it happen!
For the technically inclined, here’s what’s happening behind the scenes:
Most of the content in Pulse comes from public RSS feeds for each user who’s hooked up one more sites into their Pulse stream. We have background jobs that periodically poll each feed for each user for each service, and whenever we see anything new, we update that user’s Pulse stream accordingly. Since we don’t know in advance which users have new content at any given time, we have to keep polling each site for each user over and over again. If we want content to show up in Pulse more quickly, we have to poll the sites more frequently, and of course in any given minute, the vast majority of users haven’t shared anything new right then, so the process is rather inefficient.
While providing individual RSS feeds for user activity on web 2.0 sites is nothing new, when we launched Pulse about a year ago, we were the first site to consume and aggregate these feeds en masse. Since we have such a large user base, and since polling is rather inefficient–particularly if you want to notice any updates quickly–Plaxo puts a rather large load on these sites, and in several cases even the larger sites have had to tell us to slow down so they can keep up with the demand. Since then, several new aggregator services have followed, including FriendFeed, SocialThing, and others, and as they gain popularity, they’re further increasing the load on publisher sites.
Clearly this model will not continue to scale, and since the amount of social content being produced and consumed is rising every day, a better architecture is needed; one that efficiently routes updates from user generated content sites to social media aggregators, rather than the other way around. There are various technical ways to accomplish this today–posting updates directly to the aggregators, federated messaging protocols like xmpp (aka jabber), real-time public update streams, etc.–but few are widely deployed or easy to work with, which is why polling is still the dominant model used. Until now.
Gnip is stepping in to catalyze the shift from polling to pushing notifications by doing the hard work of consuming all the existing notification systems out there today (and polling itself, as needed) and pushing out the relevant updates to consumers like Plaxo and others. Pulse tells Gnip which users we want updates for (e.g. which Digg users have shared their public feed of dugg stories in Pulse), and whenever any of those users digg a new story, Gnip proactively notifies Plaxo, telling us which users have new activity, all within 60 seconds! Using Gnip, Plaxo no longer has to poll with high frequency to get quick updates–in fact, not only can we poll less frequently (which helps relieve the load on both Plaxo and the publishers we’re currently hammering), but we get updates much faster than we could before. It’s a classic example of a win-win created by reducing inefficiency in the system overall.
Emerging service layer for the Social WebA quick note about standards: in the fully realized social web, there should be no need for single points of full centralization, either for Identity Providers, Social Graph Providers, or Content Aggregators. In each case, open standards will allow a decentralized and competitive “service layer” to emerge. For identity, OpenID is such a standard, and its adoption is continuing to spread rapidly. For “who you know” data, OAuth provides a standard way to securely share private data between trusted services, and efforts are underway to also standardize APIs for making address book and social graph data portable. As mentioned above, in the case of sharing and aggregating social web activity data, things are a bit more nascent. Standards like xmpp may someday gain wider adoption, but until then an opportunity exists to help realize that vision sooner.
Just as companies like Clickpass have stepped in to make OpenID more user-friendly and widespread, so too is Gnip stepping in to make scalable content sharing a reality sooner rather than later. In both cases, Plaxo is supporting these efforts because they provide useful value today, and they demonstrate the potential that truly opening up the social web provides, which we believe will bring about the future we all want even more rapidly. Today marks yet another acceleration of that progress, and we all know it won’t be the last one!
–Joseph Smarr, Chief Platform Architect

Email Print

About the Author

View Plaxo Profile >>
Redgee Capili

General Manager, Plaxo.com

Related Articles