clouin.com

Techmeme’s Human Eye for the Straight Algo

Gabe Rivera of Techmeme fame has sparked a lively debate as he announced he had hired an editor to improve interestingness, reactivity and relevance of his news aggregation website (see VentureBeat, TechCrunch, and ReadWriteWeb). Rivera’s plight echoes Netflix’s challenges at boosting its movie recommendation engine, as algorithmic improvements gradually near their natural asymptote.

Techmeme is good at aggregating news overtime but not at breaking them, pointing to several common datamining and recommendation challenges:

- “Cold start” – interestingly digg’s social voting approach hasn’t been able to overcome that challenge either – in both cases, the services cannot anticipate news’ propagation and velocity;

- Context – Techmeme sometimes mixes up headlines and for instance ended up featuring news about Anna Nicole Smith’s hospitalization after she’s already been declared dead;

- Outliers or the “Napoleon Dynamite” problem, as the New York Times dubs it - identifying newsworthy pieces from uncommon sources before they make it into the mainstream is also an issue.

Interestingness and relevance are Techmeme’s other key reasons to bring in a human eye. Techmeme bets that an expert hand can be a better judge than crowdsourced implicit feedback based on clickstream or explicit feedback such as social voting. This approach seems to contradict much of the crowdsourcing mantra, although Techmeme’s case is more about rebalancing than shunning crowdsourcing.

For online retailers, Techmeme’s move to “curated news aggregation” highlights opportunities to blend human input, datamining, and recommendation:

- to add context to a product recommendation – based on usage, audience background, as well as internal needs through promotions;

- to identify new and unexplored relationships between products – for product discovery, up-sell, and outliers.< >< ><-->