Pundit Clay Shirky: Social media analysis adds BI benefits, challenges

Author and consultant Clay Shirky says social media data has lots of value, but exploiting it will require new business intelligence tools and cultural changes in organisations.

BARCELONA, Spain – “Do more with your data.” That was the primary message to customers from data warehousing specialist Teradata at its European user conference here last week.

Keynote speaker Clay Shirky, by contrast, set the audience a far more ambitious goal: to do more with their own data, certainly, but to also do more with data they find outside of the corporate firewall.

Shirky’s 2008 bestseller Here Comes Everybody sought to alert companies to the breadth and depth of online conversations that consumers were holding. His new book, Cognitive Surplus: Creativity and Generosity in a Connected Age, focuses more on the collective value of those conversations, not just for consumers but also for businesses and public-sector organisations that invest in social media analysis initiatives.

Exploiting that value, however, may require a new definition of the term “business intelligence,” Shirky said in an interview with SearchDataManagement.co.UK on the evening before his keynote presentation at the Teradata Universe event.

“Today, true business intelligence is out there in the world, in the minds of customers and in the conversations they have online about your company’s products and services,” said Shirky, who also is a technology consultant and an adjunct professor at New York University in the US.

But, he conceded, it’s difficult for companies to effectively capture and analyse that data, in part because traditional BI tools and approaches are simply not engineered to deal with the complexity of such information.

“It’s unstructured or semi-structured at best,” Shirky said. “It’s stream-oriented, making it difficult to archive. It presents an interpretative challenge – or an interpretative nightmare – that many people simply haven’t encountered before.”

Social media analysis may mean changes in corporate culture
To some extent, those barriers are being tackled by engineers experimenting with emerging technologies such as Hadoop and by the development of social media analytics software. But while new tools and BI methodologies are needed, Shirky thinks the bigger challenge for most companies looking to reap business intelligence benefits from social media analysis will be cultural.

“This isn’t just a new way to do old stuff,” he said. “If a business is going to treat the outside world as a BI resource, it needs to be able to participate in a conversation – contributing where appropriate, but also listening.”

That’s a dramatic departure from old-style marketing and communications, characterised by Shirky as “I talk and you shut up” (advertising) and “You talk and I shut up” (research).

Instead, he sees two new styles emerging. First, businesses will gather information from social networks and other external sources and take action on it. Second, they will take data they have inside their organisations, share it with customers and see what the customers think about the information and their plans.

“Shouting and spying have been the norm for business-to-consumer communications,” Shirky said. “That’s got to change.”

Jessica Twentyman is a business and technology journalist who has been a regular contributor to national newspapers and trade magazines in the UK over the past 15 years.

Read more on Business intelligence and analytics