The Big Data Boom

The New York Times has a look at the investment boom in "big data" companies, driven in part by the slow roll out of smart meters - For Start-Ups That Aim at Giants, Sorting the Data Cloud Is the Next Big Thing.

The idea of big data goes something like this: In a world of ever-increasing digital connectivity, ever larger mountains of data are produced by our cellphones, computers, digital cameras, RFID readers, smart meters and GPS devices. The huge quantity of data becomes unwieldy and difficult for companies and governments to manage and understand.

“My smartphone produces a huge amount of data, my car produces ridiculous amounts of really valuable data, my house is throwing off data, everything is making data,” said Erik Swan, 47, co-founder of Splunk, a San Francisco-based start-up whose software indexes vast quantities of machine-generated data into searchable links. Companies search those links, as one searches Google, to analyze customer behavior in real time. ...

Founded in 2004, before the term “big data” had worked its way into the vocabulary of Silicon Valley, Splunk now has some 3,200 customers in more than 75 countries, including more than half the Fortune 100 companies. ...

Founded in 2004, before the term “big data” had worked its way into the vocabulary of Silicon Valley, Splunk now has some 3,200 customers in more than 75 countries, including more than half the Fortune 100 companies. ...

The amount of data being generated globally increases by 40 percent a year, according to the McKinsey Global Institute, the consulting firm’s research arm. And while Splunk has a lead in selling software to analyze machine data, big data is big enough to create new opportunities for a multitude of start-ups, many of them using the open-source software Hadoop.

“Venture capital is absolutely foaming at the mouth over big data,” said Peter Goldmacher, an analyst and managing director at Cowen & Company. “The volume of data being created now is not 10 times bigger, it is like a thousand times bigger.”