If you’re in IT and not looking at the new concept called big data, you’re missing a critical development in technology, according to market research firm Forrester. Big data is the practice of manipulating, analyzing, and using data in such large volumes that typical tools cannot cope. Become an expert in this new world and you’ll add significant value onto the information systems degree you may already have.
Forrester claims that companies generally use less than 5 percent of the data they have available and that data grows at 40 percent to 50 percent annually. (According to IBM, 90 percent of the world’s data was created in the last two years.) Move outside a company’s walls and the amount of data management might want to examine explodes. For example, imagine trying to analyze basic market trend data that would appear on Twitter. You could conceivably be dealing with tens of millions of entries.
Using only the tools and techniques of the past, there is no way companies can ever catch up, let alone keep pace. As we mentioned last summer, according to McKinsey Global Institute there are five ways that big data can help create value:
- make information transparent and usable, unlocking its value
- collection more accurate and detailed performance information on all aspects of the business
- perform more detailed customer segmentation, opening the door to increased marketing effectiveness
- better analytics can mean better decision making
- improve the development of new products and services
Not that more data makes all things better. One principle of strategic planning is to understand that for a given decision or insight, there may be types of information that won’t help. For example, if you want to enter a new market, looking at your own historic sales data will do no good. And yet, that data may be important to other parts of the company, but only if it’s available and a company is capable of pulling the insights from it.
Handling large datasets takes different tools and techniques. For example, one common change is to use stream processing algorithms that efficiently undertake calculations like certain statistical functions as they go through a long list of numbers rather than waiting until after reading in all the data. You’ll likely have to use parallel processing and advanced visualization. Tools like the specialty database Hadoop are common.
McKinsey is sure that the ability to employ big data will become a key basis of competition. If so, then the knowledge of how to make big data work for a company will become a key distinction in the resume of any IT specialist.