Big Data & Storage (Part 1)

The impact rapidly increasing data growth has on data centers

Arguably, there's never been a better time for businesses to be ex­cited about the data their employees and business units are generating. Software such as analytics and busi­ness intelligence are enabling orga­nizations to harness big data to gain greater insight into their customers and business processes, resulting in new avenues of revenue and profit­ability, improved customer commu­nication and retention, reduced IT costs, and other benefits.

On the other hand, data is being generated perpetually and at un­precedented levels due in large part to the massive numbers of mobile and Internet connected devices now in use. Further, effectively processing and analyzing the huge amount of structured and unstruc­tured data these devices generate in the form of Geolocation, audio, video, and photo files; tweets; Web posts; and other unconventional data files is difficult via traditional databases and software.

Thus, although businesses are justi­fiably interested in adopting big data initiatives, there's the pertinent ques­tion of how adopting and preparing for a big data initiative will impact the data center in terms of storage.

Enterprise Big Data demands a lot of infrastructure.

State of storage

Undeniably, data is growing at staggering levels globally. IDC Research (www.idc.com) reported that in 2000, all new data created glob­ally totaled about 2 million terabytes. Currently, double this amount is gen­erated daily. Last year, IDC reported that 1.8 zettabytes (1.8 trillion giga­bytes) was generated worldwide, and by 2015 roughly 8,000 Exabytes will be generated and stored globally.

Helping produce this data are smartphones, tablets, digital cameras, mobile sensors, smart meters, and other always connected devices, in addition to offline and mobile data that makes its way onto company networks. Although individuals gen­erate 75% of data in the digital universe, IDC states, enterprises have some liability for 80% of it during the data's digital lifespan. In coming years, this situation will continue, though IT staff pools tasked with managing data are predicted to grow only slightly. Moreover, IDC states that digital growth is outpacing storage capacity growth.

Elsewhere, in a recent survey of 30 enterprises, Aberdeen Group (www.aberdeen.com) found 58% of respondents listed the growing demand for storage capacity as one of their three top IT job pres­sures, "making it by far the most widely felt concern among profes­sionals from companies of all sizes." Aberdeen's research also revealed that storage capacity in enterprises grew 35% from 2010 to 2011, al­though 8% of those surveyed indi­cated data grew between 90% and 140%, while another 6% said it in­creased more than 150% per year. The average growth for large en­terprises was 44%, a rate Aberdeen states would require data capacity to double every 1.5 years. "Unless organizations get smarter and more efficient, they will almost annually need twice the number of storage devices, twice the space in the data center, and twice the personnel time and effort to manage their business data," Aberdeen states.

A recent CompTIA survey of 500 U.S. businesses found that many re­spondents must address basic data needs and technology infrastructure before launching big data initiatives. "Substantial" obstacles indicated as existing related to high or moderate degrees of data silos within organiza­tions, many organizations lacking a complete or accurate understanding of their data profiles, and few com­panies having comprehensive busi­ness continuity and disaster recovery plans in place.

A recent CompTIA survey of 500 U.S. businesses found that many re­spondents must address basic data needs and technology infrastructure before launching big data initiatives.

Big data’s storage impact

In terms of storage and other data center infrastructure compo­nents, tackling a big data initiative can prove a major challenge for any business no matter its size. SMEs (small and medium-sized enterprises) specifically, says Dick Csaplar, Ab­erdeen Group senior research analyst, storage and virtualization, "will need to have capabilities built into their storage devices, as they have few, if any, trained IT resources to manage the process."

John Robosson, Navint (www.navint.com) partner, says big data com­prises a "vast array of heterogeneous data sources that requires careful analysis to determine an overall big data strategy." The notion that an av­erage SME can "capture, store, and analyze all data to produce mean­ingful business intelligence isn't often achievable," he says, which is why it's imperative to develop a big data roadmap first. Big data's biggest im­pact on data center infrastructure, he says, will involve non-relational and unstructured data that a company determines has business intelligence value. "Capturing, indexing, storing, and retrieving new data types will present significant challenges to most SME data centers," he says.

In terms of storage and other data center infrastructure compo­nents, tackling a big data initiative can prove a major challenge for any business no matter its size

Ben Woo, Neuralytix (www.neuralytix.com) managing director, doubts SMEs will introduce in house big data ac­tivities in the near future but does foresee a surge of big data-as-a-service offerings popping up. Here, he says, SMEs can leverage cloud storage and existing storage. Initially, big data likely won't have a major impact on an SME's data center infrastruc­ture storage-wise, Woo says, as most data is already in existing systems or sourced from a third party. There will be incremental storage necessary to support a big data analytics frame­work, he says, and as enterprises re­tain more data for analytical purposes, storage is then "likely to increase quite dramatically," with 100% increases in capacity demands not being unusual.

Evan Quinn, ESG (Enterprise Strat­egy Group; www.esg-global.com) senior principal analyst, says companies should expect big data storage re­quirements similar to existing data warehousing, data mart, and business intelligence solutions. As a result of a flood of third-party data into big data analytics, new data sources (such as so­cial networks), and an increased rate of real-time and near real-time data, most organizations must be "on their toes in terms of information management practices and related storage infrastruc­ture," in order to create an ongoing, fully functional big data solution.

To eliminate guesswork from the big data equation, Quinn says, many big data vendors are wrapping in­frastructure and software into an ap­pliance, because big data requires balancing CPU, memory, compres­sion, cache, storage, and network bandwidth to work right. Rather than storage, he says, availability is the biggest hurdle most companies face with big data. "And that implies the entire infrastructure," Quinn says.