Working With Big Data Requires Planning

Working With Big Data Requires Planning

March 12

As corporations grapple with Big Data—massive amounts of data from blogs, user-contributed content, social media streams, video files, and mobile communications, not to mention the data normally going through their IT infrastructure—a variety of applications is emerging to help them cope. However, you can’t just pick an analytics tool and slap it in. You must know what data you have before you can analyze it. According to Protiviti’s 2013 IT Priorities Survey, managing and classifying enterprise data continues to pose a major challenge to IT. An associated problem is storage management and planning, which remains a top concern for respondents to the survey.

Setting Up a Data Storage Architecture For maximum effectiveness in working with your big data, you need to set up a data storage framework, or architecture. Some of the requirements for data storage architecture include scalability, tiered storage, high availability and accessibility of content, the possibility of integration with legacy applications and with cloud ecosystems, and support for workflow automation, recommends Janae Lee, senior vice president of file systems and archive products at Quantum. Scalability isn’t achieved by throwing storage at the data. Instead, you must analyze your data and figure out what data you access most and to which you need the most rapid access, and which data can be tucked away in vaults. Then you can tie the kind of storage you’re using to those different needs, choosing among flash storage, disk storage, and tape, and coming up with the optimum combination as dictated by your budget. Now, the classification of data will change as it ages, and you have to revisit your data regularly and reclassify them. Then, you can move them from high-availability storage to less-available storage to the vaults as required. Use properly formulated policies to automate this process. You may also have to decide which data need to be retained onsite and which can be shunted off to less-expensive cloud storage. This is another area for automation. Also plan for security when you set up your data framework, because it’s not a question of if your data get breached, but when.

Leveraging Data for Profit Once you’ve implemented your data architecture, you need to figure out how to leverage it. Big data analytics are available from several companies, including SAS, Teradata, IBM and EMC. Or, if you want to go the open source route, you can turn to other companies, including Talend, Ikanow and Red Hat. But don’t just put your data through an analytics grinder and hope to get the right answers. As the Accenture Technology Vision 2013 report warns, you may find gaps in the resulting information because you didn’t formulate important questions when you designed the applications that query that data. Ask the questions you need to be answered first, then design applications to get those answers.

Photo Credit: photosteve101 via Compfight cc [cf]skyword_tracking_tag[/cf]