10 Best Practices in Big Data

Here are some must have practices for the best utilisation of Big Data in enterprises

Mitesh Agarwal  |  04 June 2012

Mitesh Agarwal

Researchers estimate that enterprise data grows nearly 60 percent per year (90 percent of that being unstructured) and the average amount of data stored per company is 200 terabytes. This growth is triggered by the increasing channels of data in today’s world. Examples include, but are not limited to, user-generated content through social media, web and software logs, cameras and intelligent sensors embedded in physical devices that can sense, create, and communicate data.
Companies are realizing that now is the time to put this data to work. However, several obstacles limit their ability to turn this massive amount of unstructured data, often termed as Big Data, into profit. The most prominent
obstacle being a lack of understanding on how to add Big Data capabilities to the overall information architecture to build an all pervasive Big Data architecture. Companies have to understand that planning Big Data architecture is not about understanding just what is different. It is also about understanding how to integrate what is new when compared to what is already in place. Here are a few general guidelines to build a successful big data architecture foundation:
1. Align Big Data initiative with specific business goals
One of the key characteristics of big data is value - value through low-density and high volumes of data. As we sort through the mountains of low-value-density Big Data and look for the gold nugget, do not lose sight of why we are doing this. Follow an enterprise architecture approach. Focus on the value it provides to the business. How does it support and enable the business objectives? Properly align and prioritize big data implementation with the business drivers. This is critical to ensure sponsorship and funding for the long run.
2. Ensure centralized IT strategy for standards and governance
Some of the recent analysts’ surveys indicate that one of the biggest obstacles for Big Data is skills shortage. A 60 per cent skills shortfall is predicted by 2018. Addressing such a challenge through IT governance to increase the skill level, to select and enforce standards, and to reduce the overall risks and training cost would be an ideal situation. Another strategy to consider is to implement appliances that will give enterpirses a jumpstart and quicken the pace as enterprises  develop their in-house expertise.
3. Use a center of excellence to minimize training and risk
Establishing a Center of Excellence (CoE) to share solution knowledge, plan artifacts and ensure oversight for projects can help minimize mistakes. Whether Big Data is a new or expanding investment, the soft and hard costs can be shared across the enterprise. Another benefit from the CoE approach is that it will continue to drive the Big Data and overall information architecture maturity in a more structured and systematical way.
4. Correlate Big Data with structured data
Enterprises should establish new capabilities constantly and leverage their prior investments in infrastructure, platform, Business Intelligence and Data Warehouse, rather than throwing them away. Investing in integration capabilities can enable the knowledge workers to correlate different types and sources of data, to make associations, and to make meaningful discoveries.
5. Provide high performance and scalable analytical sandboxes
When a problem occurs humans can solve it through a process of exclusion. And often, when we do not know what we are looking for, enterprise IT needs to support this “lack of direction” or “lack of clear requirement.” We need to be able to provide a flexible environment for our end users to explore and find answers. Such sandbox environments also need to be highly optimized for performance and must properly governed.
6. Reshape IT Operating Model
The new requirements from Big Data will bring certain changes to the enterprise IT operating model. Provisioning of new environments will need to be more timely and user-driven. Resource management also needs to be more dynamic in nature. A well planned out cloud strategy plays an integral role in supporting those changing requirements.
7. Embed Big Data insights into business applications
To make big data operationally feasible, business should invest in solutions that enable embedding of insights generated from big data into front-end user applications. For example a bank has to put the insights they have got from Big Data into their call center app, their web banking app and potentially their core banking app as well. Solutions like Oracle Real Time Decision (RTD) and Oracle Complex Event Processing (CEP) can provide a readymade layer that abstracts the insights making life simpler for the business to put the rules into a single engine from where they can be consumed by various front-end user applications in uniform fashion.
8. Ensure security for Big Data
Enterprises worldwide have ranked IT security as one of their top priorities as increasingly sophisticated attacks, new data protection regulations, and most recently insider fraud and data breaches, threaten to disrupt and irreparably damage their businesses. Security in the Big Data world is even more crucial. Enterprises should invest in integrated security solutions to ensure that big data insights generated from the integration of old world structured data and the new world unstructured data is not compromised in any way.
9. Look at Big Data as an extension of existing information architecture
Enterprises should look at big data investment as an extension to their existing information architecture, not a replacement, not a standalone island. Invest in solutions that integrate Open Source and Enterprise Technologies. They should look for economic advantages in their architecture through simplifying and standardizing their IT Operations, through reduce IT Investment and building their infrastructure once for enterprise scale and resilience; through unified development paradigms, and through sharing metadata at the enterprise level for integration and analytics.
10. Invest in Shareable Architectures to extend analytical capabilities
A lack of enterprise-ready statistical analytical tools prevents the kind of analysis necessary to spot trends. Enterprises should invest in Shareable Architectures to extend their analytical capabilities in visualization, semantics, spatial, and statistics analysis.

Mitesh Agarwal is CTO & Director - System Solution Consulting, Oracle India.
(Courtesy: CTO Forum)