5 Myths of Big Data Platforms

Is your federal organization struggling to use data to affect business outcomes and drive insight into day-to-day decision making? You’re not alone. Research indicates that organizations only use 12% of the data they own, while poor data quality costs organizations $13.5 million per year.

To move from data chaos to actionable insights, more organizations are turning to big data platforms, creating “data lakes,” or centralized repositories that store large amounts of structured and unstructured data. By bringing together vast amounts of data, organizations can now correlate activities in one part of the organization to effects in another, aggregate disparate siloed databases, and reduce overall operations and maintenance costs.

Implementing a big data platform is not without its challenges. For example, using proprietary analytical tools can mean organizations get locked into a one-off solution that doesn’t integrate naturally with their big data platform. As they navigate this and other challenges, federal agencies may have misconceptions about what big data platforms are, what they aren’t, and how to use them effectively.

Here are 5 myths of big data platforms for government organizations—and the reality.

Myth No. 1: If we combine all our data, we’ll have full insight into what our organization is trying to accomplish.

The Reality: Some data can’t be easily combined, and policies, restrictions, and security need to continue to be enforced. To protect sensitive data such as personally identifiable information or protected health information, organizations traditionally duplicate data in multiple data lakes or restrict access to specific users, driving up storage costs and reducing insights. Focus first on using metadata and attribute authorization schemes to enable a successful single data lake implementation and protect data at the source, record, or field level.

Myth No. 2: Once all our data is in one place, we can focus on analytics.

The Reality: You’ll need to catalog stored data to enable your analysts. Information such as data set name, formats, tagging, release-ability, retention, and more must be available and continually managed. In most organizations, data scientists spend 80% of their time “data wrangling” and only 20%performing real analysis. Focusing on data governance lets organizations begin to turn these statistics around. Data management is key to gaining value from data.

Myth No. 3: If we use industry technologies, we’ll be successful in implementing our big data platform.

The Reality: It’s not just about the technology. Successful organizations also adopt an agile culture—trying new ideas and pivoting quickly based on lessons learned. Big data platforms let organizations experiment and vet open source tools coupled with DevOps capabilities in sandbox environments before including them in the operational environment.

Myth No. 4: The more data we have, the more questions we can answer and the more knowledgeable we’ll be.

The Reality: It’s about the right data, combined with the right tools, for the right problem—not how much data is stored in the data lake. Instead of collecting all data and then asking questions, start by having a conversation about what problem you’re trying to solve and what data you need. The implementation team can gather information to capture metadata, better understand security concerns, and develop use cases enabling them to ingest and store data more effectively.

Myth No. 5: We don’t have enough data scientists to analyze the data and ask the right questions.

The Reality: Data-driven solutions need to create capabilities that democratize analytics to a wider audience. With simple designs, like a drag-and-drop user interface, more people on your team can analyze data. This allows your experienced data scientists to focus on new models while allowing a larger community to use previously built models and analytics against new data sets.

What’s holding organizations back from gaining value from their data is not technology, but rather a combination of processes, culture, and the right technology choices for the right problems. With modern data management platforms and best practices—including security—from the outset, organization could start flipping the 80-20 rule of data science. By adopting an open, standards-based big data platform and institutionalizing an agile culture, organizations can accelerate successful big data lakes and avoid falling victim to the big data swamp.