Solutions

Harnessing Big Data for Precision Medicine, Part 3

The paradigm shift of the ‘data lake’

At the heart of our approach to analytics and precision medicine is what we call the “data lake,” a new way of storing and managing data. Data is no longer locked in limited, isolated databases. Instead, all the available data is consolidated into a single pool, or “lake.” It is both stored and analyzed in the cloud, using networks of computers.

What makes this new approach possible is the way the computer finds the data. With a relational database, each piece of data is assigned a location based on rows and columns, as with a spreadsheet. Because the data has to be painstakingly formatted, this method only works well with relatively small amounts of data.

The data lake solves this problem by identifying the data in a way that doesn’t rely on rows and columns. Instead, as each piece of data is put into the data lake, it is “tagged” with accompanying details that can be used to locate it. For example, a piece of patient information, such as genomic data, can be tagged with other information about the patient, such as age, medical condition, medications, income, etc.

“All of the available data can be analyzed for insight—all at once. This opens the way for entirely new realms of inquiry in precision medicine.”

Tagging works for big data because it is much faster than conventional formatting methods. Data that would have taken days to format for relational databases can be tagged for the data lake in a matter of minutes.

This new method has several important advantages. First, we can now integrate a virtually unlimited amount of information, from any number of data sources. All of the available data can be analyzed for insight—all at once. This opens the way for entirely new realms of inquiry in precision medicine.

In addition, the data lake easily accepts unstructured data, which can now be searched for insight side-by-side with other types of data. Our approach uses advanced analytic methods to prepare and tag the data for the data lake. For example, we use machine learning and computer vision to identify the content of images. Because this can be done in an automated process, it eliminates the labor-intensive manual work required in conventional methods.

We also use natural-language processing and other techniques to make clinicians’ notes, research papers, social media content and other text available for the data lake. These methods have been used by the intelligence community to understand the ideas contained in covert messages. We are now applying them to precision medicine.