How Data Collaboration Solves Big Data’s Biggest Problems

There are a growing number of data collaboration solution providers competing to solve the IT management challenges of enterprise data fragmentation. This is the problem where siloed teams all want to work on large data sets, including data from other teams, and so copies of the data are made and shared, resulting in increased costs of storing and managing multiple copies of the same data in an enterprise.

Enter data collaboration, a relatively new approach that is shaking up big data management and is at the cutting edge of data-driven decision making. Data collaboration allows multiple parties to share access to their data, which gives all parties a more holistic view of the project and enables deeper insights. Not only do teams reap the benefits of having immediate access to a wider array of data, IT departments benefit from reduced data fragmentation and lower management costs while simplifying the deployment architecture.

These platforms also offer significant value-added features, such as rapid integration and development, as well as advanced data visualization. At the same time, they eliminate the need to distribute and control additional copies of data. Data collaboration does away with the necessity of developing APIs in applications or planning complex solutions to make the data fully accessible -- all of which can take months before teams can even get their projects started. Instead, users can directly leverage the platform tools to build solutions rapidly and turn around a digital service without the need for skilled software developers to write extensive code. Think of it as what CMS products did for web publishing, or what cell phone cameras have done for content creation. It produces instant results by removing all the heavy lifting.

While the benefits are apparent for enterprises, what about organizations or researchers that want to collaborate with third parties or partners outside of their controlled networks? With data collaboration, the biggest boon to organizations is the added security features for controlling access to the data. Since access is provided through the platform itself, the granularity of access can go down to the data cell level. This allows owners to decide which data are shared to maximize collaboration, while ensuring conformity to regulatory and security requirements. There is no need to worry about what a third party will do with personal data: the owner is always in control and no copies are ever distributed.

In business, data collaboration is the key for achieving the next level of strategic decision making. Imagine you want to expand your department store offerings to increase revenue. You already have your own data on what customers are buying in your store, but what are they buying that you don’t already sell? With data collaboration, you could partner with a payment system provider and find out what customers spend their money on. Critically, not only would you know what your customers are buying from your store versus elsewhere, but you could also see what purchases they are making on the same day. That additional data from a collaborator could tell you if you should open a grocery section and whether it can effectively compete by saving your customers an extra trip. The advantages that data collaboration offer to businesses is obvious.

Magical unicorn?

By now, data collaboration may start to sound like a magical unicorn. How is this practical or even possible? How can you achieve insight when you don’t possess the data? The reality is that with large data sets, you don’t gain insight by looking at the actual data. There’s just too much of it. Instead, you formulate queries around the questions that need to be asked. It’s the query results that will provide the insightful answers. This is already a common practice with data that you do own, so imagine the possibilities of combining data across markets and industries without the cost and expertise barriers of implementing traditional data sharing, integration and security controls.

All this added insight can be achieved without having to know who the individuals are and what each person is purchasing. As a collaborator, you can ensure that personally identifiable information is not exposed, and your customers’ privacy is maintained, because it is the queries that produce insights. These queries can be shared with other collaborators for mutual benefit. As an IT leader, you can reduce your complexity and costs for information systems and data storage. As a program manager, you can accelerate outcomes by rapidly prototyping and productizing solutions. It’s a revolution for big data with the potential to transform how we make use and make sense of data without compromising on security and privacy.

Tal Golan is a technology investor, entrepreneur and former chief strategy officer at Verb Technology.