Google Adds To BigQuery Big Data Capabilities - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Cloud // Software as a Service
09:29 AM
Connect Directly

Google Adds To BigQuery Big Data Capabilities

Google expands the capabilities of its BigQuery system to allow real-time data stream processing and event analysis.

8 Google Projects To Watch in 2015
8 Google Projects To Watch in 2015
(Click image for larger view and slideshow.)

Google has announced updates to Google BigQuery and Cloud Dataflow -- the search giant's two big data management systems that compete with Amazon Web Services' DynamoDB and Data Pipeline.

In a blog, Google's William Vambenepe, lead product manager for big data on Google's Cloud Platform, claimed Google has implemented a more thorough "cloud way" to managing big data than other IaaS providers. By that Vambenepe means the service is provided without the user needing to know anything about how it's deployed, scaled, or managed, making it a "NoOps" service.

In one update to BigQuery, Google has introduced row-level permissions, a finer-grained approach to granting access to data in a database, according to Vambenepe. With row-level permissions, it's possible to grant a user access to a particular type of data in a database without opening up neighboring data to inspection.

Row-level permissions make it easier to share internal data with a variety of users. Partners or other parties outside the company can be granted permission to access a BigQuery data set in the cloud, but still be restricted to specific rows, Vambenepe wrote in his April 16 blog post. 

[Want to learn more about BigQuery competitors? See MongoDB Eyes Bigger, Faster NoSQL Deployments.]

The default ingestion limit for BigQuery has been raised to 100,000 rows per-second, per-table with unlimited storage for handling large data analysis tasks. BigQuery works with large structured data sets for SQL analytics similar to a relational database system, or with loosely structured data assembled as JSON (JavaScript Object Notation) objects.

(Image: Google)

(Image: Google)

Several NoSQL systems, such as Cassandra and MongoDB, also work with JSON objects.

The Google Cloud Platform also introduced the beta version of a new service, Google Cloud Dataflow. Cloud Dataflow provides event/time-based data stream processing, available as an on-demand service. Stream processing can also be scheduled as a batch service, if the Google Cloud user choses.

A Cloud Dataflow user doesn't need to set up a cluster on which to run the stream-flow processing.

"Just write a program, submit it, and Cloud Dataflow will do the rest," Vambenepe wrote.

Stream processing and event-related processing are done on a data stream, such as a feed of stock trades from an exchange, with the system looking for trades at a particular level of pricing, or at particular time intervals. Stream processing can also be used against an application's server log, where it watches for particular software events in the application and triggers an alert when it spots one.

Google's BigQuery processing and Cloud Dataflow stream analysis are now connected to another service -- Cloud Pub/Sub -- to allow notice of event occurrence to selected IT administrators or business end-users. Vambenepe wrote that Cloud Pub/Sub "completes the platform's end-to-end support for low-latency data processing."

Open source data systems, such as Hadoop, Spark, and Flink's data stream processing capabilities may be used with BigQuery as well, Vambenepe wrote. Google will provide connectors between those systems and its BigQuery and Cloud Storage services.

"Scuba equipment helps humans operate under water," observed Vambenepe, but they're no match for the agility of creatures that belong in the water. "When it comes to big data and the cloud, be a dolphin, not a scuba diver," he concluded.

Attend Interop Las Vegas, the leading independent technology conference and expo series designed to inspire, inform, and connect the world's IT community. In 2015, look for all new programs, networking opportunities, and classes that will help you set your organization’s IT action plan. It happens April 27 to May 1. Register with Discount Code MPOIWK for $200 off Total Access & Conference Passes.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Ninja
4/28/2015 | 5:54:05 PM
BigQuery is a big help
For anyone who wants to learn more about Google's BigQuery, it has a web page that is a great resource for an intro.  Just head on over to cloud dot google dot com forward slash bigquery. 
User Rank: Ninja
4/21/2015 | 2:49:34 PM
Real time analytics
With so many Android Wear about, its time we had some really nice real tiem data analysis. This is a crucial technology if done right with the right amounts of mix, we may have another cloud computing leader in the making.
2021 Outlook: Tackling Cloud Transformation Choices
Joao-Pierre S. Ruth, Senior Writer,  1/4/2021
Enterprise IT Leaders Face Two Paths to AI
Jessica Davis, Senior Editor, Enterprise Apps,  12/23/2020
10 IT Trends to Watch for in 2021
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/22/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you.
Flash Poll