VMware Vs. Microsoft: 8 Cloud Battle Lines

Datadog is a newcomer in cloud operation monitoring, with a claimed 27,000 servers under its watchful, online service. It's different from other young monitoring systems, which are proliferating alongside increasing cloud use, in that it draws data from many existing tools and monitoring systems and applies analytics for a composite picture.

It's different in another way: It has major cloud vendor HP, with its OpenStack cloud, as a customer.

Today's monitoring systems tend to come either from the system's manufacturer or from a little-known, third-party startup. Datadog is one of the latter. But with composite applications often operating across several systems, or workloads running both on-premises and in the cloud, the value of monitoring is rising. Manufacturers tend to be too narrowly focused on their own systems. The most skillful third parties are likely to gain attention quickly, or become acquisition targets over the next few years as hybrid cloud operations monitoring becomes more frequent.

[ Do you agree with VMware that Puppet data is important to operations? Read VMware Invests $30 Million In Puppet Labs. ]

"Our system is collecting 15 billion data points a day" from the customer base, scrutinizing them and notifying customers if something appears to be amiss, said Alex Rosemblat, Datadog director of marketing, in an interview.

Instead of using agents on applications or network sniffing to get operational data, it's plugged instead into the configuration engines Chef and Puppet; the build engine Jenkins; the NoSQL systems MongoDB, CouchDB and Cassandra; the cloud metrics system of Amazon's CloudWatch; relational databases; and other sources.

Datadog aggregates information from various systems, digests it and puts relevant graphs, charts and specific indicators into a continuously updated dashboard to show what's going on. It collects information from open source code management systems Git and Github, Nagios system management, the Apache Web server, and Airbrake's monitoring system.

Datadog CEO Olivier Pomel said in an interview that an operations manager can drill down into the information "to find out who's doing what." Because Datadog collects information from code change management and configuration management systems, a manager can find out what's recently changed and who made the change, if a system starts to slow down or fail.

Datadog is designed to be used by both the operations managers and developer teams trying to frequently update key production systems. At one customer, "the DevOps team spends quite a bit of time studying the information on the screenboards," Pomel said. Datadog uses the term "screenboards" to describe the combination of key performance indicators, reported infrastructure software events and system metrics on its dashboards.

Screenboards can be configured to show the combination of information from the various systems that developers or operations managers think is most relevant. Information collected by Datadog can be dragged to a screenboard and dropped to become part of its display. The screenboards can be given a URL to provide information to distributed team members or external stakeholders. The screenboards are updated in real time and can combine information into charts and graphs, widgets, HTML iFrames or alerts. Users may apply their own metrics, if they want a custom display of the information available.

Datadog supports Yammer, and custom metrics for a Java application can be constructed using the Yammer Metrics library. Its analytics engine can correlate information from Nagios systems management with other information it's been directed to collect. When necessary, Datadog places its own reporting agent on the open source systems that it is monitoring.

The content delivery network MaxCDN uses Datadog to take operational snapshots of its software infrastructure. In the announcement Tuesday, Kevin Meinert, MaxCDN VP of operations, said Datadog "can take a snapshot of our infrastructure specific KPIs in seconds. We previously had to develop custom dashboards that had to process huge amounts of data ..."

Tuesday's announcement marked the addition of the screenboards to a monitoring product that has been available since summer 2012. Datadog is software-as-a-service available at a charge of $15 a month per monitored server. Datadog has 20 employees, 15 of them engineers, said Pomel.

Yesterday’s innovative data center may be today’s money pit. Is it time for a new plan? Also in the new, all-digital Data Center Decision Time issue of InformationWeek: Data center consolidation is tough, as the government's experience shows. (Free registration required.)