Apache Spark Updates, Executive Data Strategies: Big Data Roundup - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management
10:05 AM
Connect Directly

Apache Spark Updates, Executive Data Strategies: Big Data Roundup

In our big data roundup for the week of Nov. 27, we've got updates on Apache Spark and its ecosystem, a reality check on the practical use of artificial intelligence today and more.

CES 2016 Sneak Peek: 9 Cool Gadgets
CES 2016 Sneak Peek: 9 Cool Gadgets
(Click image for larger view and slideshow.)

You might expect a slow week for big data news as all the analysts and data scientists focused on optimizing meal preparation, predicting football scores, and surfacing the best prices on the most sought after holiday gifts. But you'd be wrong. This week we had news from IBM and Databricks about Apache Spark, a closer look at how a couple of different companies -- AccuWeather and McGraw-Hill Education -- have built analytics into the services they offer, a great primer on Artificial Intelligence and the future, and more.

Let's start with the IBM and Databricks announcements around Apache Spark.

You may remember that IBM Analytics contributed its SystemML engine for machine learning to the Apache Foundation earlier this year. IBM said the goal was to improve upon Spark's MLlib machine learning algorithms and libraries. This week IBM announced that its SystemML has been accepted into the Apache Incubator program.

(Image: PonyWang/iStockphoto)

(Image: PonyWang/iStockphoto)

Also this week, Databricks started previewing its software-as-a-service implementation of the next version of big data platform Apache Spark. Spark 1.6.0 is not due for general release until mid-December, but big data pros can take it for a test run on Databricks starting now. The new version is focused on improving performance. Check out the full story on Apache Spark's updates this week here.

What kind of company has been in the business of making predictions from data for a really long time already? How about companies that predict the weather. InformationWeek recently spoke to AccuWeather's Chief Commercial Officer about how the company is leveraging its proprietary analytics systems and big data to offer what amounts to predictive analytics as a service for customers. Retailers are big believers in this, but other industries are getting on board, too.

Our InformationWeek story has all the details.

We also spoke this week to the Chief Digital Officer of McGraw-Hill Education about how his company is looking at student click streams on digital curriculum and leveraging them to improve outcomes for individual students. The company is also using them to help instructors gain insights on how to best help students and to improve its own curriculum. 

Artificial Intelligence is a big topic in the industry and among science fiction writers, but how much do you really know about how it's being used today? InformationWeek recently connected with an expert on this technology for a reality check.

Meanwhile, it appears that if you are a data scientist (or studying to be one) you're in a great place in terms of salary. Data scientist is one of the job titles that is getting a big salary bump in 2016, according to our list here. Click through to find out just how much of an increase, and what other job titles can expect rich rewards next year.

[IBM Watson wants to help you leverage analytics to get the jump on hot gifts for the holidays. Read IBM Watson Trend App: Big Data Meets Holiday Shopping.]

In this shortened week, we did not have time to cover a couple of new partnerships announced by Dell Services around healthcare analytics. The company signed a multi-year agreement with Zebra Medical Vision to deliver a platform for medical imaging research. Dell Services also announced the integration of cloud-based analytics software BizEye to the Dell Cloud Clinical Archive portal.

We may have mentioned this one before, but since you are at the end of a long Thanksgiving Day weekend and perhaps still unsure about your holiday shopping strategy we thought we'd mention this one again. IBM Watson has created an iOS and web app to surface the hot gifts this holiday season. Don't know what to get for your kids, spouse, mother-in-law? You may find your answers, or at least some inspiration here

**New deadline of Dec. 18, 2015** Be a part of the prestigious InformationWeek Elite 100! Time is running out to submit your company's application by Dec. 18, 2015. Go to our 2016 registration page: InformationWeek's Elite 100 list for 2016.

Jessica Davis has spent a career covering the intersection of business and technology at titles including IDG's Infoworld, Ziff Davis Enterprise's eWeek and Channel Insider, and Penton Technology's MSPmentor. She's passionate about the practical use of business intelligence, ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
12/8/2015 | 5:48:48 AM
Re: SystemML is an algorithm processing engine
I learned a lot by reading your post but I'm just asking myself a question : A friend of I used to speak about Apache Cassandra, what is the difference between Cassandra and Spark for Big Data ? 
User Rank: Ninja
12/5/2015 | 4:42:21 PM
Re: SystemML is an algorithm processing engine
@Charlie - thanks for sharing, as technology changing... it not easy to keep up
Charlie Babcock
Charlie Babcock,
User Rank: Author
11/30/2015 | 4:09:41 PM
SystemML is an algorithm processing engine
During IBM's DataPalooza in San Francisco, IBM VP of Analytics Product Rob Thomas explained to IW that SystemML is an engine for running machine learning algorithms, such as those found in Spark's MLib library. The engine takes advantage of a server cluster to distribute work and process results efficiently. Machine learning frequently requires a scale-out architecture.
IT Salary Report 2020: Get Paid What You Are Worth
Jessica Davis, Senior Editor, Enterprise Apps,  2/12/2020
10 Analytics and AI Startups You Should Know About
Cynthia Harvey, Freelance Journalist, InformationWeek,  2/19/2020
Fighting the Coronavirus with Analytics and GIS
Jessica Davis, Senior Editor, Enterprise Apps,  2/3/2020
White Papers
Register for InformationWeek Newsletters
State of the Cloud
State of the Cloud
Cloud has drastically changed how IT organizations consume and deploy services in the digital age. This research report will delve into public, private and hybrid cloud adoption trends, with a special focus on infrastructure as a service and its role in the enterprise. Find out the challenges organizations are experiencing, and the technologies and strategies they are using to manage and mitigate those challenges today.
Current Issue
IT Careers: Tech Drives Constant Change
Advances in information technology and management concepts mean that IT professionals must update their skill sets, even their career goals on an almost yearly basis. In this IT Trend Report, experts share advice on how IT pros can keep up with this every-changing job market. Read it today!
Flash Poll