Data Scientist: Hot Big Data Job

Big Data Talent War: 10 Analytics Job Trends

IT managers who have weathered a paradigm shift or two could be forgiven if they haven't taken to heart the many warnings of a critical, impending skills shortage in the market for big-data specialists.

Every major new technology that comes along--which happens about ever two or three years--causes a skills shortage in the IT hiring market, , according to Alice Hill, managing director of IT job-ad site Dice.com.

"There are a lot of perceived gaps in available skills in the new technology just because the technology is new and there hasn't been time for people with related skills to become trained in the new one," Hill said. "If a gap continues for a long time it could put upward pressure on salaries, but what usually happens is that companies end up training existing staff and only hire a few people with specialized skills."

The current shortage--or at least the perception of a shortage--becomes more intense when the overall market for IT people heats up, though both hiring and demand for IT skills has been relatively flat during the down economy of the past few years according to U.S. Bureau of Labor Statistics hiring figures.

[ Avoid common mistakes when launching a big data project. Read 10 Big Data Migration Mistakes. ]

The unemployment rate for technology workers is about half that of the general population --4.4% for tech workers during Q1 2012 compared to 8.3% for everyone else.

Within even tepid markets there are geographic or specialty areas in which the job situation is dramatically different than the average for the rest of the IT market, Hill said.

The hot, disruptive technologies of the moment--among which big data is the current leader--is the usual victim of such short-term shortages. And it appears that all big data skills are in short supply.

Mckinsey & Co., which published the seminal analysis of the growing big data market in 2011, predicted the U.S. would face a shortage of between 140,000 and 190,000 workers with the skills to manage and analyze big data.

On a national scale, those aren't large numbers. Compared to the existing supply of new graduates or existing specialists with the right skills, however, it represents a shortage of 50% to 60% even as far into the future as 2018.

Unlike most other disruptive technologies--cloud computing, virtualization, mobile computing, for example--big-data skills require specific talents with data, numbers, and business processes, and long training or retraining periods, McKinsey concluded.

The deeper the needed skills the more intense the talent shortage will be and the longer it will take to fill, the report concluded.

Though the McKinsey report was published in January of 2011, nothing has changed to lessen the intensity of the skills shortage in the time since, according to an April story in The Wall Street Journal quoting both big-data vendors and users.

Particularly sharp is the shortage not of big-data analysts, but data scientists--an imprecise term describing specialists who can not only supervise the integration of many types of data into a single data set and then find within it nuggets of information that could produce dramatic benefits.

Forty-five percent of business intelligence projects fail due to a lack of data expertise on staff, according to an April survey from GigaOm.

Because of the greater degree of difficulty gathering and integrating data, sifting it for answers, and translating the result into knowledge business units can use, it is even more important than in most data-centric projects to have real experts on staff for big-data projects than more run-of-the-mill efforts, according to Mike Boyarski, director of product marketing for business-intelligence/big-data software vendor Jaspersoft.

12 Top Big Data Analytics Players

Data scientists "can take a data set and model it mathematically and understand the math required to build those models," according to Hilary Mason, chief scientist for the URL shortening service bit.ly., as quoted in the Journal. "That means asking the right questions, and that is usually the hardest piece."

IBM's definition of data scientist describes the role as being "part analyst, part artist."

Traditional data analysts look at data from a single source--a CRM application, for example. Data scientists examine data from multiple sources, rationalize differences among the data types, and create a data set usable by less specialized analysts that also accurately reflect trends within the data itself and how to apply them, according to IBM.

The requirements described by data specialists themselves are much more mundane, and much less likely to contain the term "data scientist," at least in their own job descriptions.

[ Are you gaining real benefits from your big data? Read How To Find Strategic Advantage From Big Data. ]

Data scientists may simply be the most senior among a group of data-analysis specialists. More likely those given or choosing to use the title are better educated, better paid, more experienced, and have a wider set of skills than others with big-data skills, according to a survey of data-analytic specialists (PDF, free registration required) recently published by BI vendor SiSense.

Only 5% of most data professionals hold a Ph.D. in a relevant specialty, for example, while 35% of data scientists hold one, the survey showed.

Data scientists also make more money than other data professionals. Those without management titles averaged between $70,000 and $90,000 per year, compared to $65,000 to $70,000 for more traditional data specialists.

On-the-job experience counts for as much as education, however. Those with 10 years of experience or more get salaries 80% higher than those with equivalent training but with three years experience or less, the survey showed.

Data specialists of all varieties, but especially those claiming the title scientist, have been doing better financially during the past two years as well. Forty seven percent reported earning between 1% and 10% more this year than last; 7% reported increases of between 10% and 20%.

Another 7% percent reported raises of more than 20% in 2012 compared to 2011.

Seventy-eight percent expect to make more during 2013 as well; 11% expect raises of 20% or more, 14% expect raises of 10% to 20%.

The upshot is that data scientists have more education, experience, and breadth of understanding than other big-data specialists and are increasingly expecting to be paid a premium for that knowledge.

While some technical skills are particularly valuable--especially BI and data warehouse skills, math and statistical abilities, and data visualization--project management, business-function expertise, and general business skills are rated even more important.

Despite advantages in career development, compensation and prestige, very few data specialists actually call themselves data scientists.

Only 15% of respondents listed their job title as containing the word "scientist," compared to 34% listing themselves as "business analysts," 27% with the title "data analyst," and 19% with management titles such as director of analytics or VP of analytics.

The reason so few who could claim the title actually do use that title isn't answered in the SiSense data or anyone else's, but the accompanying analysis does suggest a reason.

Despite the buzz around big data, "there is no clear definition of what a "data scientist" really is."

Whatever the job description really is or may eventually be defined to be, the one thing about data scientists that nearly every survey and analyst agrees on is that there are not enough of them now and that the demand for them will continue to grow during at least the next five years. Even after the first rush of demand for big-data skills passes, the requirements for becoming a data scientist ensures they will remain a rare breed.