Data scientist is the ‘Sexiest Job of the 21 Century’, so say Thomas Davenport and DJ Patil in their seminal 2012 Harvard Business Review article.
Since then, data scientists have been very much in demand and their wages have soared. Compared to statisticians, computer programmers and AI guys (who are still regarded as geeks), they are extremely well paid.
But are they worth it? Is there a good reason why data scientists should get paid more or are their pay packets over-inflated due to unnecessary hype?
According to academic research by Anders Ericsson and popularised by Malcolm Gladwell in his book ‘Outliers: The Story of Success’, that’s how long it takes to become an expert in something.
Well actually it’s a bit too general. 10,000 hours is an average across lots of disciplines including sport, business, education, music, etc.. Obviously it’s not going to take you 10,000 hours to be an expert in tiddlywinks, snap or noughts and crosses, but if you want to be an expert in 3D chess, quantum physics or neuroscience, then it probably will.
Ericsson also points out that the quality of the practice time is important too. Reading general anatomy books for 10,000 hours will not make you an expert in open heart surgery, but making specific improvements to your knowledge, technique and experience on a daily basis will.
So let’s say – for the sake of argument – that 10,000 hours is what it takes to be an expert in any serious academic pursuit.
In a typical work year of 2,000 hours, then, it’ll take you around 10 years to become an expert.
Yes, my maths is correct(ish) – I’m factoring in how much time you spend on emails, ebay, funny cat videos on TwitFace and the worthless one-hour meetings that interrupt at regular intervals.
So there you have it – dedicate yourself to your chosen discipline and you’ll be an expert in around 10 years.
Well OK, it might take 10 years to become an expert, but it’ll take much less than that to become pretty good at something. But how long?
This is where the Pareto Principle comes in handy.
According to the Pareto Principle, also known as ‘The 80/20 Rule’, you can get 80% of the required results by putting in 20% of the effort, so according to Pareto you can get 80% of the way towards being an expert in 20% of the time. In terms of Ericsson’s 10,000 hours, that’s 2,000 hours or 2 years.
That’s pretty good, really. It means that you only need to work hard for 2 years to be a nearly-expert.
When I look back over my career I sort of recognise this. After a couple of years I’m beginning to get pretty good at something, after which, improvements come much more slowly – according to the flip-side of Pareto’s coin, the final 20% of the way to becoming an expert takes 80% of the effort.
But what does this really mean? If it only takes 20% of the time to attain 80% of the knowledge, then surely the stuff that defines an expert must lie in the remaining 20% that you haven’t reached.
So working for the first 20% of the time gets you strong foundations, but doesn’t qualify you as an expert.
Making good decisions based on your knowledge and experience requires a strong foundation, but making consistently better, more informed decisions requires more than this, it requires expert knowledge and an instinct about your problem domain.
To illustrate this, consider the case – featured in Gladwell’s book – of a lieutenant firefighter who claimed that ESP saved his life and those of his crew. The firefighter’s team was fighting what appeared to be a simple kitchen fire in a one-story house. The lieutenant led his crew into the back of the building to spray water onto the fire, but the fire just kept roaring back. He thought the water should have had more of an impact so he ordered his men back to the living room to regroup. Then, feeling that something wasn’t right, he ordered his men to evacuate the building. As soon as they left, the floor they were standing on collapsed.