William Terdoslavich has spent a career in and out of tech journalism, having written for InformationWeek, The Hubbs.com, Computer Reseller News, Computer Systems News and Mobile Computing and Communications. Technology will always change. Human nature remains the same, usually crazy.
In the realm of Big Data, many people are trained to use analytics to find the right answers. But do we have enough people who can ask the right questions?
Many articles have been written in the business and tech press about intuition, the gut feeling that allows a professional to cognitively leap to an insight. It is a skill needed to ask the right question. Yet no one knows how to create an “intuitive,” much less train one. And guidance on when to use intuition is limited to “use your judgment,” which is not exactly precise, or helpful.
So where does intuition come from?
You can cultivate intuition, perhaps by finding people who have it. Or you can ignore people entirely and focus on artificial intelligence and machine learning. Which one of these methods will serve you best?
Intuitive people do not just walk in off the street to join a team. Intuition must be kindled somewhere in the team to make analysis work.
“It’s really hard to quantify and give a recipe for creation,” said Ted Dunning, chief application architect at MapR, a California-based software firm that develops data platforms based on Apache Hadoop. “It is hard doing the unexpected by rote. By definition, it can’t happen that way.”
Instead, one can exercise the team to think intuitively. Such thought exercises encourage approximation as a first step in pursuing exactness.
One way is to ask team members questions that they must answer with a range, explained Ellen Freidman, a solutions consultant at MapR. For example, what is the weight of a Boeing 747? Or what is the diameter of the moon? If the factual answer falls within the range, then the approximate answer is correct. Narrowest range wins.
“Developing the ability to approximate is a huge skill that goes underdeveloped,” Freidman said. Approximation does not contradict precision, nor is it a wild guess. It does allow people to solve a problem within the boundaries of error, she noted: “People need to recognize the difference between what they know and what they don’t know.”
Another key step: formally setting some time aside to think, even if it just 30 minutes a week, Freidman continued. This allows you to get to the essential concept of the problem you are tying to solve. “It leaves the door open to making new connections.” she said. “No one takes time to do this.”
Also, “you need to make room for them (the team) to be wrong,” Dunning added. “If you have room to be wrong 90 percent of the time and catch errors quickly, then you have solid preparation for high creativity.”
In the end, the purpose is to arrive at questions quickly, test them, and if they fail, discard the query. And start over, quickly. Intuition can be a guide to a question, but it is analysis which tests the premise, pass or fail.
Gauging the limits of intuition may still come down to the team member’s personality or academic background, noted Scott Gnau, chief technology officer at Hortonworks.
“The world is full of really great data scientists,” Gnau said. But you can’t “create” a person with a solid work ethic or intuition: “Some people are more that way. Some people are less that way.