The Levels of Doing AI
When it comes to new technologies like Artificial Intelligence, the pure technology is only a small aspect required to putting it to use. Still, given the hype that exists currently, one can easily loose sight of the big picture as announcements of new algorithms, toolboxes, or cloud services fight for grabbing our attention as the next game changer. The field itself can also feel quite overwhelming at times, as it is always expanding and keeping track of all the developments seems impossible for one individual to do.
There was a time when a single individual could cover all the bits and pieces, but given the explosive expansion that this field has been going through these past years, I think it is becoming more and more challenging. So this is my attempt to draw a bigger picture of how different roles and levels fit together to “do AI” in an enterprise setting.
So, let’s start at the very bottom, and slowly work our way up in the hierarchy.
At the very lowest level you are using some programming language to actually write code. Most people right now use Python when it comes to data analysis, but there are other contenders like Julia. This also the level where you’ll find the libraries like tensorflow or pytorch that are frequently used to implement ML algorithms.
With code, you put together machine learning or artificial intelligence algorithms (don’t ask me about the difference). Almost all of the research in ML works on figuring out new learning algorithms. There is tremendous depth to this work as you draw upon mathematics, physics, biology, optimization theory, differential equations, statistics and many other techniques to create new algorithms. Essentially enough to keep you occupied a lifetime.
Using the AI/ML methods, you write programs that provide some functionality like click-through probability prediction, computer vision, or document classification. This could be a program that is run once per night, extracting data, training a model, computing predictions and store them somewhere to be picked up. Or the program deploys a model you can query via an API. This is the part where the productionizing (is that even a word?) of AI/ML happens. Here, you’ll find both machine learning experts and data scientists, but increasingly also data engineers, and you work with tools like Apache Spark that provide infrastructure for data processing.
You design these systems usually not just for the fun of it (although that is a good reason, too), but to create a product that solve a problems for the customer. This is the domain of product managers that have to make the connection between what the customer wants or needs and how to use AI/ML technology to solve the problem. At this level, the question whether the whole effort is worth is financially also starts to become more important. After all, maybe a good enough solution does not involve AI/ML at all.
