Data Wrangling is considered as the most important process that every data scientists pass through. However, it is one of the most time consuming process and it is one of the key solutions used for decision making promptly. Breaking it down; what is Data Wrangling in simple terms? and we will learn more about data wrangling with pandas in this article.
Fasten your seat belts without any further due lets get started!!
It is often the case with data science projects that you’ll have to deal with messy or incomplete data. The raw data we obtain from different data sources is often unusable at the beginning. The activity that you do on the raw data to make it “clean” enough to input to your analytical algorithm is called data wrangling or data munging. If you want to create beautiful data visualizations, you should be prepared to do a lot of data wrangling.
As most statisticians, data analysts and data scientists will admit, most of the time spent implementing an analysis is devoted to cleaning the data itself, rather than to coding or running a particular model that uses the data. According to O’Reilly’s 2016 Data Science Salary Survey, 69% of data scientists will spend a significant amount of time in their day-to-day dealing with basic exploratory data analysis, while 53% spend time cleaning their data. Data wrangling is an essential part of the data science role — and if you gain data wrangling skills and become proficient at it, you’ll quickly be recognized as somebody who can contribute to cutting-edge data science work and who can hold their own as a data professional.
To sum it all up: take messy, incomplete data or data that is too complex and clean it so that it’s useable for analysis.
Now we’ll illustrate with the help of examples about some popular Pandas techniques that are used to make the data wrangling process easier.
Pandas is one of the most popular Python library for data wrangling. We’ll be playing with Pandas dataframes which are structured as tables where you can use Python code to easily manipulate the rows and columns.
You can get started with Pandas by reading Julia Evans’ Pandas Cookbook and by understanding the basics of Python.
We’ll use the ‘train.csv’ provided for Kaggle’s Titanic: Machine Learning from Disaster as our experimental dataset. You can download it and follow along if you’d like.
Let’s read the data as a Pandas dataframe and explore it.
The data that you’ll work with will have some missing data points. It is really important to understand to ‘how to deal with missing data?’. As you learn more data science, you’ll learn about data imputation. Here, we’ll learn to find missing data points and then we’ll drop those points from the dataset so as not to affect our analysis with bias. We’ll try to find which columns in ‘train.csv’ contain missing values and drop those missing values so you’ll have tidy data.
As we can see from the output, the columns ‘Age’, ‘Cabin’ and ‘Embarked’ contain missing or null values as they’ve indicated that the statement if there are any null values in the column is TRUE. We’d like to drop all the rows with missing values.
While printing out of the first five values in the dataframe, there are some null values in the “Cabin” column.
