Big data goes beyond volume, variety, and velocity alone. You need to know these 10 characteristics and properties of big data to prepare for both the challenges and advantages of big data initiatives.
The term big data started to show up sparingly in the early 1990s, and its prevalence and importance increased exponentially as years passed. Nowadays big data is often seen as integral to a company's data strategy.
Big data has specific characteristics and properties that can help you understand both the challenges and advantages of big data initiatives.
You may have heard of the three Vs of big data, but I believe there are seven additional important characteristics you need to know. Conveniently, these properties each start with v as well, so let's discuss the 10 Vs of big data.
Volume is probably the best known characteristic of big data; this is no surprise, considering more than 90 percent of all today's data was created in the past couple of years. The current amount of data can actually be quite staggering. Here are some examples:
-- 300 hours of video are uploaded to YouTube every minute.
-- An estimated 1.1 trillion photos were taken in 2016, and that number is projected to rise by 9 percent in 2017. As the same photo usually has multiple instances stored across different devices, photo or document sharing services as well as social media services, the total number of photos stored is also expected to grow from 3.9 trillion in 2016 to 4.7 trillion in 2017.
-- In 2016 estimated global mobile traffic amounted for 6.2 exabytes per month. That's 6.2 billion gigabytes.
Velocity refers to the speed at which data is being generated, produced, created, or refreshed.
Sure, it sounds impressive that Facebook's data warehouse stores upwards of 300 petabytes of data, but the velocity at which new data is created should be taken into account. Facebook claims 600 terabytes of incoming data per day.
Google alone processes on average more than "40,000 search queries every second," which roughly translates to more than 3.5 billion searches per day.
When it comes to big data, we don't only have to handle structured data but also semistructured and mostly unstructured data as well. As you can deduce from the above examples, most big data seems to be unstructured, but besides audio, image, video files, social media updates, and other text formats there are also log files, click data, machine and sensor data, etc.
Variability in big data's context refers to a few different things. One is the number of inconsistencies in the data. These need to be found by anomaly and outlier detection methods in order for any meaningful analytics to occur.
Big data is also variable because of the multitude of data dimensions resulting from multiple disparate data types and sources. Variability can also refer to the inconsistent speed at which big data is loaded into your database.
This is one of the unfortunate characteristics of big data. As any or all of the above properties increase, the veracity (confidence or trust in the data) drops. This is similar to, but not the same as, validity or volatility (see below). Veracity refers more to the provenance or reliability of the data source, its context, and how meaningful it is to the analysis based on it.
For example, consider a data set of statistics on what people purchase at restaurants and these items' prices over the past five years.
