Data quality for big data should include a focus on usability

Data quality for big data should include a focus on usability

Data quality processes have become more prominent in organizations, often as part of data governance programs....

For many companies, the growing interest in quality is commensurate with an increased need to ensure that analytics data is trustworthy. That's especially true with data quality for big data; more data usually means more data problems.

One of the main challenges of effective data quality management is articulating what quality really means to a company. What are commonly referred to as the dimensions of data quality include accuracy, consistency, timeliness and conformity. But there are many different lists of dimensions, and even some common terms have different meanings from list to list. As a result, solely relying on a particular list without having an underlying foundation for what you're looking to accomplish is a simplistic approach to data quality.

This challenge becomes more acute with big data. In Hadoop clusters and other big data systems, data volumes are exploding, and data variety is increasing. An organization might accumulate data from numerous sources for analysis -- for example, transaction data from different internal systems, clickstream logs from e-commerce sites and streams of data from social networks.

Additionally, the design of big data platforms exacerbates the potential problems. A company might create data in on-premises servers, syndicate it to cloud databases and distribute filtered data sets to systems at remote sites. This new world creates issues that aren't covered in conventional lists of data quality dimensions. We need to re-examine what is meant by quality in the context of a big data analytics environment. To compensate, we need to re-examine what is meant by quality in the context of a big data analytics environment. Too often, we equate the concept of data quality with discrete notions such as data correctness or currency, putting in place processes to fix data values or objects that aren't accurate or up to date. But managing data quality for big data is also likely to include measures designed to help data scientists and other analysts figure out how to effectively use what we have. In other words, we must transition from simply generating a black-and-white specification of good versus bad data to supporting a spectrum of data usability.

Share it:
Share it:

[Social9_Share class=”s9-widget-wrapper”]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

You Might Be Interested In

Using customer data platforms to monetize information

27 Feb, 2018

In my last post, I suggested that while we are in the era of big data, our core understandings of …

Read more

The Big Challenge for Data Science in Investment Banking

18 Oct, 2017

Investment banking institutions have been much slower to embrace data science techniques when compared to their retail counterparts who regularly …

Read more

Narrative Science: The Leader in Natural Language Generation Technology

14 Jun, 2018

Narrative Science helps people understand and communicate what is most important in their data. By transforming data into insightful, human-like …

Read more

Recent Jobs

Senior Cloud Engineer (AWS, Snowflake)

Remote (United States (Nationwide))

9 May, 2024

Read More

IT Engineer

Washington D.C., DC, USA

1 May, 2024

Read More

Data Engineer

Washington D.C., DC, USA

1 May, 2024

Read More

Applications Developer

Washington D.C., DC, USA

1 May, 2024

Read More

Do You Want to Share Your Story?

Bring your insights on Data, Visualization, Innovation or Business Agility to our community. Let them learn from your experience.

Get the 3 STEPS

To Drive Analytics Adoption
And manage change

3-steps-to-drive-analytics-adoption

Get Access to Event Discounts

Switch your 7wData account from Subscriber to Event Discount Member by clicking the button below and get access to event discounts. Learn & Grow together with us in a more profitable way!

Get Access to Event Discounts

Create a 7wData account and get access to event discounts. Learn & Grow together with us in a more profitable way!

Don't miss Out!

Stay in touch and receive in depth articles, guides, news & commentary of all things data.