Data Cleaning and Data Wrangling

- Overview

Businesses have long relied on professionals with data science and analytical skills to understand and leverage information at their disposal. With the proliferation of data, due to the development of smart devices and other technological advancements, this need has accelerated.  

Data structuring is the process of organizing data into a consistent format so it can be used across tools and systems. This can involve normalizing data, denormalizing data, and reformatting data fields.

Data cleaning is the process of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete data within a dataset. When combining multiple data sources, there are many opportunities for data to be duplicated or mislabeled. 

Data wrangling helps in manipulating records to transform them into the desired format. In contrast, data cleaning helps eliminate and fix inconsistencies in the data to make it reliable and consistent for analysis.

Please refer to refer to the following for more infomrmation:

• Wikipedia: Data Cleaning
• Wikipedia: Data Wrangling

- Data Cleaning and Data Wrangling

Data cleaning and data wrangling (or data munging) are both essential steps in the data process, but they have different purposes:

• Data cleaning: Ensures data is accurate and consistent by identifying and fixing errors. Data cleaning is also known as data cleansing or scrubbing.
• Data wrangling: Prepares data from various sources for analysis by organizing it and making it usable.

• Data cleaning involves: Inspecting data, Removing duplicates, Handling missing data, Filtering outliers, Standardizing data, and Verifying data.
• Data wrangling involves: Data discovery, Data structuring, Enriching data, Validating data, and Publishing data.

- Data Cleaning

Data cleaning is the process of fixing or removing inaccurate, incomplete, or irrelevant data from a dataset. 

Data cleaning is important because it makes information more accessible, easier to understand, and more accurate. It's also the foundation for data analysis.

Data cleaning prepares raw data for machine learning (ML) and business intelligence (BI) applications. Clean data helps teams make better decisions and build trust within an organization. 

Data cleaning involves: 

• Detecting incorrect, incomplete, or inaccurate parts of the data
• Replacing, modifying, or deleting the affected data
• Identifying and removing duplicate information and unrelated data
• Correcting formatting, missing values, and spelling errors

Some data cleaning tools include RingLead, SAS Data Quality, Oracle Enterprise Data Quality, Informatica, Melissa Clean Suite, Xplenty, Tibco Clarity, and Data Ladder.

Before cleaning data, it's a good idea to keep a copy of the raw data set. It's also important to establish a template for the data cleaning process so it can be done consistently.

[Butchart Gardens, Vancouver, Canada]

- Examples of Data Cleaning

Data cleaning is the process of changing data from its original format to make it more usable and accurate. 

Examples of data cleaning include handling missing data, removing duplicates, correcting inconsistencies, standardizing formats, and validating accuracy. You might use techniques like SQL commands to perform these tasks.

Examples of data cleaning include: 

• Removing duplicates: Duplicate records can cause reporting errors and increase maintenance costs.
• Handling missing values: Missing information can be populated.
• Fixing errors: Statistical and database methods can help detect and fix errors.
• Filtering irrelevant data: Data that is inaccurate, irrelevant, or not formatted correctly can be removed or changed.
• Converting data types: Data can be converted to a different type.
• Creating calculations: Calculations can be created based on existing values.
• Ensuring consistency: Data should be consistent overall.
• Formatting: Data should be formatted clearly and simply.
• Keeping data unified: Data should be kept in a unified form.

- Data Wrangling

It’s impossible to choose a single data science skill that’s most important for professionals. One thing that's certain, however, is that insights are only as good as the data that informs them. This means it’s vital for organizations to employ individuals who understand what clean data looks like and how to shape raw data into usable forms to gain valuable insights. This is where data wrangling comes into play.

Data wrangling (or data munging) is the process of transforming raw data into a more usable format for analysis or machine learning (ML). It's also known as data munging, scrubbing, or remediation.

Data wrangling involves: cleaning and structuring data, handling missing or inconsistent data, formatting data types, and merging different datasets.

The goal of data wrangling is to improve data quality and make it more accurate and meaningful. This leads to better solutions, decisions, and outcomes.

- Examples of Data Wrangling

Data wrangling is important for ensuring data is processed and managed according to legal and ethical standards. It helps organizations manage their data governance policies and reduce the risk of compliance issues.

Data wrangling can be conducted manually or automatically. In businesses with a data team, data scientists and other team members usually lead the data wrangling process. In smaller organizations, non-data professionals may be responsible for cleaning data.

Some examples of data wrangling include:

• Merging multiple data sources into a single dataset
• Identifying gaps in data and filling or deleting them
• Deleting data that's unnecessary or irrelevant
• Identifying extreme outliers in data and either explaining the discrepancies or removing them