Personal tools
 

Data Preparation

Harvard (Charles River) IMG 7718
(Harvard University - Harvard Taiwan Student Association)

- Overview

Data preparation is the process of cleaning and transforming raw data before processing and analyzing it. Data preparation techniques are typically used at the earliest stages of the ML and AI development pipeline to ensure accurate results. 

Data preparation is a key step in data analytics projects. It can involve many tasks, such as:

  • Collecting data
  • Cleaning data
  • Labeling data
  • Reformatting data
  • Making corrections to data
  • Combining datasets
  • Standardizing data formats
  • Enrichment of source data
  • Elimination of outliers

 

Data preparation can help business analysts and data scientists trust, understand, and ask better questions of their data. This can make their analyses and modeling more accurate and meaningful.

 

- Data Preparation Techniques

Data preparation is a crucial step in the machine learning (ML) pipeline. It involves collecting, cleaning, and organizing data before using it to train a model. The quality of the data used to train a model significantly impacts the accuracy of its predictions. 

Here are some data preparation techniques:  

  • Data cleansing: An essential process for preparing raw data for ML. Raw data may contain numerous errors, which can affect the accuracy of ML models.
  • Feature engineering: Involves selecting, extracting, transforming, and creating new features from the available data to improve the performance of ML algorithms.
  • Hyperparameter tuning: An essential part of the ML process that involves optimizing the model's performance by fine-tuning its hyperparameters.
  • Transform data files: Transform all the data files into a common format.
  • Explore the dataset: Use a data preparation tool like Tableau, Python Pandas, etc. to explore the dataset.
  • Pick feature variables: Use feature selection methods to pick feature variables from the dataset.

 

 - Data Preparation for ML

Data fuels ML. Leveraging this data to reshape your business, while challenging, is critical to staying relevant now and into the future. This is the survival of the most informed people, those who can use their data to make better, more informed decisions can react faster to unexpected events and uncover new opportunities. This important but tedious process is a prerequisite for building accurate ML models and analyses, and is the most time-consuming part of an ML project. 

To minimize time investment, data scientists have access to tools that help automate data preparation in various ways.

Data preparation tools include:

  • Microsoft Power BI
  • Tableau
  • Alteryx AI Platform
  • Trifacta Wrangler Enterprise
  • Altair Monarch
  • Talend Data Fabric
  • MicroStrategy

 

 

[More to come ...]

 

 

 
Document Actions