Data Cleaning Engine
The Data Cleaning Engine is a component of the AI-DAPT pipeline that detects and repairs data quality issues during the Data Design for AI phase.
It helps users prepare datasets for downstream analytics and machine learning by providing robust methods for data imputation and anomaly (error) detection, with a focus on both classical and learning-based techniques.
What the Engine Does
The engine supports two core data cleaning tasks:
-
Data Imputation Replaces missing (
NaN) values in a dataset while preserving its original structure. -
Anomaly (Error) Detection Identifies erroneous or anomalous values and replaces them with
NaN, allowing subsequent processing steps to handle them explicitly.
Both tasks are exposed through a unified interface, enabling users to switch between methods with minimal changes to their code.
Supported Data Types
The engine operates on two main data types:
-
Time Series Data Datasets indexed by time, typically consisting of a datetime column and one or more numeric variables.
-
Tabular Data General-purpose datasets with mixed column types. Support for tabular data is under active development, and available methods are documented per task.
Available Methods
The engine provides a wide range of methods, from simple statistical techniques to advanced deep-learning models.
An overview of supported methods per task and data type is provided below. Detailed documentation for each method, including parameters and usage examples, is available in the corresponding sections.
Data Imputation
| Method | Tabular | Time Series |
|---|---|---|
backward_forward_fill |
✓ |
✓ |
mean_fill |
✓ |
✓ |
median_fill |
✓ |
✓ |
mode_fill |
✓ |
✓ |
linear_interp_fill |
✓ |
✓ |
polynomial_interp_fill |
✓ |
✓ |
spline_interp_fill |
✓ |
✓ |
time_fill |
✓ |
|
nearest_fill |
✓ |
✓ |
pad_fill |
✓ |
✓ |
saits |
✓ |
|
brits |
✓ |
|
usgan |
✓ |
|
mrnn |
✓ |
|
gpvae |
✓ |
|
timesnet |
✓ |
|
nonstationary_transformer |
✓ |
|
autoformer |
✓ |
|
units |
✓ |
|
moment |
✓ |
|
e2e_ppg |
✓ |
Anomaly (Error) Detection
| Method | Tabular | Time Series |
|---|---|---|
matrix_profile |
✓ |
✓ |
zscore_outlier_detection |
✓ |
✓ |
iqr_outlier_detection |
✓ |
✓ |
isolation_forest |
✓ |
✓ |
angle_base_outlier_detection |
✓ |
|
clustering_based_local_outlier |
✓ |
|
connectivity_based_local_outlier |
✓ |
|
histogram_based_outlier_detection |
✓ |
|
k_nearest_neighbors_detector |
✓ |
|
local_outlier_factor |
✓ |
|
one_class_svm_detector |
✓ |
|
principal_component_analysis |
✓ |
|
minimum_covariance_determinant |
✓ |
|
subspace_outlier_detection |
✓ |
|
stochastic_outlier_selection |
✓ |
|
moment |
✓ |
|
e2e_ppg |
✓ |
Input and Output
-
Input
-
Time series: a CSV file with a datetime column and one or more numeric columns
-
Tabular data: a dataset with mixed column types
-
-
Output
-
A cleaned DataFrame with the same shape as the input
-
A log describing the cleaning operations performed
-
-
Note In the current implementation, the name of the datetime column must be explicitly provided for time series data.
How to Use This Guide
This User Guide is organized by task and data type:
-
Start with the Tutorial for a hands-on introduction.
-
Explore task-specific sections:
-
Refer to individual method pages for detailed parameter descriptions and examples.
Next Steps
-
Follow the Tutorial
-
Select a task and data type relevant to your dataset
-
Consult individual method pages to choose and configure an appropriate technique