Tabular Anomaly Detection Methods

This page provides an overview of the anomaly detection methods available in the AI-DAPT Data Cleaning Engine for tabular data.

Each method identifies anomalous or erroneous values in a DataFrame and replaces them with NaN, enabling explicit handling in subsequent processing steps.

All methods operate on numerical columns.

matrix_profile

Detects anomalies using matrix profiling and masks anomalous windows.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • perc_threshold (int, optional) Percentile threshold for anomaly detection. Default: 95

  • window_size (int, optional) Sliding window size. Default: 5

zscore_outlier_detection

Detects and masks outliers using Z-score analysis.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • min_threshold (float, optional) Minimum Z-score threshold for outlier detection. Default: -3.0

  • max_threshold (float, optional) Maximum Z-score threshold for outlier detection. Default: 3.0

iqr_outlier_detection

Detects and masks outliers using the Interquartile Range (IQR) method.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

isolation_forest

Detects anomalies by isolating observations using random partitioning.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

angle_base_outlier_detection

Identifies outliers based on variance in angles between data points.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

clustering_based_local_outlier

Flags outliers by analyzing distances from cluster centroids.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

connectivity_based_local_outlier

Detects anomalies by measuring connectivity and density deviations.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

histogram_based_outlier_detection

Identifies outliers through deviations in histogram density.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

k_nearest_neighbors_detector

Detects anomalies using distances to k-nearest neighbors.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

local_outlier_factor

Evaluates local density deviation to identify outliers.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

one_class_svm_detector

Learns a decision boundary to separate normal data from anomalies.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

principal_component_analysis

Flags outliers based on projection errors in principal component space.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

minimum_covariance_determinant

Detects anomalies using robust covariance estimation.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

subspace_outlier_detection

Identifies anomalies within lower-dimensional feature subspaces.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1

stochastic_outlier_selection

Uses probabilistic selection to assign outlier scores.

Parameters

  • df (pd.DataFrame) Input DataFrame containing numerical values.

  • fraction (float, optional) Expected proportion of outliers in the data. Default: 0.1