AI-DAPT Reference Architecture

Fetches the required data at rest (batch files) or data in motion (near real-time data).

Performs semantic annotations at feature structure level, mapping them to an appropriate data model supported in the Semantics Reconciliation Manager, and applies any harmonization/transformation needed on feature values.

Applies appropriate user-defined and/or automated ML-based cleaning rules.

Responsible for storing and managing metadata at dataset and feature levels, acting also as the data asset catalogue of the platform.

Serves the platform with cross-domain data models, ensuring integrity, validity, and adaptability to evolving stakeholder needs.

Helps users with feature engineering during data preparation for AI and analytics, tailoring features for model training to improve model performance.

Executes pipelines while guaranteeing performance requirements. Continuously monitors pipeline performance and execution status, stores results and metrics in the AI-DAPT Scalable Storage Services, and allocates required resources using the DAG generated by the Pipeline Manager.

Provides an environment for interactive development and early experimentation using computational notebooks.

Assists in the design, configuration, and management of both data-driven and AI-focused pipelines. Offers a collection of prebuilt data transformation functions, integrates with other AI-DAPT services to improve data processing, and features a catalogue of AI/ML algorithms for model training and deployment.

Assesses data quality and bias.

Allows controlled generation and evaluation of synthetic data using generative AI models.

Monitors data pipeline execution, detects anomalies, and raises alerts.

Tracks AI model performance over time, identifies drift, and recommends retraining or adaptive AI techniques.

Helps users ensure the robustness and performance of production models through human-in-the-loop strategies. Integrates model and data observability metrics with user-defined rules to generate relevant alerts and recommendations that may trigger model retraining or rebuilding, pending human review and approval.

Enables users to track, compare, and optimize model training experiments.

Identifies and mitigates potential adversarial attacks on AI models.

Provides a range of methods aimed at improving the understanding of trained hybrid AI models, allowing users to gain a clearer view of how the models operate internally.

Ensures secure access control.

Manages user-defined alerts and event notifications.

Facilitates secure extraction and accessibility of requested data.

Enables seamless integration with external AI-DAPT-compliant solutions.