Welcome to Data Harvester Documentation
What is the Data Harvester?
The AI-DAPT Data Harvester is the data ingestion entry point of the AI-DAPT platform. It provides:
-
A unified backend API (FastAPI/Python) and a Vue-based UI
-
Dataset ingestion from multiple sources
-
Raw object persistence in MinIO/S3-compatible object storage
-
Operational metadata storage for jobs, schedules and dataset metadata
-
Optional Keycloak authentication (with Redis caching)
-
Optional integration with the Data Documentation Engine (DDE) for DCAT/RDF publishing
Supported Ingestion Sources
-
Local files: CSV, XLSX, TXT, JSON, Parquet
-
HTTP APIs: JSON and raw file downloads
-
Databases: PostgreSQL / MySQL
Service URL
The Data Harvester UI is available at: https://data-harvester-ui.aidapt.s5projects.eu/assets/datasets
Documentation
-
User Guide – for end users and operators
-
Developer Guide – for developers and integrators