Overview

This guide explains how to use the AI-DAPT Data Harvester to ingest datasets, manage schedules, and monitor ingestion jobs.

Key Concepts

Dataset

A data asset ingested into the platform (file-based, API-based, or database-based). Raw objects are persisted in MinIO/S3 storage, while metadata is stored in PostgreSQL.

Job

A single ingestion execution, including status/progress tracking.

Schedule

A recurring ingestion definition (API ingest or DB ingest) executed by the built-in scheduler.

Metadata

Dataset-level information stored in PostgreSQL and optionally published to the AI-DAPT Data Documentation Engine (DDE) as DCAT/RDF.

Capabilities

The Data Harvester supports:

  • Ingestion from local files (CSV, XLSX, TXT, JSON, Parquet)

  • Ingestion from HTTP APIs (JSON or raw file)

  • Ingestion from databases (PostgreSQL / MySQL)

  • MinIO/S3-compatible object storage persistence

  • Jobs and schedules management (APScheduler)

  • Optional Keycloak authentication with Redis token caching

  • Optional DDE integration for catalog publication (DCAT/RDF via piveau hub-repo)

Where to start

Proceed to the Tutorial for the end-to-end workflow:

  • Login (when Keycloak is enabled)

  • Upload a file

  • Ingest from an API

  • Ingest from a database

  • Manage files and schedules

  • Monitor job status