Developer Guide Overview

This is the technical documentation of Data Documentation Engine for developers. See Architecture Diagram below for more details.

Architecture

architecture

Hub-repo

The hub-repo acts as a middleware and abstraction layer to interact with the Triplestore. It offers a RESTful interface, supporting the major RDF serializations (Turtle, JSON-LD, N-Triples, RDF/XML, Notation3). The main task is to process and harmonize RDF data. This includes the application of consistent and meaningful URI schemata, the generation of unique IDs, and the mapping to linked, existing entities. It ensures the integrity and traceability of the data in the Triplestore. This is our entry point into the Data Documentation Engine. All POST/PUT/DELETE operations must be done in this service. This is our ground truth. GET requests to this service will return datasets in RDF format, for JSON output, use hub-search.

Hub-search

The search service is responsible for encapsulating the communication between hub-repo and search engine (Elasticsearch) for enabling full-text search. The main functionality is searching indexed datasets and catalogues. This service can be used to fetch datasets in JSON format. GET requests can be performed here. The database behind this service is not stable, so no POST/PUT/DELETE operations should be performed on this API.

Lineage Tracker

The Lineage tracker is responsible for tracking the lineage of datasets. It records every dataset create/update/delete operation on a dataset, allowing users to track the lineage and family tree of any dataset. It captures details such as who performed an operation, what changed, and when that change occurred.

Metadata Generation

This is an LLM based backend engine that generates metadata of a dataset. For AIDAPT, we will start with generating the title and description of a dataset.

Metadata Schema

Data Documentation Engine supports the metadata schema based on DCAT-AP with addition of custom properties to support the AIDAPT use case. The main entities are Catalogues, Datasets and Distributions. Each entity has its own set of properties, but they also share some common properties such as title, description, creator, issued date and modified date.

Getting Started

About RDF

The Data Documentation Engine is based upon the Resource Description Framework (RDF). So the API only accepts payload in a RDF format. It supports a variety of these formats, but the mostly commonly used are Turtle and JSON-LD. Learning about RDF is helpful before using the Data Documentation Engine API. The schema used in AIDAPT is based on DCAT-AP with addition of custom properties to support the AIDAPT use case. More information about it can be found here: https://semiceu.github.io/DCAT-AP/releases/3.0.0/

Access control

Hub repo can only be authenticated using an API-Key. Every POST/PUT/DELETE requests should have this API-Key. You can obtain it by contacting Fraunhofer FOKUS. GET requests to hub search and lineage tracker can be authenticated with a Bearer token.

Metadata Schema Reference

This section describes all supported metadata properties for each entity type. Properties are grouped into standard DCAT-AP properties and AIDAPT-specific extension properties. Mandatory properties are marked with (mandatory).

Catalogue

A dcat:Catalog is the top-level container that groups related datasets. Every dataset must belong to a catalogue.

DCAT-AP Properties

Property Obligation Description

dct:title

Mandatory

The human-readable title of the catalogue. Must be a language-tagged literal (e.g. "My Catalogue"@en).

dct:description

Mandatory

A free-text description of the catalogue. Must be a language-tagged literal.

dct:publisher

Mandatory

The organisation or agent responsible for publishing the catalogue. Represented as a foaf:Agent with a foaf:name.

dcat:dataset

Optional

Reference to datasets contained in this catalogue. Managed automatically by the hub-repo.

dct:language

Optional

The language of the catalogue content, as a URI from the EU authority table (e.g. http://publications.europa.eu/resource/authority/language/ENG).

dct:issued

Optional

Date/time the catalogue was first published. Use xsd:dateTime or xsd:date.

dct:modified

Optional

Date/time the catalogue was last modified. Use xsd:dateTime or xsd:date.

foaf:homepage

Optional

A homepage URL for the catalogue.

AIDAPT-Specific Properties

There are no AIDAPT-specific extension properties defined for Catalogue at this time.

Dataset

A dcat:Dataset represents a logical collection of data. It is always associated with a catalogue and may have one or more distributions pointing to the actual data files.

DCAT-AP Properties

Property Obligation Description

dct:title

Mandatory

The human-readable title of the dataset. Must be a language-tagged literal.

dct:description

Mandatory

A textual description of the dataset. Must be a language-tagged literal.

dcat:keyword

Optional

One or more keywords describing the dataset content. Each is a plain string literal.

dct:language

Optional

The language of the dataset content, as a URI from the EU authority table.

dct:creator

Optional

The agent primarily responsible for creating the dataset. Represented as a foaf:Agent.

dcat:contactPoint

Optional

Contact information for the dataset, represented using vcard:Kind.

dct:issued

Optional

Date/time the dataset was first published.

dcat:theme

Optional

A category for the dataset, as a URI from the EU data theme vocabulary (e.g. http://publications.europa.eu/resource/authority/data-theme/EDUC).

dcat:distribution

Optional

One or more links to dcat:Distribution resources describing how to access the data.

odrl:hasPolicy

Optional

An ODRL policy governing the usage of the dataset. See ODRL Policy section.

AIDAPT-Specific Properties

Property Obligation Description

aidapt:pipelineStage

Optional

Identifies the pipeline stage this dataset belongs to within the AIDAPT workflow (e.g. "train", "test", "evaluation").

aidapt:userId

Optional

The ID of the user who created or is associated with this dataset entry.

aidapt:lineageURL

Optional

A URL or IRI pointing to the lineage backend record for this dataset.

Distribution

A dcat:Distribution represents a specific available form of the dataset — in AIDAPT this is typically a file stored in the Scalable Storage Services (MinIO). Each distribution belongs to exactly one dataset.

DCAT-AP Properties

Property Obligation Description

dcat:accessURL

Mandatory

The URL where the distribution can be accessed. In AIDAPT this is typically the MinIO object URL. Stored as an IRI.

dct:license

Optional

The license under which the distribution is made available. Can be a blank node with dct:identifier, dct:title, skos:prefLabel and skos:exactMatch.

dct:format

Optional

The file format as a URI from the EU file type vocabulary (e.g. http://publications.europa.eu/resource/authority/file-type/CSV).

dct:title

Optional

A human-readable title for this distribution, typically the filename.

dct:description

Optional

A description of this specific distribution.

dcat:downloadURL

Optional

A direct download URL for the distribution file.

dct:creator

Optional

The agent who created this distribution.

AIDAPT-Specific Properties

Property Obligation Description

aidapt:minioBucketName

Optional

The name of the MinIO bucket where the data file is stored.

aidapt:minioFileName

Optional

The name of the file within the MinIO bucket.

aidapt:valuationResult

Optional

The result from the Data Valuation Engine for this distribution. Stored as a string or IRI.

aidapt:schema

Optional

The schema of the data file, described using CSVW (CSV on the Web). Contains column definitions with name, ID and data type. See Schema Shape below.

aidapt:sampleData

Optional

Sample rows of the dataset, described using the CSVW row model.

aidapt:dataModel

Optional

A reference to the data model from the Semantics Reconciliation Manager, containing aidapt:dataModelId and aidapt:conceptName.

aidapt:parentDatasetId

Optional

The ID of the dataset this distribution was derived from, for provenance tracking.

aidapt:projectId

Optional

The ID of the AIDAPT project this distribution is associated with.

Schema Shape (CSVW)

The aidapt:schema property uses CSVW vocabulary to describe tabular data structure:

  • csvw:tableSchema — the top-level schema container

    • csvw:column — one entry per column, each with:

      • csvw:columnID — integer index of the column

      • csvw:columnName — string name of the column

      • csvw:dataType — string data type (e.g. "string", "float", "dateTime")

ODRL Policy Shape

The odrl:hasPolicy property on a dataset uses ODRL vocabulary:

  • odrl:Policy — the top-level policy

    • odrl:permission — one or more permissions, each with:

      • dct:identifier — a unique ID for the permission

      • odrl:action — the permitted action (e.g. "use", "distribute")

      • odrl:constraint — optional constraints, each with odrl:leftOperand, odrl:operator, odrl:rightOperand

Catalogues

To begin with, we have to create a catalogue. This request is to hub-repo and can only be authenticated using an API-Key.

Minimal Catalogue

A minimal payload with only mandatory properties looks like this:

PUT https://data-doc-hub-repo.aidapt.s5projects.eu/catalogues/<catalog-id>
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix dct:  <http://purl.org/dc/terms/> .

<https://piveau.io/id/catalogue/<catalog-id>>
    a          dcat:Catalog ;
    dct:title  "AIDAPT Test Catalogue"@en ;
    dct:description  "AI-DAPT pioneers data-centric AI solutions, reshaping industries with innovation and reliability."@en ;
    dct:publisher    <https://piveau.eu/def/publisher> .

Full Catalogue Payload

The following example shows all supported catalogue properties:

PUT https://data-doc-hub-repo.aidapt.s5projects.eu/catalogues/<catalog-id>
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix dct:  <http://purl.org/dc/terms/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix xsd:  <http://www.w3.org/2001/XMLSchema#> .

<https://piveau.io/id/catalogue/aidapt-test-catalogue>
    a                dcat:Catalog ;
    dct:title        "AIDAPT Test Catalogue"@en ;
    dct:description  "AI-DAPT pioneers data-centric AI solutions, reshaping industries with innovation and reliability."@en ;
    dct:publisher    <https://piveau.eu/def/publisher> ;
    dct:language     <http://publications.europa.eu/resource/authority/language/ENG> ;
    dct:issued       "2023-01-18T06:33:36Z"^^xsd:dateTime ;
    dct:modified     "2023-01-18T06:33:36Z"^^xsd:dateTime ;
    foaf:homepage    <https://www.aidapt.eu/> .

<https://piveau.eu/def/publisher>
    a          foaf:Agent ;
    foaf:name  "AIDAPT Consortium" .

If a catalogue exists already, datasets can be added to it. This request will give you a 201 Created status code.

Deleting a catalogue will delete all its datasets.

Datasets

Using the catalogue id, datasets can be created via a POST to hub-repo. The mandatory properties for a dataset are dct:title and dct:description.

Minimal Dataset

The simplest valid dataset requires only a title and description:

POST https://data-doc-hub-repo.aidapt.s5projects.eu/catalogues/<catalog-id>/datasets
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat:   <http://www.w3.org/ns/dcat#> .
@prefix dct:    <http://purl.org/dc/terms/> .

<https://piveau.io/set/data/my-dataset>
    a               dcat:Dataset ;
    dct:title       "My First Dataset"@en ;
    dct:description "A brief description of my dataset."@en .

Dataset with DCAT-AP Properties

This example demonstrates all standard DCAT-AP properties without any AIDAPT extensions:

POST https://data-doc-hub-repo.aidapt.s5projects.eu/catalogues/<catalog-id>/datasets
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat:   <http://www.w3.org/ns/dcat#> .
@prefix dct:    <http://purl.org/dc/terms/> .
@prefix foaf:   <http://xmlns.com/foaf/0.1/> .
@prefix vcard:  <http://www.w3.org/2006/vcard/ns#> .
@prefix xsd:    <http://www.w3.org/2001/XMLSchema#> .

<https://piveau.io/set/data/my-dataset>
    a                   dcat:Dataset ;
    dct:title           "EV Telemetry Dataset"@en ;
    dct:description     "Telemetry readings collected from electric vehicles in a simulated urban environment."@en ;
    dcat:keyword        "electric vehicles"@en, "telemetry"@en, "simulation"@en ;
    dct:language        <http://publications.europa.eu/resource/authority/language/ENG> ;
    dct:issued          "2024-03-01T00:00:00"^^xsd:dateTime ;
    dct:creator         <https://piveau.eu/def/creator> ;
    dcat:contactPoint   <https://piveau.eu/def/contact> ;
    dcat:theme          <http://publications.europa.eu/resource/authority/data-theme/TRAN> ;
    dcat:distribution   <https://piveau.io/set/distribution/ev-telemetry-v1> .

<https://piveau.eu/def/creator>
    a          foaf:Agent ;
    foaf:name  "AIDAPT Data Team" .

<https://piveau.eu/def/contact>
    a                   vcard:Individual ;
    vcard:fn            "AIDAPT Support" ;
    vcard:hasEmail      <mailto:support@aidapt.eu> .

Dataset with AIDAPT-Specific Properties

This example shows only the AIDAPT extension properties added on top of the mandatory fields:

POST https://data-doc-hub-repo.aidapt.s5projects.eu/catalogues/<catalog-id>/datasets
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat:    <http://www.w3.org/ns/dcat#> .
@prefix dct:     <http://purl.org/dc/terms/> .
@prefix aidapt:  <https://www.aidapt.eu/ns/voc#> .

<https://piveau.io/set/data/my-dataset>
    a                        dcat:Dataset ;
    dct:title                "EV Telemetry Dataset"@en ;
    dct:description          "Telemetry readings from electric vehicles."@en ;
    aidapt:pipelineStage     "train" ;
    aidapt:userId            "user-abc-123" ;
    aidapt:lineageURL        <https://lineage-tracker.aidapt.eu/datasets/my-dataset> .

Full Dataset Payload

The following example combines all DCAT-AP and AIDAPT-specific dataset properties, including a distribution:

POST https://data-doc-hub-repo.aidapt.s5projects.eu/catalogues/<catalog-id>/datasets
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat:    <http://www.w3.org/ns/dcat#> .
@prefix dct:     <http://purl.org/dc/terms/> .
@prefix foaf:    <http://xmlns.com/foaf/0.1/> .
@prefix vcard:   <http://www.w3.org/2006/vcard/ns#> .
@prefix skos:    <http://www.w3.org/2004/02/skos/core#> .
@prefix xsd:     <http://www.w3.org/2001/XMLSchema#> .
@prefix odrl:    <http://www.w3.org/ns/odrl/2/> .
@prefix aidapt:  <https://www.aidapt.eu/ns/voc#> .
@prefix csvw:    <http://www.w3.org/ns/csvw#> .
@prefix rdf:     <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .

<https://piveau.io/set/data/aidapt-ev-telemetry>
    a                        dcat:Dataset ;
    dct:title                "EV Telemetry Dataset"@en ;
    dct:description          "Telemetry readings collected from electric vehicles in a simulated urban environment."@en ;
    dcat:keyword             "electric vehicles"@en, "telemetry"@en, "simulation"@en ;
    dct:language             <http://publications.europa.eu/resource/authority/language/ENG> ;
    dct:issued               "2024-03-01T00:00:00"^^xsd:dateTime ;
    dct:creator              <https://piveau.eu/def/creator> ;
    dcat:contactPoint        <https://piveau.eu/def/contact> ;
    dcat:theme               <http://publications.europa.eu/resource/authority/data-theme/TRAN> ;
    dcat:distribution        <https://piveau.io/set/distribution/ev-telemetry-v1> ;
    odrl:hasPolicy           <https://piveau.io/set/policy/ev-telemetry-policy> ;
    aidapt:pipelineStage     "train" ;
    aidapt:userId            "user-abc-123" ;
    aidapt:lineageURL        <https://lineage-tracker.aidapt.eu/datasets/aidapt-ev-telemetry> .

<https://piveau.eu/def/creator>
    a          foaf:Agent ;
    foaf:name  "AIDAPT Data Team" .

<https://piveau.eu/def/contact>
    a               vcard:Individual ;
    vcard:fn        "AIDAPT Support" ;
    vcard:hasEmail  <mailto:support@aidapt.eu> .

<https://piveau.io/set/policy/ev-telemetry-policy>
    a                  odrl:Policy ;
    odrl:permission    <https://piveau.io/set/permission/ev-telemetry-use> .

<https://piveau.io/set/permission/ev-telemetry-use>
    a                  odrl:Permission ;
    dct:identifier     "perm-ev-use-001" ;
    odrl:action        "use" .

<https://piveau.io/set/distribution/ev-telemetry-v1>
    a                        dcat:Distribution ;
    dct:title                "ev_telemetry_train.csv" ;
    dct:description          "Training split of the EV telemetry dataset in CSV format."@en ;
    dcat:accessURL           <https://minio.aidapt.eu/ev-data/ev_telemetry_train.csv> ;
    dcat:downloadURL         <https://minio.aidapt.eu/ev-data/ev_telemetry_train.csv> ;
    dct:format               <http://publications.europa.eu/resource/authority/file-type/CSV> ;
    dct:license              [
                                 dct:identifier   "CC-BY-4.0" ;
                                 dct:title        "Creative Commons Attribution 4.0" ;
                                 skos:prefLabel   "CC BY 4.0" ;
                                 skos:exactMatch  <https://creativecommons.org/licenses/by/4.0/>
                             ] ;
    dct:creator              <https://piveau.eu/def/creator> ;
    aidapt:minioBucketName   "ev-data" ;
    aidapt:minioFileName     "ev_telemetry_train.csv" ;
    aidapt:valuationResult   "0.87" ;
    aidapt:parentDatasetId   "raw-ev-telemetry-2024" ;
    aidapt:projectId         42 ;
    aidapt:dataModel         [
                                 aidapt:dataModelId   "01a13fac-dea2-4db7-bc06-d3b86cf9c755" ;
                                 aidapt:conceptName   "ElectricVehicleTelemetry"
                             ] ;
    aidapt:schema            [
                                 a                  aidapt:CsvwSchema_Shape ;
                                 csvw:tableSchema   [
                                     a              csvw:Schema ;
                                     csvw:column    [
                                         csvw:columnID    1 ;
                                         csvw:columnName  "VehicleID" ;
                                         csvw:dataType    "string"
                                     ] ;
                                     csvw:column    [
                                         csvw:columnID    2 ;
                                         csvw:columnName  "SimulationDate" ;
                                         csvw:dataType    "dateTime"
                                     ] ;
                                     csvw:column    [
                                         csvw:columnID    3 ;
                                         csvw:columnName  "Speed_kmh" ;
                                         csvw:dataType    "float"
                                     ] ;
                                     csvw:column    [
                                         csvw:columnID    4 ;
                                         csvw:columnName  "Battery_pct" ;
                                         csvw:dataType    "float"
                                     ] ;
                                     csvw:column    [
                                         csvw:columnID    5 ;
                                         csvw:columnName  "Temperature_C" ;
                                         csvw:dataType    "float"
                                     ] ;
                                     csvw:column    [
                                         csvw:columnID    6 ;
                                         csvw:columnName  "Brake_pct" ;
                                         csvw:dataType    "float"
                                     ] ;
                                     csvw:column    [
                                         csvw:columnID    7 ;
                                         csvw:columnName  "Status" ;
                                         csvw:dataType    "string"
                                     ]
                                 ]
                             ] ;
    aidapt:sampleData        [
                                 csvw:row  [
                                     csvw:rowID  1 ;
                                     csvw:row    "VH-001,2024-01-15T08:00:00,72.3,85.2,21.4,0.1,ACTIVE"
                                 ] ;
                                 csvw:row  [
                                     csvw:rowID  2 ;
                                     csvw:row    "VH-002,2024-01-15T08:01:00,0.0,91.0,20.8,0.0,IDLE"
                                 ]
                             ] .

Distributions

Distributions can also be added to an existing dataset independently. This allows you to attach multiple file representations (e.g. CSV and Parquet versions) to the same dataset.

Minimal Distribution

A distribution requires at minimum an dcat:accessURL:

POST https://data-doc-hub-repo.aidapt.s5projects.eu/catalogues/<catalog-id>/datasets/<dataset-id>/distributions
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat:   <http://www.w3.org/ns/dcat#> .

<https://piveau.io/set/distribution/my-dist>
    a               dcat:Distribution ;
    dcat:accessURL  <https://minio.aidapt.eu/my-bucket/my-file.csv> .

Distribution with DCAT-AP Properties

POST https://data-doc-hub-repo.aidapt.s5projects.eu/catalogues/<catalog-id>/datasets/<dataset-id>/distributions
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat:   <http://www.w3.org/ns/dcat#> .
@prefix dct:    <http://purl.org/dc/terms/> .
@prefix skos:   <http://www.w3.org/2004/02/skos/core#> .
@prefix foaf:   <http://xmlns.com/foaf/0.1/> .

<https://piveau.io/set/distribution/sensor-data-csv>
    a                dcat:Distribution ;
    dct:title        "sensor_readings.csv" ;
    dct:description  "Sensor readings export in CSV format."@en ;
    dcat:accessURL   <https://minio.aidapt.eu/sensor-bucket/sensor_readings.csv> ;
    dcat:downloadURL <https://minio.aidapt.eu/sensor-bucket/sensor_readings.csv> ;
    dct:format       <http://publications.europa.eu/resource/authority/file-type/CSV> ;
    dct:license      [
                         dct:identifier  "SUB-LI" ;
                         dct:title       "Subscription License" ;
                         skos:prefLabel  "Subscription License" ;
                         skos:exactMatch <https://subscriptionlicense.com>
                     ] ;
    dct:creator      <https://piveau.eu/def/creator> .

<https://piveau.eu/def/creator>
    a          foaf:Agent ;
    foaf:name  "AIDAPT Data Team" .

Distribution with AIDAPT-Specific Properties

This example shows only the AIDAPT extension properties on a distribution:

POST https://data-doc-hub-repo.aidapt.s5projects.eu/catalogues/<catalog-id>/datasets/<dataset-id>/distributions
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat:    <http://www.w3.org/ns/dcat#> .
@prefix dct:     <http://purl.org/dc/terms/> .
@prefix aidapt:  <https://www.aidapt.eu/ns/voc#> .

<https://piveau.io/set/distribution/sensor-data-csv>
    a                        dcat:Distribution ;
    dct:title                "sensor_readings.csv" ;
    dcat:accessURL           <https://minio.aidapt.eu/sensor-bucket/sensor_readings.csv> ;
    aidapt:minioBucketName   "sensor-bucket" ;
    aidapt:minioFileName     "sensor_readings.csv" ;
    aidapt:valuationResult   "0.91" ;
    aidapt:parentDatasetId   "raw-sensor-2024-q1" ;
    aidapt:projectId         7 ;
    aidapt:dataModel         [
                                 aidapt:dataModelId   "f3c29e10-aa41-4e9a-b220-8d4bc1a0e123" ;
                                 aidapt:conceptName   "SensorReading"
                             ] .

Next Steps