User Guide Overview
The Data Documentation Engine is the central component in AIDAPT platform responsible for storing and maintaining metadata of datasets used for building AI pipelines. It is built with the open source solution piveau. It employs a triplestore as a primary database and offers first-class support for metadata in DCAT and RDF format.
Getting Started
Start with the Tutorial.
Key Concepts
The Data Documentation Engine has a very simple data structure:
-
On the highest level you have catalogues. For the alpha release, we will have one catalogue and metadata of all datasets can be stored in here.
-
Each catalogue can contain datasets or AI assets. This will have all metadata information and reference to the actual dataset stored in the Scalable Storage Services.
-
Each dataset holds distributions. A distribution represents the actual data/asset that is available in the Scalable Storage Services - aka hold a reference to the data/asset. A dataset can have multiple distributions, we can store some common metadata for all distributions and specific metadata for each distribution, such as the data schema of a distribution in the Scalable Storage Services.vices.
-
In addition to basic metadata, we can also store additional data lineage information such as who created a dataset and when.