Welcome to AI and Data Pipeline Manager (DAVE)
The DAVE framework comprises of the following components, as seen in Fig.1: - The Apache Airflow® component (1) - A set of Operators (2) - A set of supporting technologies (3) - User Inteface (4)
DAVE makes use of the Apache Airflow® workflow orchestrator to schedule, execute and monitor tasks and workflows, and supports integration with several open-source, scalable AI and Big Data technologies (eg. Apache Spark, Dask, Koalas, etc.). Airflow defines workflows as DAGs (Directed Acyclic Graphs), a type of data structure with nodes and edges, in which each node represents a task or process, that are interconnected by edges, creating dependecies between nodes that enable the flow of data throughout the workflow.
The tasks that are run in each node of the DAG are blocks of code we call Operators. These Operators are atomic functions that are divided in three categories, according to their functionality in the AI/ML pipeline: Connectors, Transformers and Analytics. Connector Operators are the operators that allow the connection to data sources and collection of data from these data sources. Transformers Operators are comprised of the group of operators that enable the preparation of the data for the application in AI/ML tasks. Analytics Operators are operators that provide Machine Learning algorithms. They provide functionalities for both training of AI/ML models, as well as using the trained models for prediction, classification, etc.
To use these operators, it is necessary to have supporting technologies that enable the operators to function. As such, DAVE comes prepacked with several Big Data Technologies such as Apache Spark and Apache Kafka, that enable the ingestion of high volumes of data through batch ingestion or streams of data, Koalas and Dask enable the pre-processing and transformation of large volumes of data using distributed computing, while technologies like SciKitLearn provide a variety of AI and Machine Learning algorithms to be used in the Analytics operators. MLFlow is used as a model registry to store the trained models, as well as to load them for use in operations.
To enable the easy use of the tool, DAVE provides a User Inteface that allows the users the creation of data-driven pipelines as executable workflows through visual programing. The user selects the operators that will be used in the workflow via drag-and-drop of the operator blocks into the workspace canvas. In the canvas, the user can then configure these operators as needed. The User Interface provides the following features:
-
Creation of Data Analytics Workflows: Visual programming of AI workflows, by using the drag and drop functionality to add operators and setting up dependencies between operators.
-
Save and Load Workflows: Allows the saving of workflows in a database, so that it can be loaded and executed anytime, and allowing the user to make changes to previously done workflows.
-
Creation and integration of new operators: The creation of user specific operators, through the help of guidelines, and integration of those specific on the workflow creation tool.
-
Configuration of external tools and technologies: Full integration with tools and technologies, like Keras, Tensorflow, Sklearn, allowing an all-in-one box setting.
-
Execution and orchestration of workflows: Easy plug’n’play of user created workflows directly on the Airflow interface, allowing the execution and orchestration of multiple workflows.
Operator Types
The DAVE operators are divided into three types (Connectors, Transformers and Analytics), based on their function in the workflow pipeline. These categories were defined based on the three distinct steps in the training and usage of AI/ML models. They are, in order: 1. The collection of the data to be used. Connector operators exist to collect the data and load it for the next step in the pipeline. 2. The cleaning and transformation of the data. Sometimes the dataset used contains data that in not necessary, or data that needs to be cleaned before being used. To do this, the Transformer operators provide a set of functions that allow the user to filter, clean, normalize the data, join multiple datasets, etc. 3. The application of data analytics algorithms, whether it’s to train a new model, or to use an existing model to make predictions, classification, etc. on a dataset. This is handled by the Analytics operators, that provide the user with a range of AI/ML algorithms to train a model or to apply a trained model on the data.