External Operator

The external operator is used when a task needs to be performed in a workflow, but the functionality is not present in the form of a regular internal operator (such as a simple connector, transformer or analytic), and instead, it is provided from an independent source, such as an application or service. This operator serves as a bridge that connects that external service to the workflow, without having to go back and forward between applications.

The main communication between the DAVE platform and external tools is done through configuration files, which contain all the external service required parameters in order for it to complete a task. These configuration files are normally stored in an external database, such as a configuration storage database for example, and the external operator has real-time access to that database, which provides a seamless interaction between both applications.

The current external operators can be divided into two main groups, which are closely connected to two definitions related to the role of the human in these processes: Human-in-the-loop and Human-on-the-loop. The corresponding operators can be defined as follows:

  • Human-in-the-loop external operator - Human interaction is required during the pipeline execution, blocking further tasks from being completed until the action is performed. This type of operator uses the Python Sensor, a type of Airflow Operator, to achieve this functionality. In this case, the operator will wait for a human interaction in an external User Interface to happen before ending its execution. This is done by continuously observing a specific flag that marks the human interaction via external UI as completed.

  • Human-on-the-loop external operator - Human interaction occurs before the execution of the pipeline and is reflected in a configuration file that is used during workflow execution. These operators use the Python Operator to perform the corresponding tasks, with the task itself executed in the external service accessed through an API call. This request waits for a response from the external service, either success or failure, before proceeding to the next task. The API call contains all required parameters, including the pre-created configuration file and connection parameters so that the external service can access data from the pipeline’s intermediary storage.

This documentation contains information on how to create and add the two types of external operators mentioned above. This will be demonstrated using two types of Airflow operators: the Python Operator and the Python Sensor.