Configuration

This section documents the configuration surface of the Data Harvester stack, focusing on the environment variables provided via docker-compose.yml.

Configuration domains

Configuration is grouped into:

  • Backend service configuration (FastAPI)

  • Frontend configuration (Vite environment)

  • Infrastructure configuration (PostgreSQL, MinIO, Redis)

  • Optional integrations (Keycloak, DDE)

Backend configuration (conceptual)

The backend is configured primarily through environment variables injected by Docker Compose.

Common configuration areas:

  • PostgreSQL connection (jobs, schedules, metadata)

  • MinIO/S3 endpoint, access key, secret key, bucket(s)

  • Auth enablement (Keycloak on/off)

  • Redis enablement and connection (token/public key caching)

  • DDE enablement and connection (hub-repo endpoint + API key)

Keep secrets (DB passwords, MinIO secret keys, Keycloak client secrets, DDE API keys) out of Git.

Frontend configuration (Vite)

Example .env values:

# Enable or disable Keycloak auth
VITE_USE_KEYCLOAK=true

# Keycloak config
VITE_KEYCLOAK_URL=https://auth.yourdomain.com/auth
VITE_KEYCLOAK_REALM=your-realm
VITE_KEYCLOAK_CLIENT_ID=your-client-id

# Backend API URL
VITE_API_BASE_URL=https://api.yourdomain.com

# Base URL (landing page)
VITE_BASE_URL=

Auth mode configuration

When DH_USE_KEYCLOAK=true:

  1. The frontend obtains a user JWT from Keycloak.

  2. Frontend calls backend with Authorization: Bearer <user_token>.

  3. Backend validates token using Keycloak realm keys (JWKS/certs), optionally cached via Redis.

  4. Routers enforce ownership isolation (files, jobs, schedules).

If Keycloak is disabled, deployments commonly run in a simplified "no-auth" mode suitable for local development.

DDE configuration (optional)

If DDE publishing is enabled, the backend:

  • builds DCAT/RDF Turtle metadata representations

  • publishes to the DDE hub-repo endpoint using the configured API key

This is typically used to expose datasets into a shared catalog (DCAT/RDF) for discoverability and interoperability.

See Also