Kestra integration

The goal is to integrate Kestra, a modern open-source workflow orchestration tool, with Ilum Data Lakehouse. Despite the existing Airflow integration, Ilum would benefit from a Kestra integration to further streamline workflow orchestration, enable advanced data pipeline automation, and provide users with a robust alternative for managing complex, multi-cluster data workflows. While Airflow is excellent for scheduling and orchestrating tasks, it can become complex and resource-intensive when managing high-frequency, event-driven workflows. Kestra offers a simpler, declarative approach with real-time monitoring, native event triggers, and enhanced data lineage tracking, aligning well with Ilum’s architecture and operational model. https://kestra.io/