Improved

Clearer Airflow to Ilum to Spark lineage

Lineage now connects the full chain. An Airflow-triggered run links through Ilum down to the Spark job it launches, so a pipeline shows up as one connected graph instead of detached fragments under an unknown parent.

No more orphan jobs: the "unknown parent" node is gone. Airflow, Ilum and Spark runs join into a single lineage graph.
Navigate the graph: clicking a parent or root link now selects that node and pans the graph to it.

Where to find it: open the Lineage view for any Airflow-orchestrated run.
Links: Data lineage

An Airflow-orchestrated run in Ilum's Lineage view, from the DAG through the Ilum job down to the bronze, silver and gold tables it wrote

The Airflow run, three Ilum jobs launched through Livy

Available in version: 6.7.3