Data Lineage: Richer Dataset Cards, Inline Lineage Preview & Version History

The lineage experience gets a major upgrade: more context visible upfront, version tracking built into every dataset, and a redesigned dataset catalog that lets you assess data assets without clicking into each one.

The new Datasets view presents every tracked dataset as a card showing its storage location, table format (Delta, Parquet, etc.), and the full schema spec with column names and types, all visible without opening the dataset. Each card also includes a Lineage Preview mini-diagram showing upstream and downstream connections at a glance, so you can see a dataset's position in the pipeline before drilling in. Jump straight to Details or Column Lineage from any card.

Open any dataset and switch to the Versions tab to see a complete version history. Each version entry shows the event type (CREATE, OVERWRITE, etc.), a timestamp, a Change Log describing what happened, the full Schema at that point in time, and Execution details, including state, duration, and run ID. Storage metadata (format, size) is tracked per version too. You can now answer "what changed, when, and what job caused it" from a single panel.

The lineage canvas itself gets more controls for navigating complex pipelines: Smart Job Clustering groups related jobs to reduce visual noise (with configurable min/max thresholds), Merge Edges and Orthogonal Edges toggles clean up the layout, and Depth controls let you expand or collapse how many hops are visible. Switch between Standard Pipeline and Smart layouts depending on the complexity of the graph. Layer badges (Bronze, Silver, Gold) and field-level schema remain visible on every node.

Lineage is only useful if you can navigate it quickly. The dataset cards eliminate the "click into everything to understand anything" problem. Version history closes the loop between lineage and governance, you know not just where data flows, but how it changed over time and which job execution caused each change. Together, these improvements make lineage a practical daily tool rather than a diagram you look at during audits.

Introduced in 6.7.0