Improved Feature
8 days ago

Announcing Next-Gen Data Lineage in Ilum

We’ve shipped a major upgrade to Ilum’s lineage experience, purpose-built to help data teams see how data moves across bronze → silver → gold layers, understand job health at a glance, and trace the blast radius of any change in seconds.

What’s new
  • Smart Job Clustering
    No more spaghetti graphs. Ilum automatically groups similar jobs into compact clusters, so complex pipelines stay readable while preserving drill-down to the underlying runs.
  • Layer-aware lineage (Bronze / Silver / Gold)
    Instantly track how entities evolve across refinement layers. Each table card shows key fields and types, with clear upstream/downstream edges.
  • Operational overlays on the graph
    Every job node now surfaces last run, avg duration, and success rate right on the canvas—perfect for spotting hot paths and bottlenecks during incident review.
  • Version-aware datasets
    Open any dataset and jump to the Versions tab to see schema changes and lifecycle events (e.g., OVERWRITE) before they surprise downstream consumers.
  • ERD ↔ Lineage toggle
    Switch between an entity-relationship view for model design and a lineage view for runtime flow—two perspectives, one source of truth.
  • Faster navigation
    Mini-map, zoom/pan, multi-select, and clean badges (e.g., OPERATIONAL) make large graphs effortless to explore.