New
Feature
Pipeline Trace and Cost attribution
Ilum gains a Pipeline Trace view that follows a single run end to end, from wherever it was triggered down to the Spark work it produced, with the logs for each step right alongside the trace. A companion cost attribution feature turns that activity into money, folding a daily rollup into per-job and per-table Cost tabs. Both are off by default.
- One trace, whole run: follow a run from its origin (a schedule, an API call or an orchestrator like Airflow) through Ilum down to Spark in a single timeline.
- Logs beside the trace: correlated logs sit next to each span, so you go from a slow step to its log lines in one place.
- Cost per job and per table: a daily rollup attributes compute cost down to the individual job and table, shown in dedicated Cost tabs.
- Yours to tune: the sampling rate and the cost rate card are configurable from settings.
Why it matters: when something goes wrong or the bill spikes, a run's story is normally scattered across every tool it touched. Pipeline Trace and cost attribution put the whole run, its logs and its cost in one place.
Where to find it: enable observability and the cost aggregator in your values, then open the Pipeline Trace and Cost tabs.
Where to find it: enable observability and the cost aggregator in your values, then open the Pipeline Trace and Cost tabs.
Available in version: 6.7.3