Lineage now connects the full chain. An Airflow-triggered run links through Ilum down to the Spark job it launches, so a pipeline shows up as one connected graph instead of detached fragments under an unknown parent.
No more orphan jobs: the "unknown parent" node is gone. Airflow, Ilum and Spark runs join into a single lineage graph.
Navigate the graph: clicking a parent or root link now selects that node and pans the graph to it.
Where to find it: open the Lineage view for any Airflow-orchestrated run. Links:Data lineage
Ilum now integrates OpenMetadata, an open-source data catalog and governance layer, as an opt-in module. Once enabled it discovers your tables, captures their lineage and seeds governance automatically, so your catalog is populated from day one instead of starting empty.
A catalog that fills itself: Ilum registers your Hive, Delta, object-storage and BI services and ingests their tables for you.
End-to-end lineage: column-level lineage flows in from Spark and Airflow, so you can trace a column back to where it came from.
Governance built in: table certification, SLA hours and source-system tags are seeded out of the box.
Opt-in: the whole stack is off by default. Turn it on when you want it.
Why it matters: a data catalog only earns its keep when it reflects reality. Ilum wires the catalog to your live services and pipelines, so lineage and governance stay current without hand-maintenance. Where to find it: enable OpenMetadata in your values, then open the OpenMetadata view from the Ilum UI. Links:OpenMetadata
The bundled Spark runtime moves to 4.1.2. Every engine image across the stack moves with it. The driver, the SQL engine and the history server all run the same Spark version.
Spark 4.1.2 across the stack: the default Spark image, external spark-submit and the Spark history server are updated together.
Refreshed legacy line: the Spark 3.5.8 (Scala 2.12) image line is refreshed for teams still on the older runtime.
Nessie on Spark 4.x: the Nessie Iceberg extensions now run on Spark 4.x, so Iceberg-on-Nessie works on the current runtime.
Ilum gains a Pipeline Trace view that follows a single run end to end, from wherever it was triggered down to the Spark work it produced, with the logs for each step right alongside the trace. A companion cost attribution feature turns that activity into money, folding a daily rollup into per-job and per-table Cost tabs. Both are off by default.
One trace, whole run: follow a run from its origin (a schedule, an API call or an orchestrator like Airflow) through Ilum down to Spark in a single timeline.
Logs beside the trace: correlated logs sit next to each span, so you go from a slow step to its log lines in one place.
Cost per job and per table: a daily rollup attributes compute cost down to the individual job and table, shown in dedicated Cost tabs.
Yours to tune: the sampling rate and the cost rate card are configurable from settings.
Why it matters: when something goes wrong or the bill spikes, a run's story is normally scattered across every tool it touched. Pipeline Trace and cost attribution put the whole run, its logs and its cost in one place. Where to find it: enable observability and the cost aggregator in your values, then open the Pipeline Trace and Cost tabs.
The Ilum web UI now speaks more than one language. Switch language and text direction at any time from Settings and your choice is remembered. Ilum ships with seven languages out of the box: English, Persian, Arabic, Portuguese, Spanish, Chinese and Hindi.
Switch on the fly: change language and direction from Settings without reloading. Your choice is saved for next time.
Where to find it: open Settings and pick your language from the selector.
Standing up an Iceberg + Nessie catalog is now zero-config and works on non-AWS and air-gapped object storage straight away. Ilum sets up the catalog for you and handles its storage credentials automatically. A built-in Nessie catalog just works on any S3-compatible store and under any catalog name, with no Spark settings to edit by hand.
Catalog set up for you: Ilum configures the Iceberg+Nessie catalog from your settings. Choosing a custom catalog name no longer leaves you with a broken catalog.
Credentials handled automatically: the catalog's storage credentials are applied for you and are never stored in or shown by Ilum.
Works off-AWS: the catalog's region is now set correctly, fixing the error that used to hit the very first table creation against non-AWS storage.
Iceberg-on-Nessie used to need careful, easily-broken setup that failed silently on the first write outside AWS. Now it's a default that works the first time, consistently, wherever your object storage lives. Where to find it: create or edit a Nessie catalog from the Metastores view. The region field is right there in the form. Nessie catalog creation Links:Nessie catalog Available in version: 6.7.2
The bundled JupyterLab now ships the Pipeline Exporter: turn a notebook into a running pipeline without ever leaving Jupyter. Export it as either an Ilum Spark job or an Airflow pipeline. You can also download it as a plain Python script. Ilum generates everything for you, schedules it, triggers the run and links you straight back into Ilum to watch it execute.
Export to an Ilum Spark job: turn your notebook into a one-shot job, a long-running service or a scheduled (cron) job that runs on Ilum's own Spark.
Export to Airflow: generate an Airflow pipeline straight from your notebook, with no DAG code to hand-write.
One-click run: trigger the generated pipeline and jump straight back into Ilum's History and Services views through clickable links.
Per-step retries: tag individual cells (retries:<n>) to control how many times each step retries: more for flaky I/O steps, none for parameter cells.
In the box: bundled into the standard Jupyter image. A fresh install has it ready, nothing extra to set up.
Why it matters: promoting an exploratory notebook to a scheduled pipeline normally means rewriting it by hand as pipeline code, which is exactly where logic drifts and bugs slip in. The Pipeline Exporter collapses that into a click and keeps the notebook as the single source of truth. Where to find it: open a notebook in the bundled JupyterLab and use the Pipeline Exporter panel. Pipeline Exporter embedded in JupyterLab Links:Pipeline Exporter Available in version: 6.7.2
The bundled Spark runtime moves to 4.1.1. Every engine image across the stack moves with it. The driver, the SQL engine and the history server all run the same Spark version. Your jobs, your SQL and your run history stay on one consistent, current line.
Spark 4.1.1 across the stack: the default Spark image, external spark-submit and the Spark history server are all updated together.
Matching engines and Delta: the Kyuubi SQL engine and the Spark launcher images are bumped in step, with Delta Lake 4.2.0 matched to Spark 4.1.
Refreshed capabilities: geospatial support (Apache Sedona) and the bundled storage connectors are updated for Spark 4.1.
When the driver, the SQL engine, the history server and Delta drift apart in version, you get subtle, hard-to-reproduce failures. Moving them together keeps the whole stack on one consistent, supported Spark version. Links:Apache Spark on Ilum Available in version: 6.7.2
Ilum now connects to Hive 4.2.0 metastores alongside Hive 3, letting you move to the modern Hive line for SQL and table operations whenever you're ready. The Hive 3 path is unchanged and remains the default. Existing deployments are untouched.
Hive 4.2.0 support: connect Ilum to a Hive 4 metastore for SQL and table operations, alongside continued Hive 3 support.
Safer upgrades: Ilum checks at install time that the backend and the bundled metastore agree on the Hive version and fails early with a clear message if they don't.
Clearer startup: Ilum reports the Hive version and metastore address it resolved at boot and fails fast with a readable message if the metastore is unreachable.
Hive 4 modernizes the metastore, but no platform can stay on Hive 3 forever. Ilum supports both versions at once and validates them at install time, letting you adopt Hive 4 on your own schedule without risking a mismatch. Links:Hive catalog Available in version: 6.7.2
Ilum's S3-compatible storage layer is now fully pluggable. MinIO remains the default, now joined by two new backends: RustFS and Rook-Ceph. All three sit behind a single stable address. You can change the storage backend underneath the whole platform without reconfiguring a single service. You can even migrate from one backend to another while everything else keeps running.
RustFS: a lightweight, open-source (Apache-2.0), Rust-based S3-compatible store, available as an opt-in choice. RustFS is planned to become the default in 6.8.0.
Rook-Ceph (Ceph RGW): point Ilum at your own pre-deployed Ceph cluster as the storage backend, with safeguards that prevent accidentally enabling two backends at once.
One address, every consumer: Spark, Trino, Nessie, Jupyter, MLflow, Airflow, Kestra and Langfuse all reach storage through the same address. Swapping backends needs no per-service changes.
One place for credentials: all backends share a single credentials source. Rotate keys in one place.
Guided MinIO to RustFS migration: a built-in migration copies your buckets across, with a preview run first. Your source data is never deleted unless you explicitly choose to.
Why it matters: object storage is the one dependency every other module shares, which usually makes it the hardest thing to change. Putting it behind a stable address lets you pick the store that fits your environment (lightweight RustFS, your own Ceph or classic MinIO) and migrate between them as a routine, reversible step. RustFS console Links:Object storage guide, Migrate between providers Available in version: 6.7.2
Presets let you save a named bundle of Spark / runtime settings once and reuse it everywhere, instead of re-typing the same handful of options on every cluster, service, schedule or job. Ilum ships ready-made delta and iceberg presets to get you started. You can also create, edit and share your own.
Define once, attach anywhere: capture a Delta, Iceberg or Hudi setup as a preset and attach it by name to clusters, services, schedules and jobs.
Ready to use out of the box: the delta and iceberg presets come pre-loaded, with delta already attached to the default cluster.
Governed like any resource: presets carry their own ownership, search and permissions. You control who can create or change them.
Just works at launch: a preset's settings are applied automatically when a job runs, while anything you set directly on the job still takes precedence.
Built for automation: beyond the UI, presets can be managed programmatically to fit into your scripts and CI pipelines.
Spark configuration is where copy-paste mistakes creep in: a wrong setting here, a missing option there. Presets turn a fragile, repeated block of settings into one named, reusable unit your whole team maintains in a single place.
Where to find it: manage presets in the Presets view. Attach them from any create or edit form.
Every core resource now has a clear owner and a full audit trail: who created it, who last changed it and when. You always know who's responsible for a cluster, service, schedule, job, SQL query or notebook. Ownership is enforced on delete and can be handed to another person at any time. Every transfer is recorded in the operation history.
Ownership on core entities: clusters, services, schedules, jobs, SQL queries and notebooks each have an owner, set to whoever created them.
Created and modified tracking: every resource shows who created it, who changed it last and when, in its list and detail views.
Safe delete: only the owner (or an administrator) can delete a resource. One person can't wipe another's work by accident.
Ownership transfer: hand a resource to someone else from the UI. The transfer is recorded in the operation history.
Clear guardrails: if you try to act on something you don't own, Ilum tells you who to ask and how ownership can be transferred, instead of failing with a generic error.
Tidy user removal: when a user is deleted, their resources are left un-owned rather than deleted or kept blocking the removal.
In a team, untracked resources quietly become orphaned and risky to touch. A clear owner and an audit trail make cleanup, accountability and handover routine instead of guesswork. Where to find it: owner and audit details appear in each resource's detail view and the SQL sidebar. Transfer ownership from the resource's actions.
A new operation history records who did what across the entire platform: every create, edit, delete and ownership change. It renders as one searchable, filterable timeline in a dedicated UI tab. Rather than stitching together logs from different components, you get a single chronological record that sits right next to the resources it describes.
One audit trail for everything: clusters, services, schedules, jobs, presets, users and groups all report into the same history.
Filter instead of grep: every entry records the user, the action and the resource. Filter by person, resource or event type to find exactly what changed.
Built-in UI tab: browse and drill into the history directly in the app. There's no external logging stack to deploy or query.
Replaces the old "activities" trail: the previous ad-hoc audit feature is gone, folded into one consistent history.
Why it matters: when a cluster is reconfigured, a job is deleted or ownership changes hands, you can answer "who did this and when?" in seconds instead of spending a day digging through logs. Where to find it: open the Event Log view from the main navigation.
Ilum can now run its entire metadata store on PostgreSQL as a first-class alternative to MongoDB. Choose your database when you deploy. Ilum sets up the schema, keeps it up to date and (if you're coming from MongoDB) moves all of your data across for you. Both backends behave identically, with no feature trade-off in the choice. Existing MongoDB deployments keep running as before. MongoDB stays the default.
Your database, your choice: run Ilum on PostgreSQL or MongoDB. Standardize on whatever database your team already operates.
Zero manual schema work: on a PostgreSQL deployment the schema is created and kept up to date automatically at startup. There's nothing to apply by hand.
Guided migration from MongoDB: moving an existing install over is a single guided step that copies everything (clusters, services, schedules, jobs, SQL queries and notebooks, presets, users, groups and their ownership and audit history) into PostgreSQL. It's safe to re-run. Once it's done, MongoDB can be switched off.
Nothing left behind: application data and configuration all carry over intact. The migrated platform looks and behaves exactly like the original.
Many platform teams already standardize on PostgreSQL for backups, high availability and the tooling around it. Ilum now fits that stack natively. The built-in migration turns moving off MongoDB into a guided upgrade rather than a data-export project. Links:Migration guide Available in version: 6.7.2
Ilum CLI & Module Marketplace: Install, Configure, and Extend from Terminal or UI
Ilum now gives you two ways to manage the entire platform and its modules: a new CLI for terminal-first workflows and a built-in Module Marketplace in the UI for point-and-click deployment. Same modules, same configuration. Pick whichever fits your workflow.
Ilum CLI: one command to a running platform
The CLI wraps Helm and kubectl into a single purpose-built tool with automatic dependency resolution, interactive wizards, and health checks. Get started in seconds:
That's it. quickstart checks prerequisites, detects or creates a local cluster (k3d/minikube/kind), and deploys Ilum with sensible defaults. From there, manage everything from the terminal: ilum status for release info and pod readiness, ilum access open to launch the UI, ilum module enable <name> to add modules with automatic dependency resolution. Need Langfuse? ilum module enable langfuse automatically pulls in PostgreSQL and ClickHouse. For custom setups, ilum init walks you through cluster selection, module choices, and configuration before ilum install deploys it all.
Available via pip install ilum, pipx, or uv. Works on Linux, macOS, and Windows.
Module Marketplace: deploy from the UI
The new Ilum Modules page in the UI puts every available integration in one place, organized by category: data catalogs (Hive Metastore, Project Nessie, Unity Catalog, Apache Polaris, Apache Gravitino, DuckLake), transformation and workflows (Apache Airflow, Apache NiFi, dbt, Mage AI, Dagster, n8n, Kestra, Apache Kafka), data governance (OpenLineage, Spline, Open Metadata, Apache Atlas), and more.
Each module has an Overview with key features and resource requirements (CPU, memory), a Config tab to customize every setting (image tags, storage options, replicas, dependent services), and a Docs tab linking to full documentation. When ready, hit Install Module or Deploy Module and Ilum handles the Helm deployment on your cluster. Already installed modules show a Manage button for updates and reconfiguration.
Why it matters
Adding a new tool to your data stack used to mean researching Helm charts, editing YAML, resolving dependency order, and hoping nothing breaks. Now it's either one CLI command or a few clicks in the UI. The same 32 modules are available in both: CLI for automation and CI/CD, UI for exploration and one-off deployments. Either way, dependencies are resolved automatically and configurations are validated before deploy.
The SQL Editor has received a major UI/UX overhaul, making it faster to navigate multi-query workflows and easier to move between engines without rewriting SQL.
Tab management
You can now open multiple queries and notebooks as tabs across the top of the editor. Switch between active work, reference queries, and tutorials without losing your place. The left sidebar organizes all saved queries by engine, so you can search, browse, and open anything in one click.
SQL Translate: convert between engines instantly
This is the big one. Hit Translate and Ilum will convert your SQL from one dialect to another, across all three supported engines: DuckDB, Trino, and Spark SQL. Moving a quick prototype from DuckDB to a production Spark job? Translate handles the syntax differences so you don't have to. It works in every direction: DuckDB → Spark, Trino → DuckDB, Spark → Trino, and so on.
What else is new in the editor
Format - auto-format your SQL for readability with one click.
Explain - view the query execution plan before running, useful for catching expensive operations early.
Engine picker - switch between DuckDB, Trino, and Spark SQL per query tab. Different tabs can target different engines simultaneously.
Inline results - query output appears directly below the editor with row counts, execution time, and an Export option. Multiple result sets are tabbed so you can compare runs side by side.
Why it matters
Teams working across engines no longer need to context-switch between tools or manually translate SQL dialects. Write exploratory queries in DuckDB for speed, translate to Spark SQL when the workload scales, and keep everything in one place, same editor, same catalog, same lineage.
Inline Query Profiling: Column Stats & Data Distribution on Every Result
Run a query, get a full data profile, automatically. The new Profile view turns any query result into an instant statistical breakdown with per-column distributions, completeness checks, and issue detection. No extra queries, no separate profiling tools.
What you see at a glance
Every column in your result set is listed with its data type, value range, a mini distribution chart, and a completeness indicator, all in one scrollable view. A summary bar at the top shows overall completeness, flags columns with potential issues, and breaks down column types (numeric, string, date, etc.) so you can spot problems before they reach downstream.
Drill into any column
Expand a column to get the full picture: non-null and null counts, completeness percentage, distinct values, min/max, mean, median, quartiles (Q1, Q3), IQR, standard deviation, skewness, sum, and an interactive histogram showing value distribution. For categorical columns, you can load top values to see frequency breakdowns.
Why it matters
Data profiling usually means writing dedicated queries or exporting to a separate tool. Now it's built into the query workflow, run your SQL, click Profile, and immediately understand the shape of your data. Catch missing values, skewed distributions, or unexpected ranges before they become pipeline bugs. It works with every engine: DuckDB, Trino, and Spark SQL.
Where to find it
Run any query in the SQL Editor, then switch to the Profile tab in the results panel (next to the table and chart views).
Visual Query Plan & Optimization Hints: Understand and Fix Queries Before They Run
The Explain view now renders your query's execution plan as a fully interactive visual graph, paired with a Plan Insights summary and Optimization Hints that point to specific problems and tell you how to fix them.
Plan Insights: the big picture in seconds
The right-hand panel gives you a structural overview of your query: total operators, tree depth, max fan-out, CTE count, exchanges, filters, aggregates, and columns read. Below that, a Join Types breakdown shows exactly how many of each join strategy the planner chose, hash joins, nested loop joins, cross products, and others, so you can spot expensive patterns immediately.
Interactive plan graph
The full execution plan renders as a zoomable, navigable tree. Each node shows its operator type (scan, filter, projection, window, CTE, aggregate), the expression or columns involved, and the estimated cardinality. Click any node to inspect it. Zoom out to see the full shape of a 100+ operator plan, or zoom in to trace a specific branch.
Optimization Hints: actionable, not generic
This is where it gets practical. Ilum analyzes the plan and surfaces concrete issues with severity indicators:
Duplicate scan detection - flags when a table is scanned multiple times (e.g., 7×) and suggests caching or CTE restructuring to eliminate redundant reads.
Deep plan warning - alerts when the plan tree is unusually deep, recommending you break the query into stages with temporary tables or cached CTEs.
Each hint includes a "Go to node" link that jumps directly to the problematic operator on the graph, so you don't have to hunt through a complex plan manually.
Why it matters
Most SQL users run EXPLAIN and get a wall of text. This turns that into a visual debugging tool with built-in intelligence - you see the plan, understand the shape, and get told what to fix, all in one view. Catch redundant scans, expensive joins, and over-complex plans before they waste compute. Works with DuckDB, Trino, and Spark SQL.
Where to find it
Write your query in the SQL Editor and click Explain. The plan graph opens in the results panel with Plan Insights and Optimization Hints visible on the right.
Managing queries and files in Ilum is now faster and more consistent. Every item across the platform, queries, notebooks, folders, storage files, now has a right-click context menu with the full set of actions available in one place. No more hunting through toolbars or modal dialogs.
Query management
The left sidebar organizes all your saved queries and notebooks with folder grouping (e.g., by engine) and a Pinned section for frequently used items. Right-click any query to Open, Rename, Duplicate, Move to…, Copy to…, Download, or Delete, all from a single menu. Search across all queries from the top of the sidebar.
File browser sidebar
A new compact file tree gives you direct access to your S3 storage hierarchy from inside the SQL Editor. Browse buckets and folders, right-click any file to Download, Copy Path, or Create SQL Query directly from the file, turning a raw data file into a queryable starting point in one click.
File Manager improvements
The full File Manager view now supports Create File alongside existing Upload and Create Folder actions, giving you a way to create files directly in storage. Right-click any file for the complete set of operations: Preview, Edit, Download, Copy Path, Copy Name, Rename, Move to…, Delete — plus data-oriented actions like Create Table from File, Analyze with SQL, and Profile Data. When files are selected, a bottom action bar appears with the most common bulk actions for quick access.
Why it matters
Small workflow friction adds up. Context menus mean fewer clicks, fewer page switches, and a more natural way to work with your data assets. Whether you're reorganizing queries into folders, copying a storage path into a notebook, or profiling a CSV you just uploaded, the action is always one right-click away.
Data Lineage: Richer Dataset Cards, Inline Lineage Preview & Version History
The lineage experience gets a major upgrade: more context visible upfront, version tracking built into every dataset, and a redesigned dataset catalog that lets you assess data assets without clicking into each one.
Dataset catalog with inline schema & lineage preview
The new Datasets view presents every tracked dataset as a card showing its storage location, table format (Delta, Parquet, etc.), and the full schema spec with column names and types, all visible without opening the dataset. Each card also includes a Lineage Preview mini-diagram showing upstream and downstream connections at a glance, so you can see a dataset's position in the pipeline before drilling in. Jump straight to Details or Column Lineage from any card.
Version History: schema, execution, and storage in one timeline
Open any dataset and switch to the Versions tab to see a complete version history. Each version entry shows the event type (CREATE, OVERWRITE, etc.), a timestamp, a Change Log describing what happened, the full Schema at that point in time, and Execution details, including state, duration, and run ID. Storage metadata (format, size) is tracked per version too. You can now answer "what changed, when, and what job caused it" from a single panel.
Lineage graph improvements
The lineage canvas itself gets more controls for navigating complex pipelines: Smart Job Clustering groups related jobs to reduce visual noise (with configurable min/max thresholds), Merge Edges and Orthogonal Edges toggles clean up the layout, and Depth controls let you expand or collapse how many hops are visible. Switch between Standard Pipeline and Smart layouts depending on the complexity of the graph. Layer badges (Bronze, Silver, Gold) and field-level schema remain visible on every node.
Why it matters
Lineage is only useful if you can navigate it quickly. The dataset cards eliminate the "click into everything to understand anything" problem. Version history closes the loop between lineage and governance, you know not just where data flows, but how it changed over time and which job execution caused each change. Together, these improvements make lineage a practical daily tool rather than a diagram you look at during audits.
Powered by FeedBear