Ilum 6.3.0 is here, a robust update designed to supercharge your data workflows and elevate your Spark experience. This release brings a host of new features, performance enhancements, and essential integrations that empower advanced data processing and governance for Spark users.

New Features in This Release

1. Introducing Gitea Integration: Enhancing Version Control Inside Ilum
We’re excited to announce that Gitea—a lightweight, self-hosted Git repository manager—is now built into Ilum. Manage version control for Jupyter notebooks, Spark job scripts, Airflow DAGs, and more, all without leaving the platform. This seamless integration simplifies collaboration for data engineers, data scientists, and DevOps teams.

2. Internal Security System Improved
Our new internal security system strengthens authorization and authentication. User data is securely stored in a dedicated database, with Role-Based Access Control (RBAC) enabling you to organize users into roles and groups, ensuring every action is properly permissioned.

3. Enhanced SQL UI with Notebook-Style Execution
Experience a more interactive SQL Editor that now supports notebook-style operations. Write and execute SQL queries in structured cells—just like in Jupyter Notebook—so you can explore and analyze big data more efficiently.

Other Improvements

Ilum 6.2.1 is a robust update that will supercharge your data workflows and enhance your experience with Spark.

We’re excited to bring you one of our strongest releases to date, with new features, performance enhancements, and essential integrations that enable advanced data processing and governance for Spark users. 

SQL API/UI Section: Making Data Easier with Ilum.
Initiate a New Era of Querying Data! Execute SQL queries directly on your cluster!  Full SQL dialect support allows you to work natively with any modern open formats like Delta Lake, Apache Iceberg, and Apache Hudi. Also, you can also use multiple engines SparkSQL, Trino(Comming) and Apache Flink(Comming).

Advanced Spark Metrics Integration.
With enhanced metrics, you can analyze your spark jobs better. Check the resources, jobs, and health of running applications quickly and troubleshoot the issues quickly. Whether you're a data engineer or a developer or a scientist, you can drive efficiency with these metrics.

Auto-Scaling for Interactive Sessions.
Efficient Resource Utilization Is The Key! Inactive interactive jobs will automatically scale down to zero after inactivity, freeing up resources for other jobs. When the activity resumes, the sessions smoothly scale back up by themselves.

Automated Spark job scheduling with cron
No need to trigger jobs by hand! With our new cron-based scheduling, you can automate your Spark job executions using a familiar cron-like syntax. Our system is integrated with Kubernetes so that your jobs run on time with automated retries and notifications.

Integration with Superset.
Get better visualisation and analytics through Superset integration. You can check the insights in your data by using Analytics, and you can share them too.

Spark Memory Settings Section.
Customize Spark's settings for memory for better performance tweaks. Adjust Spark memory settings for better resource use and improved job performance.

Integration with MLflow.
Speed up your machine learning! Using MLflow Integration, manage your ML experiments, track result and help streamline model deployment within Ilum.

Prepare to take your data journey to the next level with Ilum 6.2.1 – enhancing your analytics, Spark job management, and data interactions like never before. Get the latest version and see!

Happy Data Processing! 🚀✨

Introducing Ilum 5.0.

We're excited to share with you one of our biggest updates yet, packed with our most popular feature request, significant performance improvements, a revamped UI, plus plenty of additional improvements, tweaks and fixes. With this update, you can expect to see an enhanced user experience and improved performance.

Watch the video!

Some of the new features: