We’re excited to announce that n8n, the powerful automation tool, is now available as a module in Ilum!

With n8n, you can build custom workflows, automate data tasks, and connect external services, all through a simple, visual interface right inside the Ilum platform.

What You Can Do with n8n in Ilum:

⚙️ Automate Data Pipelines – Trigger Spark jobs, move data between systems, or schedule SQL queries without writing scripts.
🔌 Integrate with Anything – Connect Ilum to external APIs, messaging systems, cloud storage, Slack, Git, and more.
🧠 Build Logic Visually – Drag, drop, and configure nodes to create advanced automation workflows with ease.
📅 Schedule and Monitor – Set up workflows to run on cron schedules, webhook triggers, or conditionally based on data states.

This new module brings no-code/low-code automation to your data platform, giving teams the power to create dynamic, event-driven pipelines and integrations in minutes.

Explore the n8n module today and bring your workflows to life! 🚀

We’ve added Trino SQL engine support to Ilum’s SQL interface, giving you even more flexibility and power when querying your data lakehouse.

From now on, you can use both Apache Spark SQL and Trino SQL, even within the same SQL notebook, making it easier to choose the right engine for the job without switching tools.

Key Highlights:

🧠 Dual SQL Engine Support – Mix and match Spark SQL and Trino SQL inside a single workspace.
⚡ Faster Interactive Queries – Use Trino for low-latency SQL queries over your distributed data sources.
🔌 Same UI, More Power – No need to learn a new interface. Trino is fully integrated into Ilum’s existing SQL editor.
📁 Query External Data – Trino makes it easy to connect to external sources like Hive, Iceberg, S3, and more.
🧪 Flexible Execution – Choose the best engine per query or per cell, depending on performance or data access needs.

This feature unlocks new possibilities for real-time exploration, federated querying, and fine-tuned performance optimization—right inside Ilum.
Try it today and experience multi-engine querying done right! 🚀

We’ve upgraded the Ilum SQL Editor with a powerful new feature: select and run specific queries directly from a single view—no need to execute the entire notebook or clean up your workspace first.

Key Enhancements:

🧩 Query Picker – Quickly choose which query to execute from a list of all defined queries in your notebook.
🎯 Focused Execution – Run only the SQL block you need, saving time and avoiding unnecessary data scans.
🧭 Better Navigation – Easily jump between queries and manage your workflow more efficiently.
💡 Cleaner Debugging – Isolate and test specific parts of your SQL logic without affecting the rest of the notebook.

This improvement brings more control and flexibility to your data exploration and debugging workflows. Whether you're refining a single query or testing logic step-by-step, Ilum makes it faster and easier.

Give it a try and streamline your SQL journey! 🚀

We’ve completely redesigned the Spark session creation form for Jupyter Notebooks (Sparkmagic), making it faster and easier than ever to configure your sessions without dealing with raw JSON.

This is just the beginning! 🎉 We’ll be enhancing the form with every release, making Spark session management in Jupyter even more smooth.

We’ve enhanced Ilum’s SQL section with full Markdown support, transforming it into a fully capable notebook solution and a powerful alternative to our Jupyter Notebooks module.

Now, you can mix SQL queries with Markdown annotations, making it easier to document insights, explain query logic, and create interactive data reports, all within Ilum.

We’ve added Kubernetes Resource Quotas, CPU Limit Range, and Memory Limit Range to Ilum, giving you granular control over resource allocation within your cluster or namespace. Now, you can isolate workloads, prevent resource overuse, and ensure fair distribution across teams.

With this update, you can set per-job CPU and memory limits, so no more worrying about a single user consuming 1TB of RAM and starving other workloads. Instead, each team can have its own dedicated and controlled resource space, improving stability, efficiency, and cost management.

This enhancement brings more flexibility and governance to your Ilum-powered Spark environment, allowing you to manage big data workloads with confidence. Try it out today and experience smarter, more efficient resource management! 🔥

We've upgraded Ilum’s SQL Editor to support notebook-style operations, making it easier than ever to explore, query, and analyze big data at scale. Starting now, you can write & execute SQL in structured cells, just like in Jupyter Notebook, instead of running your Apache Spark SQL queries one by one. This enables you to run SQL or SQLite queries similar to a Jupyter Notebook. So instead of running Apache Spark SQL queries one by one, you can write and execute SQL in structured cells. 

Key Improvements: 
- Notebook-style SQL execution – Run and organize queries in cells for a more interactive SQL analytics experience. 
- Persistent query history – Save and revisit SQL jobs for improved workflow management. 
- Multi-cell execution – Execute individual cells or entire SQL notebooks for fast, scalable data exploration. 
- Integrated with BI & Data Platforms – Query data stored in cloud data lakes, data warehouses, and on-prem storage via JDBC integration. 

With this update, it is easier to query SQL inside Ilum and manage data lakehouse, business intelligence and big data analysis.  Check it out today and meet the better way to manage your structured and semi-structured data! 🚀

Simply said, this version includes a new internal security system that aims to give Ilum better authorization and authentication. This feature keeps all user data within a dedicated database and servers make use of Role-Based Access Control (RBAC) which makes sure every action made on the server is permissioned. You can now organize Users into Roles and Groups which allow you to consistently assign permissions to them across the platform.

The new framework provides default roles such as ADMIN, USER, DATA_ENGINEER and DATA_SCIENTIST to define defaults for common users. Each role has its own set of permissions that allow users to perform jobs and run services. Access is also provided to various modules that include File Explorer, Jupyter/ Zeppelin Notebooks, History Server, etc. Admin can also see the logs which capture all the activity done in the app with details.

https://ilum.cloud/docs/security/internal_security/

We're excited to share that we've added Gitea right inside Ilum. Gitea is a lightweight and self-hosted Git repository manager. Our new module lets teams work together freely on projects with Jupyter notebooks, Spark job scripts, Airflow DAGs and more!
 
By adding Gitea into Ilum means you can now keep the version control of all your data and code artifacts within Ilum itself without relying on any other Git services. Consequently, it provides easy workflows for Data Engineers, Data Scientists and DevOps staff since they can now conveniently execute their work without having to switch contexts.
 
Utilizing GitOps principles allows you to automate your deployments, enforce approvals and manage your infra as code. With these capabilities, you can easily monitor updates, commit changes, and revert versions quickly, ensuring reproducibility of your experiments and smooth collaboration with your team.
 
Get started with this integration that adds more security and compliance and enhances your workflows in Ilum. Discover how Gitea integration allows you to track updates, commit changes, and revert to older versions quickly for reproducibility of experiments.

 We're excited to introduce a game-changer in the world of data interaction - the brand new SQL API/UI section in Ilum! 🚀 Now, you can effortlessly run SQL queries directly on the cluster, revolutionizing the way you interact with your data. 

 

 With full SQL dialect capabilities at your fingertips, including seamless integration with modern data formats like Delta Lake, Apache Iceberg, and Apache Hudi, data querying and manipulation have never been more straightforward. Plus, beyond just SparkSQL, Ilum now supports Trino and Apache Flink, offering you a versatile toolkit to meet your diverse data processing needs. 

 

 Say goodbye to complex data interactions and hello to streamlined analytics and data management tasks directly from our platform. Whether you're diving into different data formats or exploring various processing engines, Ilum's SQL API/UI section empowers you to navigate your data environment with ease. 

 

 Join us in embracing this new era of data interaction and experience firsthand how Ilum is reshaping the landscape of data processing. Get ready to elevate your data game and unlock new possibilities with our latest SQL API/UI feature. Your data journey is about to get a whole lot smoother - start exploring today!

https://ilum.cloud/docs/features/sql_viewer/

 

We are thrilled to announce the latest update to Ilum - the addition of more advanced spark metrics! 🚀 Now, you can dive even deeper into monitoring the performance and efficiency of your Spark jobs with enhanced metrics at your fingertips.

 

With these new metrics, you can gain valuable insights into the resource utilization, task execution, and overall health of your Spark applications. Understanding these metrics will empower you to optimize your jobs, troubleshoot any issues more effectively, and ultimately boost your productivity.

 

Whether you are a data engineer, a developer, or a data scientist, these advanced spark metrics will provide you with the visibility you need to make informed decisions and drive better outcomes. Say goodbye to guesswork and hello to data-driven insights that will take your Spark experience to the next level!

 

Keep an eye out for these new metrics in the Ilum interface, and start harnessing the power of data to supercharge your Spark jobs today. We can't wait to see how these advanced metrics will elevate your Spark experience and help you achieve even greater success in your data projects. Happy Sparking! ✨

 

We're excited to introduce a new feature that will streamline your interactive sessions on Ilum! Say goodbye to inactive sessions taking up unnecessary resources. With our latest update, we've implemented a smart solution to automatically pause interactive sessions that have been inactive for over a week.

 

Here's how it works: if an interactive session remains idle for a specified amount of time, Ilum will scale it down to zero, freeing up resources for other tasks. But don't worry about losing your progress or data. As soon as ilum-core detects any activity or a request to resume the session, it will promptly scale it back to one session, allowing you to pick up right where you left off.

 

This enhancement not only optimizes resource utilization but also ensures a smoother and more efficient experience for all Ilum users. No more manual intervention required to manage inactive sessions. Ilum takes care of it for you, so you can focus on your work without interruptions.

 

Experience the convenience of seamless session management with Ilum's latest update. Stay productive, stay efficient, and let Ilum handle the rest. We're committed to enhancing your Spark session management experience, making your workflow even more effortless and enjoyable. Try it out today and see the difference!

 

We are excited to announce a new enhancement to Ilum that will take your job management experience to the next level! Introducing the new `tag` field for jobs, designed to make job identification and organization a breeze.

 

With the addition of the `tag` field, you can now add custom tags to your jobs, helping you easily categorize and filter them based on your specific needs. Whether you want to label jobs by project, team, priority, or any other criteria, the `tag` field allows for seamless organization and quick retrieval of information.

 

We are excited to announce a major enhancement to Ilum that will revolutionize how you manage your big data environments - the official release of our Hive Metastore integration feature! 🚀 Say goodbye to the beta phase. This essential tool is now a standard part of Ilum, ready to supercharge your data management capabilities.

 

With the new Hive Metastore integration, you can efficiently handle metadata for your big data projects, ensuring a seamless way to manage schema and metadata across different storage solutions and formats. This means less time spent wrestling with data organization and more time focusing on what truly matters - deriving valuable insights from your data.

 

But that's not all - the Ilum team is already looking ahead to the future. We have ambitious plans to expand our data catalog capabilities further by integrating with Unity Catalog, Polaris, and Nessie. These upcoming additions will take Ilum to the next level, empowering you to manage and govern large-scale data across distributed environments with ease. Our goal? To keep Ilum at the forefront of data lakehouse technology, supporting your complex data operations and governance needs every step of the way.

 

Join us on this exciting journey as we continue to innovate and elevate your data management experience with Ilum. Stay tuned for more updates, and get ready to unleash the full potential of your big data projects like never before!

 

We are excited to announce the latest feature update for Ilum - Integration with Apache Sedona! 🚀

 

With this new integration, you can now leverage the power of Apache Sedona within your Spark sessions on Ilum. Apache Sedona, formerly known as GeoSpark, is a cluster computing system specifically designed for processing large-scale spatial data. The combination of Apache Sedona and Ilum's interactive Spark sessions enables seamless analysis and visualization of spatial data.

 

Whether you're working with geospatial datasets, location-based services, or any other spatial data applications, the Integration with Apache Sedona on Ilum empowers you to handle complex spatial queries and operations with ease. Say goodbye to the hassle of managing spatial data separately and welcome a more integrated and efficient workflow.

 

Ready to take your spatial data analysis to the next level? Experience the Integration with Apache Sedona on Ilum now and discover new opportunities for spatial data processing and analysis in your Spark tasks.

With the integration of open lineage and Marquez, every Spark job in our system now comes equipped with built-in Data Lineage capabilities. This means you can easily track the journey of your data, understand its origins, transformations, and destinations, all within the familiar environment of Ilum.

 

Understanding the flow of your data is crucial for making informed decisions and ensuring data integrity. With Data Lineage seamlessly integrated into Ilum, you can now visualize and trace the path of your data with ease, empowering you to make data-driven choices confidently.

 

Take advantage of this powerful feature today and gain deeper insights into your data processing workflows. Enhance your data management experience with Ilum’s Data Lineage, setting a new standard for transparency and control in your Spark jobs! ✨

 

We understand the importance of strong authentication mechanisms. That's why we've introduced both internal and LDAP authentication methods. With internal authentication, you can now create a static list of users and grant them access to Ilum. On the other hand, LDAP authentication enables you to leverage your existing user directory, making user management a breeze.

We've also added OAuth2 support. With Enhanced Security, we've put your data protection first. We believe that security shouldn't be complicated, and with Ilum, it doesn't have to be. So go ahead, explore the new features, and experience the peace of mind that comes with knowing your data is in safe hands.

 

Planned for release in 6.1.0

 

Hey there, Ilum users!

 

We have some awesome news to share with you today. We've just rolled out a new feature that we know you've all been eagerly waiting for, The support for other storage options!

 

Up until now, Ilum has been tightly integrated with S3 interface, and we've received numerous requests from our incredible user community to expand our horizons. We've listened, and we're excited to announce that Ilum now supports not only S3, but also GCS, HDFS and Azure Blob Storage!

 

What does this mean for you? Well, it means you now have the flexibility to choose the storage option that best suits your needs and seamlessly connect it with Ilum. Whether you're a fan of HDFS, use Azure for your data storage, or prefer Google Cloud Storage, Ilum has got you covered.

Planned for release in 6.1.0

 

With this update, we've taken user feedback to heart and made significant improvements to the Ilum interface, making it even easier and more intuitive to manage your Spark sessions and jobs. We believe that a seamless user experience is the key to unlocking the full potential of Apache Spark and Kubernetes, and that's exactly what we're delivering with our enhanced UI.

 

Our team has been hard at work, refining every aspect of the user interface to ensure a smooth and efficient workflow. We've revamped the layout, simplified navigation, and added new features that will make interacting with Ilum a breeze. Whether you're a seasoned data scientist or just getting started with Spark, UI v3 will empower you to unleash the full power of your data with ease and confidence.

 

We are thrilled to announce the latest update to our Ilum platform - the introduction of the "Embedded Notebook" feature! 🚀

 

At Ilum, we are always striving to enhance your experience and make your data analysis journey seamless. With the new Embedded Notebook, we are taking your productivity to the next level. Say goodbye to switching between multiple tools and platforms - now you can access a Jupyter notebook directly within our user-friendly web interface.

 

Imagine the convenience of having all your data exploration, visualization, and machine learning tasks in one place. No more juggling between different applications or wasting time on tedious setup. With our Embedded Notebook, you can seamlessly transition from exploring your data to building models, all within the familiar Ilum environment.

This feature will be a part of version 6.0.0

 

We are thrilled to announce the latest enhancement to Ilum - Python Spark Jobs support! 🚀

 

At Ilum, we are committed to providing a user-friendly and versatile data processing platform. With this new feature, we are broadening our usability by allowing Python-based Spark jobs. Now, data scientists and developers who prefer Python can leverage Ilum to process big data, perform machine learning tasks, and more, all within their preferred language environment.

 

Our goal has always been to make data processing as straightforward as possible, and this update takes us one step further. By extending Ilum's capabilities to include Python, we are making our platform more inclusive and catering to the needs of a wider range of users.

 

With Python Spark Jobs support, you can now seamlessly integrate Python code into your Spark sessions managed through Ilum. Say goodbye to the hassle of switching between different language environments or learning new programming languages. Whether you are a Python enthusiast or have existing Python-based workflows, Ilum has got you covered.

 

As you know, Ilum sets a new standard in integrating Apache Spark with Kubernetes. With Python support, we are revolutionizing data processing by giving you the flexibility to choose your preferred language without compromising on performance or ease of use. Connect to your Kubernetes cluster, submit Python Spark jobs, and monitor them effortlessly using our user-friendly web interface or REST API.

 

Stay tuned for more exciting updates as we continue to enhance Ilum and empower you to unlock the full potential of your data with ease. We appreciate your valuable feedback and encourage you to reach out to us with any suggestions or questions.

 

Thank you for being a part of the Ilum community! 🌟

 

Hey Ilum users! We've got some exciting news to share with you today. We are thrilled to announce the latest update to our product that will take your big data tasks to new heights: the integration of Apache Airflow with Ilum!

 

Introducing "Airflow integration" - a game-changer feature that brings streamlined orchestration, management, and monitoring of data pipelines right to your fingertips. With just a flick of a switch, the `airflow.enabled=true` flag in our Ilum Helm chart will deploy Apache Airflow alongside Ilum, creating seamless connections and interfaces required for integration.

 

So, what does this mean for you? It means that complex big data tasks and workflows just got a whole lot simpler. With Airflow integration, you can easily manage and monitor your data pipelines between Airflow and Ilum, all in one place. No more jumping between different tools or drowning in a sea of logs - everything you need is now at your fingertips.

 

At Ilum, we believe in making your life easier. That's why we set out to create a product that eliminates the need for executing commands via CLI or spending endless hours searching for errors in logs. And now, with Airflow integration, we're taking it a step further by offering you a streamlined solution for orchestrating your data pipelines.