With the integration of open lineage and Marquez, every Spark job in our system now comes equipped with built-in Data Lineage capabilities. This means you can easily track the journey of your data, understand its origins, transformations, and destinations, all within the familiar environment of Ilum.

 

Understanding the flow of your data is crucial for making informed decisions and ensuring data integrity. With Data Lineage seamlessly integrated into Ilum, you can now visualize and trace the path of your data with ease, empowering you to make data-driven choices confidently.

 

Take advantage of this powerful feature today and gain deeper insights into your data processing workflows. Enhance your data management experience with Ilum’s Data Lineage, setting a new standard for transparency and control in your Spark jobs! ✨

 

We understand the importance of strong authentication mechanisms. That's why we've introduced both internal and LDAP authentication methods. With internal authentication, you can now create a static list of users and grant them access to Ilum. On the other hand, LDAP authentication enables you to leverage your existing user directory, making user management a breeze.

We've also added OAuth2 support. With Enhanced Security, we've put your data protection first. We believe that security shouldn't be complicated, and with Ilum, it doesn't have to be. So go ahead, explore the new features, and experience the peace of mind that comes with knowing your data is in safe hands.

 

Planned for release in 6.1.0

 

Hey there, Ilum users!

 

We have some awesome news to share with you today. We've just rolled out a new feature that we know you've all been eagerly waiting for, The support for other storage options!

 

Up until now, Ilum has been tightly integrated with S3 interface, and we've received numerous requests from our incredible user community to expand our horizons. We've listened, and we're excited to announce that Ilum now supports not only S3, but also GCS, HDFS and Azure Blob Storage!

 

What does this mean for you? Well, it means you now have the flexibility to choose the storage option that best suits your needs and seamlessly connect it with Ilum. Whether you're a fan of HDFS, use Azure for your data storage, or prefer Google Cloud Storage, Ilum has got you covered.

Planned for release in 6.1.0

 

With this update, we've taken user feedback to heart and made significant improvements to the Ilum interface, making it even easier and more intuitive to manage your Spark sessions and jobs. We believe that a seamless user experience is the key to unlocking the full potential of Apache Spark and Kubernetes, and that's exactly what we're delivering with our enhanced UI.

 

Our team has been hard at work, refining every aspect of the user interface to ensure a smooth and efficient workflow. We've revamped the layout, simplified navigation, and added new features that will make interacting with Ilum a breeze. Whether you're a seasoned data scientist or just getting started with Spark, UI v3 will empower you to unleash the full power of your data with ease and confidence.

 

We are thrilled to announce the latest update to our Ilum platform - the introduction of the "Embedded Notebook" feature! 🚀

 

At Ilum, we are always striving to enhance your experience and make your data analysis journey seamless. With the new Embedded Notebook, we are taking your productivity to the next level. Say goodbye to switching between multiple tools and platforms - now you can access a Jupyter notebook directly within our user-friendly web interface.

 

Imagine the convenience of having all your data exploration, visualization, and machine learning tasks in one place. No more juggling between different applications or wasting time on tedious setup. With our Embedded Notebook, you can seamlessly transition from exploring your data to building models, all within the familiar Ilum environment.

This feature will be a part of version 6.0.0

 

We are thrilled to announce the latest enhancement to Ilum - Python Spark Jobs support! 🚀

 

At Ilum, we are committed to providing a user-friendly and versatile data processing platform. With this new feature, we are broadening our usability by allowing Python-based Spark jobs. Now, data scientists and developers who prefer Python can leverage Ilum to process big data, perform machine learning tasks, and more, all within their preferred language environment.

 

Our goal has always been to make data processing as straightforward as possible, and this update takes us one step further. By extending Ilum's capabilities to include Python, we are making our platform more inclusive and catering to the needs of a wider range of users.

 

With Python Spark Jobs support, you can now seamlessly integrate Python code into your Spark sessions managed through Ilum. Say goodbye to the hassle of switching between different language environments or learning new programming languages. Whether you are a Python enthusiast or have existing Python-based workflows, Ilum has got you covered.

 

As you know, Ilum sets a new standard in integrating Apache Spark with Kubernetes. With Python support, we are revolutionizing data processing by giving you the flexibility to choose your preferred language without compromising on performance or ease of use. Connect to your Kubernetes cluster, submit Python Spark jobs, and monitor them effortlessly using our user-friendly web interface or REST API.

 

Stay tuned for more exciting updates as we continue to enhance Ilum and empower you to unlock the full potential of your data with ease. We appreciate your valuable feedback and encourage you to reach out to us with any suggestions or questions.

 

Thank you for being a part of the Ilum community! 🌟

 

Hey Ilum users! We've got some exciting news to share with you today. We are thrilled to announce the latest update to our product that will take your big data tasks to new heights: the integration of Apache Airflow with Ilum!

 

Introducing "Airflow integration" - a game-changer feature that brings streamlined orchestration, management, and monitoring of data pipelines right to your fingertips. With just a flick of a switch, the `airflow.enabled=true` flag in our Ilum Helm chart will deploy Apache Airflow alongside Ilum, creating seamless connections and interfaces required for integration.

 

So, what does this mean for you? It means that complex big data tasks and workflows just got a whole lot simpler. With Airflow integration, you can easily manage and monitor your data pipelines between Airflow and Ilum, all in one place. No more jumping between different tools or drowning in a sea of logs - everything you need is now at your fingertips.

 

At Ilum, we believe in making your life easier. That's why we set out to create a product that eliminates the need for executing commands via CLI or spending endless hours searching for errors in logs. And now, with Airflow integration, we're taking it a step further by offering you a streamlined solution for orchestrating your data pipelines.

 

Introducing Ilum 5.0.

We're excited to share with you one of our biggest updates yet, packed with our most popular feature request, significant performance improvements, a revamped UI, plus plenty of additional improvements, tweaks and fixes. With this update, you can expect to see an enhanced user experience and improved performance.

Watch the video!

Some of the new features: