New Feature: Integration with Apache Sedona

We are excited to announce the latest feature update for Ilum - Integration with Apache Sedona! 🚀
With this new integration, you can now leverage the power of Apache Sedona within your Spark sessions on Ilum. Apache Sedona, formerly known as GeoSpark, is a cluster computing system specifically designed for processing large-scale spatial data. The combination of Apache Sedona and Ilum's interactive Spark sessions enables seamless analysis and visualization of spatial data.
Whether you're working with geospatial datasets, location-based services, or any other spatial data applications, the Integration with Apache Sedona on Ilum empowers you to handle complex spatial queries and operations with ease. Say goodbye to the hassle of managing spatial data separately and welcome a more integrated and efficient workflow.
Ready to take your spatial data analysis to the next level? Experience the Integration with Apache Sedona on Ilum now and discover new opportunities for spatial data processing and analysis in your Spark tasks.

Version 6.1.0

  1. Data Lineage Support Added: Enhanced data management with new lineage tracking capabilities.
  2. Base Storage Support for HDFS/Azure/GCS: Expanded storage options to include HDFS, Azure Blob Storage, and Google Cloud Storage.
  3. LDAP/OAuth2 Support: Strengthened security with added support for LDAP and OAuth2 authentication methods.
  4. YARN Web UI Access: Improved monitoring and management with access to the YARN web user interface.
  5. Prometheus Metrics in Spark Jobs: Enabled Prometheus metrics for detailed monitoring within Spark jobs.
  6. Hadoop Config in Config Maps: Simplified configuration by allowing Hadoop settings to be passed to Spark jobs via Kubernetes config maps.
  7. Cluster Storages Spark Config to Spark Jobs: Streamlined job configuration with the ability to pass cluster storage settings directly to Spark jobs.
  8. Detecting Spark Driver Pod Failure: Increased reliability with new mechanisms to detect Spark driver pod failures.
  9. Fetching Metrics from the History Server: Enhanced analytics and troubleshooting with the ability to fetch metrics from the Spark history server.
  10. Tags for Jobs and Groups: Improved organization and management with tagging capabilities for Jobs and Groups.
  11. Exposing YARN Jobs for Ilum's Prometheus: Expanded monitoring options by making YARN jobs visible to Ilum's Prometheus setup.
  12. Tons of Small Improvements: Implemented numerous minor enhancements across the platform to boost performance and user experience.

Introducing Data Lineage Integration with OpenLineage and Marquez

With the integration of open lineage and Marquez, every Spark job in our system now comes equipped with built-in Data Lineage capabilities. This means you can easily track the journey of your data, understand its origins, transformations, and destinations, all within the familiar environment of Ilum.
Understanding the flow of your data is crucial for making informed decisions and ensuring data integrity. With Data Lineage seamlessly integrated into Ilum, you can now visualize and trace the path of your data with ease, empowering you to make data-driven choices confidently.
Take advantage of this powerful feature today and gain deeper insights into your data processing workflows. Enhance your data management experience with Ilum’s Data Lineage, setting a new standard for transparency and control in your Spark jobs! ✨

Enhanced Security in Ilum: LDAP Integration and OAuth2 Support

We understand the importance of strong authentication mechanisms. That's why we've introduced both internal and LDAP authentication methods. With internal authentication, you can now create a static list of users and grant them access to Ilum. On the other hand, LDAP authentication enables you to leverage your existing user directory, making user management a breeze.

We've also added OAuth2 support. With Enhanced Security, we've put your data protection first. We believe that security shouldn't be complicated, and with Ilum, it doesn't have to be. So go ahead, explore the new features, and experience the peace of mind that comes with knowing your data is in safe hands.
Planned for release in 6.1.0

New Storage Options Added: HDFS/Azure/GCS

Hey there, Ilum users!
We have some awesome news to share with you today. We've just rolled out a new feature that we know you've all been eagerly waiting for, The support for other storage options!
Up until now, Ilum has been tightly integrated with S3 interface, and we've received numerous requests from our incredible user community to expand our horizons. We've listened, and we're excited to announce that Ilum now supports not only S3, but also GCS, HDFS and Azure Blob Storage!
What does this mean for you? Well, it means you now have the flexibility to choose the storage option that best suits your needs and seamlessly connect it with Ilum. Whether you're a fan of HDFS, use Azure for your data storage, or prefer Google Cloud Storage, Ilum has got you covered.

Planned for release in 6.1.0

Version 6.0.0

Introducing Ilum UI v3: An Enhanced User Interface Experience

With this update, we've taken user feedback to heart and made significant improvements to the Ilum interface, making it even easier and more intuitive to manage your Spark sessions and jobs. We believe that a seamless user experience is the key to unlocking the full potential of Apache Spark and Kubernetes, and that's exactly what we're delivering with our enhanced UI.
Our team has been hard at work, refining every aspect of the user interface to ensure a smooth and efficient workflow. We've revamped the layout, simplified navigation, and added new features that will make interacting with Ilum a breeze. Whether you're a seasoned data scientist or just getting started with Spark, UI v3 will empower you to unleash the full power of your data with ease and confidence.

Introducing Embedded Jupyter Notebook in Ilum

We are thrilled to announce the latest update to our Ilum platform - the introduction of the "Embedded Notebook" feature! 🚀
At Ilum, we are always striving to enhance your experience and make your data analysis journey seamless. With the new Embedded Notebook, we are taking your productivity to the next level. Say goodbye to switching between multiple tools and platforms - now you can access a Jupyter notebook directly within our user-friendly web interface.
Imagine the convenience of having all your data exploration, visualization, and machine learning tasks in one place. No more juggling between different applications or wasting time on tedious setup. With our Embedded Notebook, you can seamlessly transition from exploring your data to building models, all within the familiar Ilum environment.

This feature will be a part of version 6.0.0

Version 5.2.0

Introducing Python Spark Jobs Support

We are thrilled to announce the latest enhancement to Ilum - Python Spark Jobs support! 🚀
At Ilum, we are committed to providing a user-friendly and versatile data processing platform. With this new feature, we are broadening our usability by allowing Python-based Spark jobs. Now, data scientists and developers who prefer Python can leverage Ilum to process big data, perform machine learning tasks, and more, all within their preferred language environment.
Our goal has always been to make data processing as straightforward as possible, and this update takes us one step further. By extending Ilum's capabilities to include Python, we are making our platform more inclusive and catering to the needs of a wider range of users.
With Python Spark Jobs support, you can now seamlessly integrate Python code into your Spark sessions managed through Ilum. Say goodbye to the hassle of switching between different language environments or learning new programming languages. Whether you are a Python enthusiast or have existing Python-based workflows, Ilum has got you covered.
As you know, Ilum sets a new standard in integrating Apache Spark with Kubernetes. With Python support, we are revolutionizing data processing by giving you the flexibility to choose your preferred language without compromising on performance or ease of use. Connect to your Kubernetes cluster, submit Python Spark jobs, and monitor them effortlessly using our user-friendly web interface or REST API.
Stay tuned for more exciting updates as we continue to enhance Ilum and empower you to unlock the full potential of your data with ease. We appreciate your valuable feedback and encourage you to reach out to us with any suggestions or questions.
Thank you for being a part of the Ilum community! 🌟

Introducing Airflow Integration: Streamlined Orchestration for Ilum

Hey Ilum users! We've got some exciting news to share with you today. We are thrilled to announce the latest update to our product that will take your big data tasks to new heights: the integration of Apache Airflow with Ilum!
Introducing "Airflow integration" - a game-changer feature that brings streamlined orchestration, management, and monitoring of data pipelines right to your fingertips. With just a flick of a switch, the `airflow.enabled=true` flag in our Ilum Helm chart will deploy Apache Airflow alongside Ilum, creating seamless connections and interfaces required for integration.
So, what does this mean for you? It means that complex big data tasks and workflows just got a whole lot simpler. With Airflow integration, you can easily manage and monitor your data pipelines between Airflow and Ilum, all in one place. No more jumping between different tools or drowning in a sea of logs - everything you need is now at your fingertips.
At Ilum, we believe in making your life easier. That's why we set out to create a product that eliminates the need for executing commands via CLI or spending endless hours searching for errors in logs. And now, with Airflow integration, we're taking it a step further by offering you a streamlined solution for orchestrating your data pipelines.

Version 5.0 Live NOW! 🎉😊

Introducing Ilum 5.0.

We're excited to share with you one of our biggest updates yet, packed with our most popular feature request, significant performance improvements, a revamped UI, plus plenty of additional improvements, tweaks and fixes. With this update, you can expect to see an enhanced user experience and improved performance.

Watch the video!

Some of the new features:
  • Apache Zeppelin integration
  • Jupyter integration
  • New UI
  • OpenAPI support
  • Introduced the ability to configure gRPC communication, enabling more efficient and flexible data exchange.
  • Transitioned the default Spark Docker image to, providing users with the latest and most optimized version.
  • Shifted the default communication mode to gRPC, emphasizing speed and performance.
  • Expanded offerings with the addition of Helm charts for Ilum Livy Proxy, Jupyter SparkMagic, and Zeppelin, streamlining deployment and management processes.
  • Introduced Ilum Table Format, a cutting-edge storage format designed for efficient data processing and optimized query performance, bringing innovation and enhanced capabilities to the platform.
Powered by FeedBear