We're excited to introduce a game-changer in the world of data interaction - the brand new SQL API/UI section in Ilum! 🚀 Now, you can effortlessly run SQL queries directly on the cluster, revolutionizing the way you interact with your data. 

 

 With full SQL dialect capabilities at your fingertips, including seamless integration with modern data formats like Delta Lake, Apache Iceberg, and Apache Hudi, data querying and manipulation have never been more straightforward. Plus, beyond just SparkSQL, Ilum now supports Trino and Apache Flink, offering you a versatile toolkit to meet your diverse data processing needs. 

 

 Say goodbye to complex data interactions and hello to streamlined analytics and data management tasks directly from our platform. Whether you're diving into different data formats or exploring various processing engines, Ilum's SQL API/UI section empowers you to navigate your data environment with ease. 

 

 Join us in embracing this new era of data interaction and experience firsthand how Ilum is reshaping the landscape of data processing. Get ready to elevate your data game and unlock new possibilities with our latest SQL API/UI feature. Your data journey is about to get a whole lot smoother - start exploring today!

 

We are thrilled to announce the latest update to Ilum - the addition of more advanced spark metrics! 🚀 Now, you can dive even deeper into monitoring the performance and efficiency of your Spark jobs with enhanced metrics at your fingertips.

 

With these new metrics, you can gain valuable insights into the resource utilization, task execution, and overall health of your Spark applications. Understanding these metrics will empower you to optimize your jobs, troubleshoot any issues more effectively, and ultimately boost your productivity.

 

Whether you are a data engineer, a developer, or a data scientist, these advanced spark metrics will provide you with the visibility you need to make informed decisions and drive better outcomes. Say goodbye to guesswork and hello to data-driven insights that will take your Spark experience to the next level!

 

Keep an eye out for these new metrics in the Ilum interface, and start harnessing the power of data to supercharge your Spark jobs today. We can't wait to see how these advanced metrics will elevate your Spark experience and help you achieve even greater success in your data projects. Happy Sparking! ✨

 

We're excited to introduce a new feature that will streamline your interactive sessions on Ilum! Say goodbye to inactive sessions taking up unnecessary resources. With our latest update, we've implemented a smart solution to automatically pause interactive sessions that have been inactive for over a week.

 

Here's how it works: if an interactive session remains idle for a specified amount of time, Ilum will scale it down to zero, freeing up resources for other tasks. But don't worry about losing your progress or data. As soon as ilum-core detects any activity or a request to resume the session, it will promptly scale it back to one session, allowing you to pick up right where you left off.

 

This enhancement not only optimizes resource utilization but also ensures a smoother and more efficient experience for all Ilum users. No more manual intervention required to manage inactive sessions. Ilum takes care of it for you, so you can focus on your work without interruptions.

 

Experience the convenience of seamless session management with Ilum's latest update. Stay productive, stay efficient, and let Ilum handle the rest. We're committed to enhancing your Spark session management experience, making your workflow even more effortless and enjoyable. Try it out today and see the difference!

 

We're thrilled to introduce a game-changing feature to Ilum that will revolutionize the way you schedule and automate your Spark jobs! Say hello to our brand-new "Advanced Spark Jobs Scheduling" functionality. 🎉

 

With Ilum's cron-based scheduling, you can now effortlessly set up and manage automated Spark job executions at specific times or intervals using familiar cron syntax. This seamless integration with Kubernetes not only ensures scalability but also offers the flexibility you need to streamline your data processing tasks effectively.

 

Say goodbye to manual job triggering and hello to a more efficient workflow. Our intuitive interface empowers you to configure, monitor, and handle errors in your scheduled Spark jobs with ease. Plus, with features like retries and notifications in place, you can rest assured that your data processing will run smoothly and on time, every time!

 

Experience the convenience of automated job scheduling like never before with Ilum's latest feature. Simplify your Spark job management, boost productivity, and stay ahead of your data processing needs effortlessly. Get started today and unlock a new level of efficiency with Ilum! ✨

 

We are excited to announce a new enhancement to Ilum that will take your job management experience to the next level! Introducing the new `tag` field for jobs, designed to make job identification and organization a breeze.

 

With the addition of the `tag` field, you can now add custom tags to your jobs, helping you easily categorize and filter them based on your specific needs. Whether you want to label jobs by project, team, priority, or any other criteria, the `tag` field allows for seamless organization and quick retrieval of information.

 

We are excited to announce a major enhancement to Ilum that will revolutionize how you manage your big data environments - the official release of our Hive Metastore integration feature! 🚀 Say goodbye to the beta phase. This essential tool is now a standard part of Ilum, ready to supercharge your data management capabilities.

 

With the new Hive Metastore integration, you can efficiently handle metadata for your big data projects, ensuring a seamless way to manage schema and metadata across different storage solutions and formats. This means less time spent wrestling with data organization and more time focusing on what truly matters - deriving valuable insights from your data.

 

But that's not all - the Ilum team is already looking ahead to the future. We have ambitious plans to expand our data catalog capabilities further by integrating with Unity Catalog, Polaris, and Nessie. These upcoming additions will take Ilum to the next level, empowering you to manage and govern large-scale data across distributed environments with ease. Our goal? To keep Ilum at the forefront of data lakehouse technology, supporting your complex data operations and governance needs every step of the way.

 

Join us on this exciting journey as we continue to innovate and elevate your data management experience with Ilum. Stay tuned for more updates, and get ready to unleash the full potential of your big data projects like never before!

 

We are excited to announce the latest feature update for Ilum - Integration with Apache Sedona! 🚀

 

With this new integration, you can now leverage the power of Apache Sedona within your Spark sessions on Ilum. Apache Sedona, formerly known as GeoSpark, is a cluster computing system specifically designed for processing large-scale spatial data. The combination of Apache Sedona and Ilum's interactive Spark sessions enables seamless analysis and visualization of spatial data.

 

Whether you're working with geospatial datasets, location-based services, or any other spatial data applications, the Integration with Apache Sedona on Ilum empowers you to handle complex spatial queries and operations with ease. Say goodbye to the hassle of managing spatial data separately and welcome a more integrated and efficient workflow.

 

Ready to take your spatial data analysis to the next level? Experience the Integration with Apache Sedona on Ilum now and discover new opportunities for spatial data processing and analysis in your Spark tasks.

With the integration of open lineage and Marquez, every Spark job in our system now comes equipped with built-in Data Lineage capabilities. This means you can easily track the journey of your data, understand its origins, transformations, and destinations, all within the familiar environment of Ilum.

 

Understanding the flow of your data is crucial for making informed decisions and ensuring data integrity. With Data Lineage seamlessly integrated into Ilum, you can now visualize and trace the path of your data with ease, empowering you to make data-driven choices confidently.

 

Take advantage of this powerful feature today and gain deeper insights into your data processing workflows. Enhance your data management experience with Ilum’s Data Lineage, setting a new standard for transparency and control in your Spark jobs! ✨

 

We understand the importance of strong authentication mechanisms. That's why we've introduced both internal and LDAP authentication methods. With internal authentication, you can now create a static list of users and grant them access to Ilum. On the other hand, LDAP authentication enables you to leverage your existing user directory, making user management a breeze.

We've also added OAuth2 support. With Enhanced Security, we've put your data protection first. We believe that security shouldn't be complicated, and with Ilum, it doesn't have to be. So go ahead, explore the new features, and experience the peace of mind that comes with knowing your data is in safe hands.

 

Planned for release in 6.1.0

 

Hey there, Ilum users!

 

We have some awesome news to share with you today. We've just rolled out a new feature that we know you've all been eagerly waiting for, The support for other storage options!

 

Up until now, Ilum has been tightly integrated with S3 interface, and we've received numerous requests from our incredible user community to expand our horizons. We've listened, and we're excited to announce that Ilum now supports not only S3, but also GCS, HDFS and Azure Blob Storage!

 

What does this mean for you? Well, it means you now have the flexibility to choose the storage option that best suits your needs and seamlessly connect it with Ilum. Whether you're a fan of HDFS, use Azure for your data storage, or prefer Google Cloud Storage, Ilum has got you covered.

Planned for release in 6.1.0

 

With this update, we've taken user feedback to heart and made significant improvements to the Ilum interface, making it even easier and more intuitive to manage your Spark sessions and jobs. We believe that a seamless user experience is the key to unlocking the full potential of Apache Spark and Kubernetes, and that's exactly what we're delivering with our enhanced UI.

 

Our team has been hard at work, refining every aspect of the user interface to ensure a smooth and efficient workflow. We've revamped the layout, simplified navigation, and added new features that will make interacting with Ilum a breeze. Whether you're a seasoned data scientist or just getting started with Spark, UI v3 will empower you to unleash the full power of your data with ease and confidence.

 

We are thrilled to announce the latest update to our Ilum platform - the introduction of the "Embedded Notebook" feature! 🚀

 

At Ilum, we are always striving to enhance your experience and make your data analysis journey seamless. With the new Embedded Notebook, we are taking your productivity to the next level. Say goodbye to switching between multiple tools and platforms - now you can access a Jupyter notebook directly within our user-friendly web interface.

 

Imagine the convenience of having all your data exploration, visualization, and machine learning tasks in one place. No more juggling between different applications or wasting time on tedious setup. With our Embedded Notebook, you can seamlessly transition from exploring your data to building models, all within the familiar Ilum environment.

This feature will be a part of version 6.0.0

 

We are thrilled to announce the latest enhancement to Ilum - Python Spark Jobs support! 🚀

 

At Ilum, we are committed to providing a user-friendly and versatile data processing platform. With this new feature, we are broadening our usability by allowing Python-based Spark jobs. Now, data scientists and developers who prefer Python can leverage Ilum to process big data, perform machine learning tasks, and more, all within their preferred language environment.

 

Our goal has always been to make data processing as straightforward as possible, and this update takes us one step further. By extending Ilum's capabilities to include Python, we are making our platform more inclusive and catering to the needs of a wider range of users.

 

With Python Spark Jobs support, you can now seamlessly integrate Python code into your Spark sessions managed through Ilum. Say goodbye to the hassle of switching between different language environments or learning new programming languages. Whether you are a Python enthusiast or have existing Python-based workflows, Ilum has got you covered.

 

As you know, Ilum sets a new standard in integrating Apache Spark with Kubernetes. With Python support, we are revolutionizing data processing by giving you the flexibility to choose your preferred language without compromising on performance or ease of use. Connect to your Kubernetes cluster, submit Python Spark jobs, and monitor them effortlessly using our user-friendly web interface or REST API.

 

Stay tuned for more exciting updates as we continue to enhance Ilum and empower you to unlock the full potential of your data with ease. We appreciate your valuable feedback and encourage you to reach out to us with any suggestions or questions.

 

Thank you for being a part of the Ilum community! 🌟

 

Hey Ilum users! We've got some exciting news to share with you today. We are thrilled to announce the latest update to our product that will take your big data tasks to new heights: the integration of Apache Airflow with Ilum!

 

Introducing "Airflow integration" - a game-changer feature that brings streamlined orchestration, management, and monitoring of data pipelines right to your fingertips. With just a flick of a switch, the `airflow.enabled=true` flag in our Ilum Helm chart will deploy Apache Airflow alongside Ilum, creating seamless connections and interfaces required for integration.

 

So, what does this mean for you? It means that complex big data tasks and workflows just got a whole lot simpler. With Airflow integration, you can easily manage and monitor your data pipelines between Airflow and Ilum, all in one place. No more jumping between different tools or drowning in a sea of logs - everything you need is now at your fingertips.

 

At Ilum, we believe in making your life easier. That's why we set out to create a product that eliminates the need for executing commands via CLI or spending endless hours searching for errors in logs. And now, with Airflow integration, we're taking it a step further by offering you a streamlined solution for orchestrating your data pipelines.

 

Introducing Ilum 5.0.

We're excited to share with you one of our biggest updates yet, packed with our most popular feature request, significant performance improvements, a revamped UI, plus plenty of additional improvements, tweaks and fixes. With this update, you can expect to see an enhanced user experience and improved performance.

Watch the video!

Some of the new features: