New Release
9 months ago

Version 6.1.0

  1. Data Lineage Support Added: Enhanced data management with new lineage tracking capabilities.
  2. Base Storage Support for HDFS/Azure/GCS: Expanded storage options to include HDFS, Azure Blob Storage, and Google Cloud Storage.
  3. LDAP/OAuth2 Support: Strengthened security with added support for LDAP and OAuth2 authentication methods.
  4. YARN Web UI Access: Improved monitoring and management with access to the YARN web user interface.
  5. Prometheus Metrics in Spark Jobs: Enabled Prometheus metrics for detailed monitoring within Spark jobs.
  6. Hadoop Config in Config Maps: Simplified configuration by allowing Hadoop settings to be passed to Spark jobs via Kubernetes config maps.
  7. Cluster Storages Spark Config to Spark Jobs: Streamlined job configuration with the ability to pass cluster storage settings directly to Spark jobs.
  8. Detecting Spark Driver Pod Failure: Increased reliability with new mechanisms to detect Spark driver pod failures.
  9. Fetching Metrics from the History Server: Enhanced analytics and troubleshooting with the ability to fetch metrics from the Spark history server.
  10. Tags for Jobs and Groups: Improved organization and management with tagging capabilities for Jobs and Groups.
  11. Exposing YARN Jobs for Ilum's Prometheus: Expanded monitoring options by making YARN jobs visible to Ilum's Prometheus setup.
  12. Tons of Small Improvements: Implemented numerous minor enhancements across the platform to boost performance and user experience.