Improved
Apache Spark 4.1.1
The bundled Spark runtime moves to 4.1.1. Every engine image across the stack moves with it. The driver, the SQL engine and the history server all run the same Spark version. Your jobs, your SQL and your run history stay on one consistent, current line.
- Spark 4.1.1 across the stack: the default Spark image, external spark-submit and the Spark history server are all updated together.
- Matching engines and Delta: the Kyuubi SQL engine and the Spark launcher images are bumped in step, with Delta Lake 4.2.0 matched to Spark 4.1.
- Refreshed capabilities: geospatial support (Apache Sedona) and the bundled storage connectors are updated for Spark 4.1.
When the driver, the SQL engine, the history server and Delta drift apart in version, you get subtle, hard-to-reproduce failures. Moving them together keeps the whole stack on one consistent, supported Spark version.
Links: Apache Spark on Ilum
Available in version: 6.7.2
Links: Apache Spark on Ilum
Available in version: 6.7.2