Improved
Apache Spark 4.1.2
The bundled Spark runtime moves to 4.1.2. Every engine image across the stack moves with it. The driver, the SQL engine and the history server all run the same Spark version.
- Spark 4.1.2 across the stack: the default Spark image, external spark-submit and the Spark history server are updated together.
- Refreshed legacy line: the Spark 3.5.8 (Scala 2.12) image line is refreshed for teams still on the older runtime.
- Nessie on Spark 4.x: the Nessie Iceberg extensions now run on Spark 4.x, so Iceberg-on-Nessie works on the current runtime.