New
Feature
File Analytics (from preview → quick checks → generated SQL)
You can now run lightweight analytics on a file directly from the preview modal and promote it to a table without leaving ILUM.
What you get
-
Create Table (generated SQL):
- Infers schema and sample values.
- Generates CREATE TABLE + INSERT … SELECT SQL (Delta/Iceberg/Hudi or plain Parquet/CSV).
- One click to Open in SQL Editor for execution and scheduling.
-
Analyze Data:
- Quick row count and column sniffing to confirm delimiter, header, and basic types.
- Shows the exact SQL used so results are reproducible.
-
Data Profile:
- Column-level stats: distincts, nulls, min/max, quartiles/IQR, skewness/kurtosis, top values, simple histograms.
- All profiling queries are visible as SQL and can be copied or opened in the editor.
-
Quality Checks:
- Summary dashboard (issues by type/severity, completeness by column, type distribution).
- Flags typical problems (missing values, mixed types, constant columns, weak IDs) and surfaces suggestions.
- Outputs the SQL used for checks so you can version it alongside your pipelines.
-
Spark Optimizations:
- Contextual hints and config snippets for faster loads (e.g., repartition/coalesce, AQE).
How to use
- Open File Manager → select a file → Preview.
- Use the top tabs: Create Table, Analyze Data, Data Profile, Quality Checks.
- When ready, click Open in SQL Editor to run or schedule the generated statements.
Operational notes
- Profiling/quality checks default to sampling to keep runs fast.
