dbt integration

We propose integrating dbt as a native module in Ilum. Dbt allows users to write transformations in SQL which can be managed like code. This allows for the ETL processes to live alongside data cataloging, exploring, and BI tooling. By incorporating them into a single platform, we hope to automate data engineering workflows involving data transformation (dbt), cataloging (datahub), exploration (pandas, notebooks, superset), and machine learning (spark). https://github.com/dbt-labs/dbt-core Including dbt in Ilum will cut down extra data work. - When we manage transformations as code, we get versioning, testing, and dependency management to be orderly which brings down manual errors. - The inclusion of dbt in Ilum will drive - centralization of tools - no context switching between tools for data engineers on transformations, cataloging, and analysis. - When work is version controlled, teams work better reviewing changes made and can be integrated with CI/CD pipelines. - Automated dependency tracking and testing ensured by dbt maintain the consistency of the data pipeline. It also creates a scalable and robust data infrastructure. 1. The dbt transformations and pipelines will be a module in Ilum, allowing users to build, test, and manage SQL-based transformations directly from within Ilum. - Merge transformation processes with current data catalog and BI tools for a combined experience for users. 2. The integration of native Git functionality will be provided to dbt projects, leading to all transformations versions while tracking changes through accurate means. - You can visualize and handle dependencies between different transformation steps, ensuring data integrity and lineage. 3. Let’s users initiate dbt runs from the Ilum dashboard, track job statuses, and view execution logs in real-time. - Use dbt to automatically test integrity before running the data transformations. 4. Use Ilum for all your data transformation and analysis to eliminate the distraction of switching between multiple tools. - Offer workspaces with documentation plus shared environments for easy team work and knowledge gathering. The addition of dbt as an in-built module to Ilum will ease the maintenance and version control of data transforms, improve collaboration between the data team and ensure data consistency across the board. This improvement will allow the data engineers to build dependable, scalable pipelines in a unified environment. We look forward to feedback from the community and our developers as we implement the integration.