Red wine and cheese V dbt and Fivetran.
Yesterday, someone from my network described dbt and Fivetran as the second-best pairing after wine and cheese. They talked to me how the two tools complement each other in their ability to streamline data modelling, analysis, and integration. This comparison highlights that, as in many things in life, finding the right combination for your modern data stack can achieve significant efficiencies in your data efforts.
DBT is an open-source command-line tool that helps data analysts and engineers to transform, model, and test their data in a consistent, repeatable way. It allows you to define and organise your data transformations using a simple, readable syntax, and to manage and version your transformations using Git. With dbt, you can perform data transformations such as filtering, joining, and aggregating, and also enables to use the power of SQL to perform complex data transformation.
Fivetran, on the other hand, is a fully managed, cloud-based data integration platform that automates the process of extracting, loading, and transforming data from various sources into a data warehouse. With Fivetran, you can connect to a wide range of data sources, including databases, cloud applications, and flat files, and replicate that data into your data warehouse in “near” real-time.
When used together, dbt and Fivetran complement each other as well as red wine and cheese. Fivetran automates the process of pulling data from various sources into a centralised data warehouse, where it can be easily accessed by dbt for further modelling, testing, and analysis. With Fivetran, you can have your data pipeline and data warehouse set up and configured, and dbt allows you to perform the data transformation on top of the data in your data warehouse, making it ready for analysis and modeling.
Data professionals can also utilise Fivetran’s connectors to pull data directly
from the sources and perform the transformation, significantly reducing the ETL load in the pipeline.
Data professionals using Fivetran can use dbt Core, which is free and open-source, and have Fivetran handle the scheduling, logging and alerting of any issues that may arise during the data transformation process.
To achieve this integration, these are the steps that are taken:
– Create models locally in a DBT Core project.
– Compile the models into SQL statements to transform the data in the data warehouse.
– Publish the project to a Git repository like Github to allow teams to collaborate.
– Fivetran pulls the most recent copy of the transformations from the Git repository.
– You schedule the execution of SQL code in the Snowflake data warehouse using Fivetran
Who else thinks dbt and Fivetran is a match made in heaven and what other modern data stack pairings do you recommend?
