Fivetran, today announced product enhancements extending dbt Core with integrated scheduling and data lineage graphs. Fivetran previously announced support for dbt Core by dbt Labs, an open source transformations frameworks in the data analyst community. Today’s news signals a deeper integration of open source dbt Core into Fivetran and delivers new features to help companies simplify the complexities of the modern data stack, cut costs through ELT (extract-load-transform) automation, and accelerate data-driven decisions.
Transformations are a critical step in the ELT process, as they turn raw data into clean analytics-ready datasets for use in downstream data analytics workflows — from basic reporting to data science. Without an effective and reliable way to sanitize and standardize these datasets, companies are often unable to translate raw data into analytics-ready form.
According to the 2021 State of Data Engineers survey, nearly half of companies say key data is not usable for decision making. The same study found 68 percent of data engineers don’t have enough time to extract maximum value from existing data. Through Fivetran Transformations, Fivetran reduces the complexity of transformations by automating the process, ultimately allowing companies to have faster, more reliable access to the data they need to make data-driven decisions.
By adding integrated scheduling to Fivetran Transformations, users can now schedule their dbt Core models to run automatically following the completion of a Fivetran connector sync. This reduces data latency and speeds up the end-to-end ELT pipeline, while also helping customers save money on unnecessary compute costs by only running transformations on new or updated data.
Another way Fivetran accelerates analytics for the dbt community is with pre-built data models. Fivetran data models are packaged SQL scripts for popular data source connectors and analytics use cases that can be run in dbt to generate new reports quickly without data engineering overhead. Fivetran has released and maintains data models for over 40 data sources, with more being added on a regular basis. The full list can be found on dbt hub. Using data models and integrated scheduling together introduces new levels of automation to provide businesses with a modern and optimized approach to ELT.
“It’s very valuable to have Fivetran as one provider who can take care of both steps — getting data into the destination through its connectors, and then using dbt to transform the data in the data warehouse,” said Gustav Lindqvist, Chief Data Officer at Meditopia.
The addition of data lineage graphs delivers a visualization of dbt Core data models in Fivetran, allowing users to better track and manage their end-to-end data pipelines through enhanced data observability tooling. This eliminates the need for data analysts to comb through SQL code to determine relationships between models, and provides a visual representation that data engineers can use to map data movement throughout the transformation process. The graphs also present an easily digestible format that can be shared with data analysts and other business users across the organization for a more collaborative experience.
“Fivetran’s ability to orchestrate dbt Core models brings the E, L and T together and eliminates previous gaps in the process. Our users are clear about their need for the freshest data while controlling their transformation costs,” said Fraser Harris, vice president of product at Fivetran. “With Fivetran Transformations, our complete ELT data pipelines empower our customers to make revenue-impacting, data-driven decisions. This is fulfilling Fivetran’s mission to make access to data as simple and reliable as electricity.”
“Fivetran’s integrated scheduling is quite powerful. Given our customers consume data in the standard format, it’s very important for us to tightly couple the data sync with its transformation to avoid data being out of sync and to speed up the whole process,” said Albert Gozzi, CEO and co-founder at Aleph. “Having more control over when to run the transformations ensures fresh, timely data, and doing so within Fivetran removes the need to develop and maintain that functionality ourselves.”
Supporting more than 8,000 companies, dbt Core is one of the most popular transformations tools in the data community. dbt Core brings best practices from software development, such as testing, version control and auto-generated documentation, to make data transformation more accessible for a modern ELT approach.
