Airbyte, creators of the fastest-growing open-source data integration platform, launched its cloud service for moving data, which will unify other data integration systems that companies use. It comes with pricing based on compute time, which can be 10 times less expensive, the company said.
Airbyte Cloud moves away from the industry-norm volume-based pricing, which prevents companies from replicating data sources with high volumes, like databases, because the cost becomes prohibitive.
“Typically, companies use an ETL or ELT (extract, load, transform) technology to move data from the most common APIs (application programming interface), or they build in-house scripts for the less common ones, and have yet another technology for database replication. Now, companies can do it all from Airbyte Cloud at a fraction of the cost,” said Michel Tricot, co-founder and CEO of Airbyte. “Using compute time as the basis for pricing is a well-understood concept and data processing platforms like Snowflake have already adopted that model.”
In addition to providing hosting and management, Airbyte Cloud enables companies to have multiple workspaces and access management for their teams. It also supports OAuth authentication to enable less technical users to connect their tools.
Today, Airbyte has over 5,000 companies using its technology, as compared with 250 at the end of January. The company is establishing the new standard of moving and consolidating data from different sources to data warehouses, data lakes, or databases in a process referred to as ELT. Businesses can create data pipelines from sources such as PostgreSQL, MySQL, Facebook Ads, Salesforce, Stripe, and connect to destinations that include Redshift, Snowflake, and BigQuery.
With the introduction of Airbyte Cloud as an alternative to an on-premises deployment, the company announced a new licensing model last month where the user community and maintainers of data connectors have incentives to deliver new features and bug fixes for the continuously growing list of connectors.
Recently, Airbyte reached 130 data connectors just one year after the company began operations. More and more connectors are coming as contributions from the community, which now stands for 20% of the connectors built. At this pace, Airbyte’s data integration platform will have the most connectors in the industry by the end of this year.
Using the Airbyte CDK enables users to build connectors to data sources in less than two hours, as compared with at least two days for most REST application programming interfaces (APIs). It standardises the way connectors are built, maintained, and scaled, thus commoditising data integration. There is no expectation for users to maintain the connectors they create.
With its approach, Airbyte said, it resolves two long-standing issues: First, companies always had to build and maintain connectors on their own — most “long tail” data connectors are not supported by closed-source ELT technologies, plus customisations are often required on common data connectors. Second, data teams always had to use a separate data replication system because volume-based pricing is cost prohibitive.
Open-source addresses both issues with a growing user community that supports more and more connectors that can be shared and used widely.
