Airbyte, an open source data integration platform for developing ELT data pipelines, has made its cloud service generally available. The firm is also announcing a new remuneration plan for open source developers that design and maintain data connectors for the platform, as well as a new command-line interface that will be available later this week.
Airbyte’s promise is that it will make it simple for enterprises to develop durable ELT (extract, load, and transform) pipelines by allowing them to use the company’s pre-built connectors or design their own. The fast-growing company, which recently raised a $150 million Series B funding round, first launched its cloud service in a limited beta last October and has since improved a number of user experience aspects, including the onboarding process, as well as taken the time to learn how its cloud users’ needs differ from those of its open source users.
The company announced earlier this year that deployments increased by 6x in 2021, from 400 per month in Q1 to 2,500 by the end of the year. It presently has over 165 data hookups, with plans to increase that number to 500 by the end of the year.
Compensation for open source contributors is currently a hot topic. Airbyte has chosen to compensate developers who design, maintain, and fix connectors for their platform, which is an innovative concept.
“We’ve always had contributions — and a growing number of contributors every week,” Airbyte co-founder and COO John Lafleur said. “One way to make sure that part of the community is focused on what has the most impact for the user community is to incentivize them. We’ve been doing a beta test for that over the last few weeks and it’s about giving some kind of bounty, some rewards, to address some issues in priorities. Meaning getting a connector from alpha to beta and from beta to production and address what is needed to get to GA certification — or bug fixing.”
In reality, this implies that a developer can earn up to $300 for adding an API stream to a connector, with more incremental updates earning up to $200.
Tricot observed that Airbyte has two types of users who will benefit from the new command-line access. Data engineers who want programmatic access to Airbyte and data analysts who want a UI-based service are the more technical data engineers. Airbyte has always provided programmatic access, but Trico observed that the community has requested a mechanism to access the service programmatically that took into account all of the GitOps best practises. Users can now save their configuration and connection state on Git and incorporate them into their existing CI workflows using this new command line.
Looking ahead, the Airbyte team stated that it is striving to allow cloud users to manage both the data and control planes. “It’s part of our general north star forever, which is ubiquity and reliability. Ubiquity means we need to be able to access data wherever it is, whether it’s on our cloud, whether it’s on AWS, Azure, GCP or in different geographic zones,” Tricot says.
He also mentioned that while Airbyte is now focusing on data centralization into data warehouses, the next phase of the journey would focus on transferring that data into APIs to operationalize it (basically, the other side of the coin of data pipelines). For many data teams, the first stage is to lay a foundation, which is what Airbyte is doing right now, but the obvious next step is to find new uses for that data.
“With Airbyte Cloud, we remove the headache of building and maintaining custom data infrastructure by providing a simple, economical way for enterprises to move data as needed,” said Tricot. “Just as importantly, we’re able to assure that as Airbyte data connectors enter general availability, they meet quality goals, especially for the long tail of data connectors that would otherwise be orphaned, by providing compensation for maintainers. We also feel it is critical, considering the recent Linux Foundation findings that contributors not just be recognized, but be compensated.”
