Last Friday I had the pleasure of joining the Data Sharks for an interesting conversation about the industry landscape of data management, and data pipelines in particular. We recorded it on a YouTube live stream, and you can watch the YouTube video recording here. It’s also available under Intricity101 in your favorite podcast app so you can listen on your commute or whenever you sneak in a podcast.
The Data Sharks have a loyal and diverse listener base - everyone from students to enterprise architects. Not surprisingly, we covered a lot of different territory on the show, from Etleap’s founding and early customer stories to a technical deep dive on Etleap’s VPC deployments. A couple of topics stood out for me.
Citizen data engineer
The Sharks brought up the idea of a “citizen data engineer.” Data engineering has historically been a world for specialists, often with years of very deep tool-specific technical experience. From the early founding of Etleap, we have believed that this high specialization creates a barrier to insight and data analytics, creating inherently long ETL projects when a data user needs and requests a data source.
While the citizen data engineer term is a new one for me, it does a good job of describing a trend we’ve seen at many of our customers recently. Empowering a wider community through the right tools can democratize an organization’s data integration and remove the bottleneck of a centralized team of specialists.
We often describe how Etleap ETL significantly reduces the manual coding time for data engineers. It also directly enables the ranks of these citizen data engineers, further accelerating the path to gaining insights from data.
End-to-end pipelines in the wider ecosystem
The other interesting topic that stood out for me was around the increasingly disaggregated data management ecosystem and where an end-to-end data pipeline begins and ends. The Sharks had interesting perspectives, and we had a good discussion on how pipelines integrate with business intelligence (BI), data catalog, and data quality tools. Listen in on this and other topics, and I’d love to hear other perspectives and feedback as well.
Thanks to Jared, Rich, and Arkady for a fun Friday conversation! Click here to watch on YouTube or add Intricity101 to your podcast playlist
