It's good to be back!
Last week, several of us from the Etleap team participated in AWS re:Invent in Las Vegas... in person! After a nearly two year pause from large events, the energy and enthusiasm were high. It was a busy week for us, highlighted by Moderna’s Carlos Peralta speaking about their analytics journey with Etleap, multiple new Etleap feature announcements and AWS integrations, and countless (masked) face-to-face conversations about ETL, data lakes, and warehouses.
With serverless at the heart of many of AWS’ announcements, there was a notable focus on simplifying adoption for analytics users. That is music to Etleap’s ears as accelerating time to value and minimizing extensive coding requirements are among our core goals.
Here’s a roundup of the major Etleap activities at AWS re:Invent 2021.
Amazon Redshift Streaming Ingestion
Many customers rely on Etleap to land various data sources into an Amazon Redshift data warehouse. Historically, all Redshift customers would have to ingest data streams into S3 and then copy them into Redshift from there. At re:Invent, AWS announced that this additional step is no longer required. Redshift’s native Streaming Ingestion service now enables ingesting data directly from AWS Kinesis Data Streams into Redshift.
Thanks to our close partnership with AWS, Etleap customers will be able to ingest data into Redshift directly from an Etleap-managed Kinesis stream, as part of the AWS Streaming Ingestion Preview Program. This enables organizations to ingest hundreds of megabytes of data per second and query it at exceptionally low latency - in many cases only 10 seconds after entering the Kinesis stream.
We displayed a live demo dashboard in our booth to illustrate the new capabilities. The dashboard showed the largest stock gains of the day, but rather than querying a static data set, Etleap streamed the live NASDAQ data. Minute-by-minute, visitors could see the underlying data set grew from over 1 billion records and the top gainers update accordingly.
Moderna and Etleap
Moderna’s data engineering leader Carlos Peralta delivered a talk at re:Invent sharing how the biotech firm handled rapid data growth over the past two years. “When COVID hit and we realized our vaccine would lead to company hypergrowth, we knew we had to accelerate building our new analytics platform.”
AWS re:Invent registrants can now watch the full 15-minute recording here.
Key among Moderna’s requirements were:
- Handling many data sources
- Operating in a secure environment, and
- Minimizing data engineering staffing needs.
Peralta shared the details of Moderna’s architecture as well as their rapid progress, all accomplished while maintaining a lean data engineering team. The AWS re:Invent team will make presentation recordings available later this month in case you missed it.
AWS also announced the availability of Redshift Serverless at re:Invent last week. This is an important development for Redshift, enabling customers to gain the benefits of the data warehouse without the need to set up, manage, and tune clusters. As an Amazon Redshift Service Ready Partner, Etleap was able to quickly update Etleap ETL to ensure customers can seamlessly ingest diverse data sources in this new serverless environment.
With timesaving tools like Etleap ETL and Redshift Serverless, the time and effort barriers to building and managing a data analytics environment have fallen drastically from the past.
See you next time!
The final highlight of the week was reconnecting with customers, colleagues, and other data analytics professionals at re:Invent. If you didn’t get a chance to ask your questions or we had to cut a demo short, please contact us at firstname.lastname@example.org.