Etleap Launches Native Amazon Redshift Console Integration

SAN FRANCISCO, Calif. – December 9, 2020 — Etleap, a leading provider of cloud-native ETL solutions for Amazon Web Services (AWS), today announced availability of its hosted Software-as-a-Service (SaaS) offering through an integration with the Amazon Redshift Console, which is now in Public Preview. This is designed to allow Amazon Redshift customers to accelerate data onboarding by ingesting data from a plethora of data sources through Etleap.

By launching this Amazon Redshift Console integration, Etleap enables data and analytics teams to deploy a cloud native ETL solution for their Amazon Redshift data warehouse with just a few clicks. After selecting Etleap in the Amazon Redshift Console, users are transferred to a setup wizard on Etleap’s site where they can select and configure data sources in a few clicks. Etleap ingests data from the data sources into the Amazon Redshift cluster, and keeps data up-to-date over time. From there, users can access their data in Amazon Redshift, and also use Etleap to model their data for analytics.

Continue reading “Etleap Launches Native Amazon Redshift Console Integration”

How Etleap Integrates with Amazon Redshift Data Sharing to Provide Isolation of ETL and BI Workloads

This article was also posted on AWS’s blog here.

By Christian Romming, Founder and CEO at Etleap
Neeraja Rentachintala, Principal Product Manager with Amazon Redshift
Jobin George, Sr. Partner Solutions Architect at AWS
Caius Brindescu, Member of Technical Staff at Etleap

Amazon Redshift is a fast, scalable, secure, and fully managed cloud data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing extract, transform, and load (ETL), business intelligence (BI), and reporting tools.

Tens of thousands of customers use Amazon Redshift to process exabytes of data per day and power analytics workloads such as BI, predictive analytics, and real-time streaming analytics.

In this post, we discuss how Etleap integrates with new data sharing and cross-database querying capabilities in Amazon Redshift to enable workload isolation for diverse analytics use cases in customer environments.

Etleap is an AWS Advanced Technology Partner with the AWS Data & Analytics Competency and Amazon Redshift Service Ready designation.

Introduction to Etleap and New Amazon Redshift Capabilities

Etleap is a fully-managed ETL solution built specifically for Amazon Web Services (AWS) that doesn’t require extensive engineering work to set up, maintain, and scale. It reduces ETL setup time from months to days, automates most maintenance work, and provides transparency and full control over data pipelines.

Etleap integrates with any data source and runs either as a hosted solution (SaaS) or inside your virtual private cloud (VPC).

The new cross-database query capability in Amazon Redshift allows customers to query across databases in an Amazon Redshift cluster using a simple three-part notation <database>.<schema>.<table>.

Customers will continue to connect to a specific Amazon Redshift database from their BI and analytics tools, and applications via JDBC/ODBC similar to how they do today. Now, however, they can seamlessly query data from all the databases they have permissions to access on the cluster without having to connect to one database at a time.

The new Amazon Redshift data sharing capability allows customers to securely and easily share data across different Amazon Redshift clusters. Data sharing improves the agility of organizations by giving them instant, granular, and high-performance access to live data across Amazon Redshift clusters without the need to manually copy or move it.

Once data is shared, customers can use cross-database query functionality to query the shared databases from the consuming clusters and join the shared data with local data on the consumer cluster.

Continue reading “How Etleap Integrates with Amazon Redshift Data Sharing to Provide Isolation of ETL and BI Workloads”

Etleap Achieves Snowflake Ready Status and Announces Snowflake ELT Features

SAN FRANCISCO, Calif. – November 18, 2020 — Etleap announced today that it has achieved the Snowflake Ready status for Snowflake Data Cloud. This recognizes that Etleap’s Snowflake integration has been successfully validated by a third party, and that Etleap adheres to Snowflake best practices for ETL/ELT tools.

Etleap also announced availability of its modeling layer for Snowflake, which takes advantage of Snowflake’s MPP (massively parallel processing) infrastructure to transform data after load in order to combine data from multiple sources and to enable faster query performance.

“We’re excited to welcome Etleap as a Snowflake Ready Technology partner,” Snowflake Director of Technology Alliances, Tarik Dwiek said. “In addition to making it easy to bring data into Snowflake from a multitude of sources, Etleap’s innovative ELT features for Snowflake are opening up new possibilities for customers to further reduce engineering effort and improve time to value for their data. We’re looking forward to seeing Snowflake customers take advantage of these features.”

Continue reading “Etleap Achieves Snowflake Ready Status and Announces Snowflake ELT Features”

Etleap launches AWS Quick Start

Etleap, a leading provider of cloud-native ETL solutions for Amazon Web Services (AWS), today announced availability of an AWS Quick Start for Etleap VPC to enable customers to create, maintain, and scale data warehouses and lakes on AWS without extensive engineering work. The AWS Quick Start automates the deployment of Etleap VPC using AWS CloudFormation and works both with deployments through AWS Marketplace and with licenses provided by Etleap.

Etleap VPC lets data teams deploy ETL pipelines inside their own Amazon Virtual Private Cloud (Amazon VPC) that source data from databases, web applications, file stores, and streams, and load data into Amazon Redshift warehouses, Amazon Simple Storage Service (Amazon S3), and AWS Glue data lakes. They can transform data using Etleap’s Data Wrangler and SQL-based Data Modeler to make the data analytics-ready.

“Customers are looking for ways to integrate their data sources quickly so they can take full advantage of Amazon Redshift’s speed and scalability,” says Chris Grusz, Director, AWS Marketplace, Amazon Web Services, Inc.. “It’s great to see Etleap making an AWS Quick Start available so teams can start ingesting data to Amazon Redshift with just a few clicks, while at the same time run their ETL pipelines inside their own Amazon VPC.”

Continue reading “Etleap launches AWS Quick Start”

Run Etleap Inside Your VPC

Today is a big day for the whole team here at Etleap! We’re announcing that we’re making Etleap VPC available on AWS Marketplace, which means customers can run Etleap inside their own AWS Virtual Private Cloud (VPC) with just a few clicks. This marks the beginning of an era where we’re not just responsible for the success of customers running pipelines on our hosted infrastructure, but also of those running Etleap pipelines on their own AWS infrastructure. This product launch has been in the making for many months, and I am very proud of what our team has accomplished!

So what is Etleap VPC and why did we build it? First I’ll make the case for the VPC SaaS model being the future of enterprise ETL. Then I’ll highlight some of the reasons why being hosted SaaS first has been tremendously beneficial to Etleap’s development as a company.

How did we get here?

Every organization that has a data warehouse or lake has to deal with ETL – after all, what good is a fancy data repository without up-to-date and clean data in it? Traditionally, ETL has been expensive and time-consuming. Learning complex ETL software, building Kimball modeling processes, and setting up compute clusters dedicated to ETL led to projects measured in months or years with 7-figure budgets. And that’s before the operational costs of handling changes and errors.

Enter the cloud. Data warehouses like Amazon Redshift and Snowflake have taken advantage of cloud computing primitives to deliver vastly superior user experiences, offering scalability and flexibility at a fraction of the cost of traditional data warehouses. Today, cloud-native ETL products tailored to these technologies and to modern data teams’ workflows are starting to eliminate the headaches of traditional ETL projects.

Continue reading “Run Etleap Inside Your VPC”

SPEEDING UP ETLEAP MODELS AT AXS WITH AMAZON REDSHIFT MATERIALIZED VIEWS

This blog post was written in partnership with the Amazon Redshift team, and also posted on the AWS Big Data Blog.

The materialized views feature in Amazon Redshift is now generally available and has been benefiting customers and partners in preview since December 2019. One customer, AXS, is a leading ticketing, data, and marketing solutions provider for live entertainment venues in the US, UK, Europe, and Japan. Etleap, an Amazon Redshift partner, is an extract, transform, load, and transform (ETLT) service built for AWS. AXS uses Etleap to ingest data into Amazon Redshift from a variety of sources, including file servers, Amazon S3, relational databases, and applications. These ingestion pipelines parse, structure, and load data into Amazon Redshift tables with appropriate column types and sort and distribution keys.

Improving dashboard performance with Etleap models

To analyze data, AXS typically runs queries against large tables that originate from multiple sources. One of the ways that AXS uses Amazon Redshift is to power interactive dashboards. To achieve fast dashboard load times, AXS pre-computes partial answers to the queries dashboards use. These partial answers are orders of magnitude smaller in terms of the number of rows than the tables on which they are based. Dashboards can load much faster than they would if they were querying the base tables directly by querying Amazon Redshift tables that hold the pre-computed partial answers.

Etleap supports creating and managing such pre-computations through a feature called models. A model consists of a SELECT query and triggers for when it should be updated. An example of a trigger is a change to a base table, that is, a table the SELECT statement uses that defines the model. This way, the model can remain consistent with its base tables.

The following screenshot shows an Etleap model with two base table dependencies.

Etleap represents their models as tables in Amazon Redshift. To create the model table, Etleap wraps the SELECT statement in a CREATE TABLE AS (CTAS) query. When an update is triggered, for example, due to base table inserts, updates, or deletes, Etleap recomputes the model table through the following code:

CREATE TABLE model_temporary AS SELECT …
DROP TABLE model;
RENAME TABLE model_temporary TO model;

Continue reading “SPEEDING UP ETLEAP MODELS AT AXS WITH AMAZON REDSHIFT MATERIALIZED VIEWS”

How Etleap automates its infrastructure process with Terraform & Ansible

Introduction

“Infrastructure as Code”, IaC, is a term every system administrator has heard by now. We can think about it as the process of managing and provisioning IT infrastructure through source-code instead of performing tasks manually. As we will explore, this helps DevOps teams efficiently and safely adapt infrastructure to meet the always-changing requirements dictated by the business. This approach helps to manage infrastructure in a way that enables the devops team to better serve the organization.

How can this paradigm help you? It encourages the adoption of software development practices like keeping infrastructure’s definition and configuration scripts in a source control system, automated code testing, and doing peer reviews. This benefits infrastructure management in numerous tried-and-true ways.

If you’re starting your journey into IaC, there are many resources you can reference to familiarize yourself with the concepts and terminology associated with this approach. Kief Morris’ “Infrastructure as Code: Managing Servers in the Cloud” is an essential book on the topic (alternatively, Martin Fowler’s blog gives a great overview).

At Etleap, we embrace IaC to build and improve our service every day. This practice helps us in our ongoing effort to make Etleap the best ETL platform it can be.

“IaC is makes it possible to effortlessly and reliably spin up any element of an infrastructure at any time, or even the entire infrastructure, in a matter of minutes.”

Let’s take a look at a few examples of how using IaC has helped Etleap build a better product.

Continue reading “How Etleap automates its infrastructure process with Terraform & Ansible”

AWS re:Invent 2019 Roundup

Materialized Views, Amazon Redshift Ready, and more!

Last week Etleap put on another exciting show at AWS re:Invent, where we announced some new features and integrations with AWS services, were interviewed by the tech experts over at “theCUBE,” hosted a session all about data lakes, and most importantly, spoke with countless attendees about ETL. Here’s a roundup of all the Etleap action you may have missed at AWS re:Invent 2019.


Etleap’s booth was a veritable oasis of ETL discussion and Etleap product demos
Amazon Redshift Launches materialized views with help from etleap

Among AWS’ numerous announcements at re:Invent this year was the availability of Materialized Views in preview on Amazon Redshift. The Materialized Views feature is designed to help customers achieve up to 100x faster query performance on analytical workloads such as dashboarding queries from Business Intelligence (BI) tools and ELT data processing. Etleap helped launch this feature by integrating it into a beta version of Etleap Models (let us know if you want to be included in the beta!) and showing that it can give an ~8x performance boost. The Redshift team showcased our results in their chalk talk on “Accelerating performance with Materialized Views.”


Yannis (seated, left) and Vuk (standing, right) from the Amazon Redshift team showcase Etleap at their Redshift Materialized Views Chalk Talk

“We are delighted to have Etleap help launch the Materialized Views feature in Amazon Redshift,” said Andi Gutmans, Vice President, Analytics, Amazon Web Services, Inc. “Amazon Redshift Materialized Views allow customers to realize a significant boost in query performance in ETL pipelines and BI dashboards. By integrating Etleap with this new functionality, customers can seamlessly get the benefits of Amazon Redshift Materialized Views without needing to make any application changes.”

You can read the full Etleap press release about Amazon Redshift Materialized Views here.

Continue reading “AWS re:Invent 2019 Roundup”

On-Demand Webinar: Etleap presents “Customer First Technology”

In this webinar, we explore how and why eMoney puts their customers first by choosing technologies that solve customer challenges and their use case for running Etleap and Looker within a highly secure VPC environment.

Ready to try Etleap for yourself? Click here to get started!

Stay tuned to this blog for more webinars and other Etleap content, and for all things ETL you’re already in the right place.

Etleap Achieves Amazon Redshift Ready designation

Recently-announced designation distinguishes Etleap on the Redshift platform

SAN FRANCISCO, Calif. – December 4, 2019 — Etleap announced today that it has achieved the Amazon Redshift Ready designation. This designation recognizes that Etleap has demonstrated successful integration with Amazon Redshift. 

Achieving the Amazon Redshift Ready designation differentiates Etleap as an AWS Partner Network (APN) member with a product integrating with Amazon Redshift and is generally available and fully supported for AWS customers. AWS Service Ready Partners have demonstrated success building products integrated with AWS services, helping AWS customers evaluate and use their technology productively, at scale and varying levels of complexity. 

“Etleap is proud to achieve Amazon Redshift Ready status,” said Christian Romming, Founder and CEO of Etleap. “Our team is dedicated to helping companies achieve maintenance-free, enterprise-grade ETL by leveraging the agility, breadth of services, and pace of innovation that AWS provides. Our status as an Amazon Redshift Ready partner shows our continued commitment to Amazon Redshift and the AWS ecosystem.”

Continue reading “Etleap Achieves Amazon Redshift Ready designation”