Jennifer Pip
Jennifer Pip

Data Lakes with a Purpose: What are Delta Lake and Kylo?


Kylo and Delta Lake are two open source platforms that provide management capability for building out your own data lake solution. While they are both open source there are some differences where one might fit better into an existing architecture or meet plans for a bigger picture architecture.

Delta Lake –

The Delta Lake ( Project is open source and seemingly its main contributor is DataBricks though it is actually maintained under the Linux Foundation making is a serious OSS player. It’s an at scale extendible system with ACID transaction stability and built to handle big data workloads. Because of its build out it currently ingest data through its API, stores data in object storage, and allows query output through the Delta JDBC driver. Storage options for read and write support HDFS, AWS S3 and Azure Storage options (Blob, etc.). Some might argue that previously Delta Lake seemed more like a database decoupled architecture but this might be only a near-sighted perspective on the larger set of features that Delta Lake provides such as their Time Travel feature, which is a nice feature for those who undertake AI/ML initiatives.

Our take on Delta Lake is that we are definitely fans. There are some short comings in comparison to larger platform architectures such as building a Data Lake in Google Cloud Platform or AWS with the direct connectivity and extensibility of that architecture. But if there is a small team getting off the ground and looking for an immediate solution for large data volumes that needs snapshotting, etc. like a small ML initiative, this may be a good self-contained option.

Kylo –

Kylo seeks to be a much broader end-to-end data lake management software solution. It was originally developed and released by Teradata, the well know application company with a focus on retail customers. It has many great features that we’ve implemented with success, and often not using all the features for each organization. The immediate use of ingest, streaming, wrangling, searching, and output. Kylo seems to be going strong with most of its core code written in Java while Delta Lake is mainly written in Scala which is arguably faster for development lifecycles.

Kylo has some pre-built sandbox environments for download and getting started has a bit of learning curve installing in a stand alone environment but once it is up and running you can immediately see the value with the myriad of options. Incorporating and implementing those options such as search and monitoring.

So, What’s the Verdict?

Both are compelling options and give an open source alternative to a build it yourself, cloud platform, Zaloni, Upsolver, and some others. A key focus of these systems is fast, reliable self-service. As we continue to educate a key of any data lake system is a defined purpose with context to the bigger picture of how the data lake will provide enterprise value and data access to those users that need it to create value. Both are good options but if looking to build something for the enterprise we might recommended bringing in a professional to design and hash out the purpose for selecting and/or going forward with the platforms. A green light for either is possible, but be sure it’s the right fit for what you’re doing.

More to explorer

International Women's Day 2024

International Women’s Day 2024: Empowerment and Progress

As we commemorate International Women’s Day on March 8th each year, it’s a time to honor the resilience, accomplishments, and contributions of women worldwide. In 2024, this day holds particular significance as we take stock of the strides made, acknowledge persistent challenges, and recommit ourselves to the pursuit of gender equality.

Bank grade security

5 Steps to Configure Key Pair Authentication in Snowflake

Key pair authentication is a secure way to access your Snowflake data warehouse without relying solely on traditional username and password authentication. In this step-by-step guide, we will walk you through the process of setting up key pair authentication in Snowflake. We’ll also cover how to install OpenSSL, a crucial tool for generating the necessary key pair.


Streamlining Your Bullhorn CRM: Mastering Duplicate Data Management

Discover the most effective strategies for eliminating duplicate records in your Bullhorn CRM. Duplicates can hinder your productivity, lead to data inaccuracies, and impact your relationships with clients and candidates. In this insightful session, we will guide you through best practices, cutting-edge tools, and proven techniques to ensure a clean and efficient CRM database.

Scroll to Top