Do the below points sound familiar? We have more data than ever, yet we still don’t have clarity. Why don’t two reports show the same number? We invested millions in data and cloud… Why are decisions still slow? This is the paradox of the modern enterprise: Data is abundant. Trust is scarce That is exactly […]
Data Governance: Stop Treating It as an Afterthought Read More »
Quantum computing will shatter the cryptographic foundations of today’s digital systems. Every encrypted data lake today could become readable tomorrow. Quantum computers won’t knock, they’ll break in. Are you ready?
Is Your Data Lake Quantum-Safe? Read More »
Overview This blog post outlines a Proof of Concept (PoC) for migrating Databricks resources from Azure to AWS using Terraform. While still requiring some manual adjustments, this approach simplifies much of the resource migration using the experimental exporter from the Terraform Databricks Provider. We also touch upon best practices, additional resource migrations (like Key
Breaking Cloud Boundaries: A Terraform Story of Databricks Migration Azure -> AWS Read More »
DBT (Data Build Tool) is a powerful framework for data transformation, but like any powerful tool, small missteps can lead to big issues — especially at scale. In this post, I’ll walk you through three common pitfalls I’ve seen in production DBT projects that can cause performance degradation, operational issues, or incorrect lineage tracking —
Common Mistakes in DBT That Can Lead to Performance and Operational Issues Read More »
When we design an orchestration to build and populate Data Lake, It is highly recommended to build robust and scalable pipelines for the further possibility of growing & a variety of data, sources, and analytics needs. When we consider data flow in any data analytics need there is a high possibility to increase the number
Data Lake Orchestration: A robust and scalable approach Read More »
Concept of SCD-2 Slowly Changing Dimension (SCD) SCD Type 1: Overwrites the existing value with the new value and does not retain history. SCD Type 2: Adds a new row for the new value and maintains the existing row for historical(inactive) and reporting purposes. SCD Type 3: Creates a new current value column in the
Case Study Cost-Efficient Data Engineering Optimizing Snowflake Costs for High-Volume Airflow-DBT PipelinesAuthor – Arpan Shah Overview A client was experiencing high Snowflake costs due to inefficiencies in their DBT jobs and warehouse configurations. With over 500+ daily DBT jobs, running for approximately 13 hours with a maximum parallel execution of 39 jobs, optimizing performance while
Cost-Efficient Data Engineering Read More »
