Data Engineering

DBT (Data Build Tool) is a powerful framework for data transformation, but like any powerful tool, small missteps can lead to big issues — especially at scale. In this post, I’ll walk you through three common pitfalls I’ve seen in production DBT projects that can cause performance degradation, operational issues, or incorrect lineage tracking — […]

When we design an orchestration to build and populate Data Lake, It is highly recommended to build robust and scalable pipelines for the further possibility of growing & a variety of data, sources, and analytics needs. When we consider data flow in any data analytics need there is a high possibility to increase the number

Concept of SCD-2 Slowly Changing Dimension (SCD) SCD Type 1: Overwrites the existing value with the new value and does not retain history. SCD Type 2: Adds a new row for the new value and maintains the existing row for historical(inactive) and reporting purposes. SCD Type 3: Creates a new current value column in the