The Warehouse

As enterprise data volumes continue to grow and the velocity of data change increases, companies are more challenged than ever to provide timely and complete data sets for business analysis. Change Data Capture is a methodology that has been employed for some time to effectively extract changes into target data environments in near-real time by avoiding regularly scheduled batch cycles and loading source data as it updates. The objective is to have a provisioned data store be as reflective of the current source system data as possible. There are several approaches to CDC employed by data engineers, but in the end, which is the most effective and least costly strategy that insures complete ...

We have all seen how more and more companies are moving to the cloud for their data management platforms. Snowflake, Azure Synapse, AWS Redshift, and Google Big Query are leading this charge towards low-admin, instantly scalable cloud database solutions. Accompanying this is a migration to cloud-hosted data integration and low-code ETL solutions like Matillion and Fivetran. It is tempting to assume that with all these low-overhead data management platforms the concept of data modeling may be a thing of the past, relegated to the pile of on-premise databases that this brave new world is supplanting.

In reality, data modeling is more important than ever. A key to understanding this importance i...