The Warehouse

Short reads. Big insights.
Industry trends. Thought Leadership. Opinions. Hot Tips. And so much more.
 

Jun
20
Change Data Capture and Matillion Data Loader

As enterprise data volumes continue to grow and the velocity of data change increases, companies are more challenged than ever to provide timely and complete data sets for business analysis. Change Data Capture is a methodology that has been employed for some time to effectively extract changes into target data environments in near-real time by avoiding regularly scheduled batch cycles and loading source data as it updates. The objective is to have a provisioned data store be as reflective of the current source system data as possible. There are several approaches to CDC employed by data engineers, but in the end, which is the most effective and least costly strategy that insures complete ...


May
27
Three Levels of Data Maturity – Part Two

In the first installment of this series, we examined how mid-sized enterprises can quickly get started on their journey to data maturity by implementing an operational reporting platform in as little as 4 to 6 weeks. The target users for this type of data service are the mid-line operational managers looking for actionable, tactical insight into system operations. The next user group to reach on the data maturity journey are the decision-makers at the departmental level (finance, sales, marketing, supply chain, etc) who require strategic insight for planning and resource management. The architecture required for this next level of data analytics is the 2-tiered subject-oriented data warehou...


Mar
22
Three Levels of Data Maturity – Part One

Today’s businesses continue to strive for data maturity and to create a culture that is data-driven and data-literate. But the perception remains that starting such a journey is expensive and time-consuming. There is a belief that a combination of high upstart costs and lengthy implementation time prevents the business from seeing any near-future value in its data and analytics investment. The reality, however, is that today’s cloud-based data platform technologies, specifically the triumvirate of Snowflake (data storage and compute), Matillion (data loading and transformation), and ThoughtSpot (data analytics and insight), enable rapid analytic ability with minimum initial inv...


Mar
18
3 Killers of Data Cloud Modernization Planning

The time it takes from beginning a data cloud initiative to realizing any of tangible value can be years.


After all, there is a lot to consider. First you need to ensure that there is business alignment, between the business and I.T., on what the short, mid, and long-term objectives are. Second, there is the small task of assessing your entire data estate. It is important to figure out with each vendor which modern data stack configuration, workload option, availability zone, and pricing plan is the optimal configuration for you.


Third, you need to build a migration plan, defining a strategy for each possible candidate analytical application and data set, that will drive value to the business ...


Dec
03
Creating Usage Dashboards with Snowsight

One of the challenges that many Snowflake administrators face is the daily monitoring of user and resource activity in their environment. The advantages of Snowflake's consumption-based pricing model (instantly-scalable compute sizing, usage-focused scheduling, etc.) are best employed when compute and storage activity is transparent to the business. Snowflake includes an ACCOUNT_USAGE view-based schema in the out-of-the-box Snowflake database than contains all the information related to account activity. This data can, of course, be directly queried like any other data, but wouldn't it be nice to have a single view of key metrics around data storage and activity that can be monitored day to ...


Nov
08
Pandata Group and SqlDBM: Completing the Cloud Data Landscape

Pandata Group has always embraced and promoted the effectiveness of a robust data model as a means of “mapping” the enterprise data landscape (see our blog post here). Today this is more important than ever in the increasingly expanding cloud-native data analytics universe. A good data model provides not only a guideline for data engineers, but also a means of codifying and standardizing data usage from a governance perspective. Seamless integration of a data modeling tool into a cloud-native data management platform increases efficiency, collaboration and transparency. This is why we are so excited to be named a Gold Partner with SqlDBM, the leading cloud-based modeling solution...


Oct
21
Change Enablement and The Modern Data Culture

In a previous blog post, we discussed the how an organization can foster a modern data culture that is focused, fast, flexible and fun. But building the bridge to engender that culture is easier said than done. Resistance to change is commonplace among organizations both large and small. According to a recent Gartner survey, only 20% of analytics insights will deliver business outcomes. This is not necessarily due to technical shortcomings, but rather cultural ones: lack of data literacy, siloed data processes and workflows, inadequate training, and most often, fundamental communication gaps. All of these barriers can be addressed through a systematic approach to modern data culture impl...


Oct
12
Data Modeling in the Cloud Era

We have all seen how more and more companies are moving to the cloud for their data management platforms. Snowflake, Azure Synapse, AWS Redshift, and Google Big Query are leading this charge towards low-admin, instantly scalable cloud database solutions. Accompanying this is a migration to cloud-hosted data integration and low-code ETL solutions like Matillion and Fivetran. It is tempting to assume that with all these low-overhead data management platforms the concept of data modeling may be a thing of the past, relegated to the pile of on-premise databases that this brave new world is supplanting.


In reality, data modeling is more important than ever. A key to understanding this importance i...


Sep
30
Easily Connect to Any API Source From Matillion ELT

If you are like many ETL developers you’ve struggled with an easy way to source cloud services data via REST API. Although standards are in place for REST API web services protocols, it seems that every vendor has their own variation of them, creating new challenges for each new source. Matillion’s cloud ELT product has long featured an API profile creator that sources from JSON files and creates RSD (Real Simple Discovery, an XML format) scripts for use with API query components. The effectiveness of this approach, however, is only as good as the quality of JSON files provided by the vendor.


Now, with version 1.47, Matillion introduces much more simplified functionality for extra...


Sep
27
Data Technology Needs a Data Culture


We have all heard the notion that data should be viewed as an asset, the “new oil”, a crucial business resource for the enterprise. When viewed in this context, a business’s investment in data must be assessed by the quality of its return. Measuring this return is more than just “how many cool-looking dashboards” you can now create; rather, a CTO/CDO must look beyond the technical capabilities and determine if data is having a true impact on the business culture. This means evolving a BI and Data Science community that engages employees at all levels of the organization. The path to this positive return is creating a modern data culture that permeates the enter...


Pandata GroupLess

Chicago

WeWork/ Fulton Market

220 N. Green Street

Second Floor

Chicago, IL 60607

Madison

316 W Washington Ave

Suite 525

Madison, Wisconsin 53703

Send Message