The Warehouse

BLOG & REsources

Sep
03
Easily Connect to Any API Source From Matillion ELT

If you are like many ETL developers you’ve struggled with an easy way to source cloud services data via REST API. Although standards are in place for REST API web services protocols, it seems that every vendor has their own variation of them, creating new challenges for each new source. Matillion’s cloud ELT product has long featured an API profile creator that sources from JSON files and creates RSD (Real Simple Discovery, an XML format) scripts for use with API query components. The effectiveness of this approach, however, is only as good as the quality of JSON files provided by the vendor.


Now, with version 1.47, Matillion introduces much more simplified functionality for extra...


Jun
22
Matillion Data Loader: The Fast, Easy (and Free!) Way to Populate Your Cloud Data Warehouse

There is no question that companies are moving their on-premises data warehouses to the cloud at an increasing pace. The benefits of a cloud data warehouse (instant scalability, minimal up-front costs, rapid deployment, ubiquitous access, etc.) are being fully realized and appreciated by a growing number of enterprises both large and small. The major players in cloud DW (Snowflake, AWS Redshift, Azure Synapse, Google BigQuery) are all vying for market share, and the customers are seeing benefits from the varying and competitive costs and features of each platform.


But how do you get your data to the cloud? Isn’t the time and cost comparable to any data load project, whether it is on-pre...


Mar
30
Lakes, Swamps, and Puddles: The "Data Wetlands" Ecosystem

If you feel like you’re “drowning” in jargon and buzzwords surrounding the recent developments in data lakes and their ilk, you are not alone. A recent TDWI survey showed rapidly increasing adoption of data lakes as a source of big data analytics, though it also revealed barriers to success and confusion around implementation value. Much of this confusion stems from myths and misperceptions around the technical and business uses of a data lake. This article will examine the proper use of a data lake, and how proper governance can prevent it from becoming the dreaded data swamp.


To be clear, a data lake is not a data management platform, in that it is not an integrated, ce...


Mar
23
Where's the "T?" A look at ETL vs. ELT

In a previous blog post, we examined the differences between traditional ETL (extract, transform and load) and ELT, where the “heavy-lifting” of data transformation is handled by the robust and scalable (usually cloud-hosted) target platform. In today’s modern cloud data warehouse environment, ELT maximizes the speed at which data is staged and ingested, while leveraging massive computing power in the cloud to cleanse, aggregate and otherwise prepare that data for general consumption. But where is the best place to manage that transformation piece? Is it using cloud-friendly ETL tools, or is it within the management consoles of the cloud DWs themselves?


A common perception a...


May
29
Talend v. Matillion for Cloud Migration

The two leading ETL/ELT tools for cloud data migration are Talend and Matillion, and both are well-positioned for moving and transforming data into the modern data warehouse. So if you’re moving to any type of cloud-hosted DW, whether it is a cloud-dedicated warehouse such as Snowflake, or part of a larger cloud platform such as AWS Redshift, Azure SQL Data Warehouse or Google BigQuery, which tool should you use to move your existing on-prem data?


Both Talend and Matillion can source any kind of on-prem data and land it in a cloud-hosted data environment. They can also move data to and from AWS’s cloud data-storage S3 as well as Azure’s Blob storage (which can be used to s...


Mar
08
Data Integration Roadmap - Part One: The Logical Data Model

Our recent blog series on the data integration portfolio introduced a variety of new architectures that help the enterprise manage their data resources, including replication, virtualization and cloud data warehousing. Organizations are now able to integrate multiple data management solutions to address a variety of business sources and requirements. But it is important to understand that the foundation of any enterprise data management portfolio remains the same . . . a roadmap to data management must be created that is independent of the underlying technology. This series of blogs will examine the three main elements of the data integration roadmap: the logical data model, master data ma...


Sep
06
The Data Integration Portfolio - Part Four: Putting It All Together (In The Cloud)

This blog series has examined the hybrid data portfolio as a mix of technologies and approaches to a data foundation for the modern enterprise. We’ve examined a variety of strategies and technologies in data integration, including virtualization, replication and streaming data. We’ve shown that there is no “one size fits all” approach to an integrated data foundation, but instead have seen how a variety of disciplines that suit specific business and technical challenges can make up a cohesive data policy.


This final chapter puts it all together under the umbrella of “time-to-value" and its importance to the agile enterprise data platform. No matter what the techn...


Jun
11
Pandata Group Snowflake Partner Announcement


temp-post-image



Pandata Group's partners help us empower our clients and deliver value from data and analytics using the best technology for their particular business needs, which is why we're excited to announce we've added Snowflake to that list. After getting to know the Snowflake team and the ins and outs of their product, we see the immediate value Snowflake can bring to clients in search of a modern data warehousing solution.



Why Snowflake?



  • A fully managed, born-in-the-cloud data warehouse that delivers power, flexibility, and simplicity.

  • Hosted on Amazon Web Services (AWS), the platform is elastic as you need it to be. Start small and scale as you grow. Conversley you can scale back, on-the-fly, as ...

Oct
10
MEET ALLEN POPPE, PRACTICE DIRECTOR - DIGITAL BUSINESS

Pandata Group is happy to announce that Allen Poppe has joined the company as Practice Director to lead the growth of our newly formed Digital Business practice. With his passion and forethought in technology along with his history working in multiple industries, he is a natural choice to head Pandata Group’s Digital Business practice and keep us at the forefront of thought leadership in the space.


We think you should get to know Allen and his role at Pandata Group. We’ve included a brief interview with Allen to highlight his goals, strategies, and the big picture perspective that he wants to deliver to local clients.



1.) What area of the Pandata Group will you be focusing on?


A. I ...


Pandata GroupLessLess
Pandata GroupLess

Madison

Pandata Group, LLC

316 W. Washington Avenue

Suite 525

Madison, WI 53704

(877) 350-5192

Madison

Pandata Group, LLC

316 W. Washington Avenue

Suite 525

Madison, WI 53704

(877) 350-5192

Less

Pandata Group © 2020

Pandata Group © 2020

Less


Chicago

WeWork/ Fulton Market

220 N. Green Street

Second Floor

Chicago, IL 60607


Chicago

WeWork/ Fulton Market

220 N. Green Street

Second Floor

Chicago, IL 60607

Less