DATA VAULT ETL ARCHITECTURE DESIGN

Data Vaults are designed using a data modeling method that is designed to provide long-term historical storage of data coming in from multiple operational systems. Data Vaults are also especially well-suited to managing data and governance issues such as auditing, tracing the lifecycle of data and datasets, loading speed and resilience to change as well as emphasizing the need to trace where all the data in the database came from. This means that every row in a data vault must be accompanied by record source and load date attributes, enabling an auditor to trace values back to the source.

Data vault modeling makes no distinction between incoming data being 'good' or 'bad' ('bad' data meaning it doesn't conform to business rules). A data vault stores 'A single version of the facts.' Dan Linstedt, who created the Data Vault method, explains it as, "All the data, all of the time." This means also capturing data with errors and storing it exactly as it arrived. Valid data, invalid data, clean data or dirty data, it doesn't matter because everything coming in gets stored in the Data Vault. This is unlike other data analytics methods which aim to store "a single version of the truth." With traditional analytics ETL, data which does not conform to the definitions are removed or "cleansed" instead.

The modeling method is designed to be resilient to change in the business environment and the data it consumes and produces, by explicitly separating structural information from descriptive attributes. Data vault models are designed to enable parallel loading as much as possible, so that very large implementations can scale out without the need for major redesign.

Chameleon metadata's data vault information lifecycle flow

Dan Linstedt, the creator of the Data Vault method, describes the resulting database as follows:

The Data Vault Model is a detail oriented, historical tracking and uniquely linked set of normalized tables that support one or more functional areas of business. It is a hybrid approach encompassing the best of breed between 3rd normal form (3NF) and star schema. The design is flexible, scalable, consistent and adaptable to the needs of the enterprise. Click the button below

Click Here To Visit Dan Linstedt's Site & More About Data Vaults

“© 2018 - Dynamic Database Support Systems, Inc. - All Rights Reserved
Chameleon Metadata® and What Color Is Your Data® and Chameleon Metadata logos are Dynamic Database Support Systems trademarks Reproduction is prohibited without prior written permission”