Data Integration Architecture
14. May 2008 by Rick Abbott.
In the “causes of failure” department, the lack of a sound architecture is a close second only to the lack of a sound strategy. And only because the go/no-go decision should be made in the strategy phase, where a number of failed projects should have been weeded out before they ever got started.
I won’t bore you with the details of a data integration (or data management, data warehouse, etc…) architecture - you can download my white paper if you want to dive deeper. First I’ll touch on the key elements of a good architecture, then list the top 3 mistakes:
Key Elements:
- Data acquisition - this includes identification of sources, integration approach (virtual vs physical), data quality processing, and any other step needed to gather data and prepare it for use in an analytical capacity.
- Data storage - Physical storage includes the old standbys (operational data stores, data warehouse, data marts, etc…) although these distinctions are blurring. Virtual storage includes enterprise information integration (EII) and other methods that provide a virtual view of data across disparate underlying physical stores.
- Delivery encompasses all forms of information dissemination, including traditional forms of business intelligence such as reporting, OLAP, and dashboard. Also included is integration into other applications such as a marketing automation system. Often overlooked is the integration back into transactional systems to provide real-time analytics - read The New Age of Innovation(if you haven’t already) to get a view of the value of analytics through the business lens.
- Meta data - this is the glue that holds the whole thing together. When done properly, the first three categories are driven off an integrated meta data repository that supports the development, operations, and end user communities.
Top 3 mistakes:
- Not defining the architecture - just like building a house, you must plan out the solution end-to-end if you want all the pieces to fit together.
- Taking a product vendor driven approach - buying a tool does not translate into defining an architecture. The technical architecture should precede and transcend the toolset. Buy tools that fit into your architecture, not the other way around.
- Technology for technologies sake - the term “business alignment” is overused and often misunderstood, but the technical implementation should operate within the parameters set during the strategy process.