SOI, Data Integration | 20 Sep 2007 4:50 PM | |

One Size Fits All Data Integration by billm |

There are about as many different ways of building data integration as there are stars in the sky. OK, that may be just a little bit of an exaggeration. There are a lot of different methods of implementing data integration. Variety might be the spice of life, but it can also get a little confusing. In the case of software design, too much variety can drive up costs and drive down quality. It is helpful when there are standard, one size fits all ways of doing things; thus the now ubiquitous use of things like relational databases and application servers. There are so far no such standard, widely adopted, methods for implementing data integration; there should be.

But hold on you say. There are a lot of different data integration methods because there are a lot of different kinds of data integration problems. There are Extract Transform and Load (ETL) problems where you have to grab a snapshot of data, or an archive of data, and load it into warehouse, or migrate it to a new system with different structures and formats. There are Enterprise Information Integration (EII) problems where multiple applications exchange operational data. There are Enterprise Application Integration (EAI) problems where transactions occur across multiple applications and systems, where transaction integrity is the key. Sometimes these problems are big and enterprise-wide, so they justify those six-figure and multi-person-year investments in those integration platform products from those big “Enterprise 1.0″ software product companies (you know who you are). Often they are a little smaller, let’s say project-based, and can’t justify the big investment and long learning and implementation cycles for those big products. In this majority of cases, solutions are still implemented with code. Data integration shows up in a very large percentage of software projects in some fashion. That is a lot of projects and lots of variations on the data integration theme. This is why it seems there are about as many ways of solving data integration as there are stars in the sky. This can and should change.

Two years ago Data Management Review attempted to explain the varying approaches in a column titled “Data Integration: The Advent of E3″. Good analysis for 2005, but things are changing fast. Emerging Web 2.0 and SOA standards support fast, reliable, re-usable ways of exchanging data. The cost of acquiring products from new and emerging “Enterprise 2.0″ software vendors, offering broad and deep, “multi-purpose”, data integration functionality, is dropping far and fast versus the old line six-figure solutions from the Enterprise 1.0 vendors. These new products are easy to learn and use, easier even than writing code to solve data integration problems, even if you already know how to write that code, and much cheaper to maintain thanks to standards.

One size fits all, inexpensive, easier, standard, lower cost data integration is just around the corner; and it’s about time!

del.icio.us · digg this · spurl · reddit · furl this