Home Page for the TeradataForum

Archives of the TeradataForum

Message Posted: Wed, 05 May 2004 @ 09:18:11 GMT

  <Prev Next>   <<First <Prev Next> Last>>  

Subj:   Re: ETL: Push vs Pull
From:   Anomy Anom

<-- Anonymously Posted: Tuesday, May 04, 2004 22:24 -->

  One problem I've seen with the PULL strategy occurs when the source database is being changed frequently: since the source DB developers don't "own" the extraction process, they may not think about the impact of their changes on it and may not communicate change information to the DW side.  

We too have seen this too. Lots of hours have been spent running around analyzing incoming data and fixing code.

  An approach which is working well for us is a "PUSH" extraction which automatically generates control files describing the data files (record count, field names and formats). The DW loading process can check these against expected values and immediately catch any unanticipated format changes.  

We recently implemented this with a 3rd party, and it works pretty well. We export data to them, with the metadata indicated above. They use our metadata to check our feed.

Then, they turn around and send the data (with some changes) back to us, this time with their own metadata. Our loading process then runs a check against their metadata.

One possible extension of this process, is that the DW loading process could automatically detect any format changes via the source metadata, and ALTER the target model accordingly.


  <Prev Next>   <<First <Prev Next> Last>>  
  Top Home Privacy Feedback  
Copyright for the TeradataForum (TDATA-L), Manta BlueSky    
Copyright 2016 - All Rights Reserved    
Last Modified: 23 Jun 2019