Home Page for the TeradataForum
 

Archives of the TeradataForum

Message Posted: Wed, 19 Jun 2002 @ 17:28:50 GMT


     
  <Prev Next>   <<First <Prev
Next>
Last>>
 


Subj:   Re: Testing Methodology
 
From:   Jose Lora

Hi Thomas.

Some of your reasons to move ETL away from the Teradata system are not really valid, for example :

  - batch windows get smaller as the value of the warehouse grows  


It always possible and recommendable to design your data warehouse without having Batch Windows as a requirement. In fact, NCR is moving in that direction with the Active Data Warehouse concept and the improvements in the loading tools. Your design will also be affected for user's Data availability requirements.


  - at the same time data volumes and business logic complexity grows  


This reason will affect both ETL strategies and I should say that an out-of-Teradata strategy will be more affected because more business logic could mean more lockup transformations and bigger data volumes will involve expensive non parallel IO to move the data required for the transformation outside Teradata


  - I purchased the database for the user and not for the ETL process  


This is a good point and the best solution is making the user a part of the ETL process and making him understand the benefits of this approach. In my experience, user are very happy to have access to the original data in the database (temporarily) to validate if the changing business rules.


  - a separate ETL server is cheaper to expand than a Teradata system  


That's true, however, an increase in CPU power for your ETL process will also increase the CPU power for the business process. In the other hand, having an ETL server that only work during batch hours is not a very good way to use that box and do not improve user response time.


  - there are high performance tools that transform extremely fast, moving more data than I have typically seen  


Sure there are, an after test some of them I couldn't find a match for the inherent parallelism of Teradata (for large data volumes)

However, there is always a good reason to make transformation outside Teradata (number or date validation are not easy on Teradata) and in this case, I would prefer using Inmod routines, Access Modules or tools like Genio (Hummingbird) that can stream data directly to Fastload (doing all sort of simple row based transformations on the fly), to avoid the expensive non parallel IO.


Regards.

Jose Lora
Systems Architect
Meredith Corporation



     
  <Prev Next>   <<First <Prev
Next>
Last>>
 
 
 
 
 
 
 
 
 
  
  Top Home Privacy Feedback  
 
 
Copyright for the TeradataForum (TDATA-L), Manta BlueSky    
Copyright 2016 - All Rights Reserved    
Last Modified: 27 Dec 2016