Home Page for the TeradataForum

Archives of the TeradataForum

Message Posted: Mon, 20 May 2003 @ 00:10:11 GMT

  <Prev Next>  

Subj:   ETL metrics using User Exits
From:   Thomas Buehlmann


At my current client we use AbInitio as an ETL tool. Depending on the sub project and its need more or less use of bteq (in AbInitio RunSQL) is made. We run bteq in btet mode. (This is typically used if the data volume is large and already in Teradata.)

My question

For the purpose of creating metrics we have to figure out how many records got inserted (INSERT INTO SELECT FROM), updated (UPDATE) or deleted (DELETE). To some extent we are interested in information pertaining to DROP table (success/failure). The original approach the client took, was to emit the same query twice, once for the purpose of metrics (using a count instead of really doing the update/insert/delete) followed by the "real" query. I would prefer not doing this and am looking into alternatives.

Here what I can see as a solution:

   A) write a user exit for bteq
   + relatively straight forward
   + good metrics collection
   + no need to redo activities
   + information can be captured as the query process processes, potential
   to redirect resulting figures
   - not easy to associate query to result in that we have the plain query
   text but not context (which part of ETL ran this?)

   B) log file scraping
   + metrics and context in one file
   - requires developers to adhere to conventions in order to support the
   scraping process (this is tricky....)
   - scraping is inherently unsafe
   - can only be done after the fact

I wonder what the audiences experience and approach to this requirement is.



Thomas Buehlmann
Phoenix, AZ 85016

  <Prev Next>  
  Top Home Privacy Feedback  
Copyright for the TeradataForum (TDATA-L), Manta BlueSky    
Copyright 2016 - All Rights Reserved    
Last Modified: 15 Jun 2023