Home Page for the TeradataForum
 

Archives of the TeradataForum

Message Posted: Tue, 28 Sep 2004 @ 14:18:12 GMT


     
  <Prev Next>   <<First <Prev Next> Last>>  


Subj:   Re: Loss of decimal precision values
 
From:   Fred Pluebell

Ah, the wonders of finite precision arithmetic...

Loss of precision is due to inability to fit the exact quotient (or mantissa for a FLOAT) into a fixed number of bits. Arithmetic properties one learns in school (commutative, associative, distributive) do not apply to finite precision calculations; order of operations matters. Often the simplest workaround is to rearrange the expression to postpone division as late as possible within the calculation. In other words, (100*x)/y is likely a better way to calculate percentage than (x/y)*100.

Also review the rules for Binary Arithmetic Results in Teradata SQL Reference V5. And remember that tools such as Queryman/SQL Assistant do their own formatting of binary numeric results - if you want Teradata to do the formatting and you're not using BTEQ in Record Mode, you have to cast the result to [VAR]CHAR.



     
  <Prev Next>   <<First <Prev Next> Last>>  
 
 
 
 
 
 
 
 
  
  Top Home Privacy Feedback  
 
 
Copyright for the TeradataForum (TDATA-L), Manta BlueSky    
Copyright 2016 - All Rights Reserved    
Last Modified: 15 Jun 2023