Home Page for the TeradataForum
 

Archives of the TeradataForum

Message Posted: Tue, 14 Sep 2004 @ 15:32:59 GMT


     
  <Prev Next>   <<First <Prev Next> Last>>  


Subj:   Re: Interesting Calculations If U spend time !
 
From:   paul.burns

you wrote:

  "Can anyone explain why we get an error when I do CAST with more than 6 decimal points for the above calculation?  


     SELECT 6*393.2600*1.57000*(CAST(10 AS DECIMAL(18,7))/60.00) Res - Gives an error"

The obvious answer would be in the data types. The 'first' number of the expression seems to determine the data type for the expression. Hence this being the first part SELECT 6*393.2600*1.57000 gives a result that is store as a 'smaller' data type the the second part *(CAST(10 AS DECIMAL(18,7))/60.00). Then the result of the two part is outside the range of the first data type.

You could change the first part to SELECT 6.00*393.26*1.57 and still maintain the decimal places. But that still does not give a accurate result. The result should be 393.26 * 1.57 = 617.4182.

Try this:

     SELECT (6(float))*393.2600*1.57000 * ((10 (float)) / 60 ) res

Paul



     
  <Prev Next>   <<First <Prev Next> Last>>  
 
 
 
 
 
 
 
 
  
  Top Home Privacy Feedback  
 
 
Copyright for the TeradataForum (TDATA-L), Manta BlueSky    
Copyright 2016 - All Rights Reserved    
Last Modified: 15 Jun 2023