Home Page for the TeradataForum
 

Archives of the TeradataForum

Message Posted: Mon, 16 Feb 2015 @ 14:37:43 GMT


     
  <Prev Next>   <<First <Prev Next> Last>>  


Subj:   Re: Casting to integer with ** operator resulting incorrect
 
From:   Geoffrey Rommel

The underlying math for Teradata, or for any database running on Linux, is determined by the processor and the OS. In this case, I believe what's happening is this:

select 10**4;

In Teradata Studio, this result is shown as 10000.0. The '.0' on the end tips us off that this is actually a floating-point number, so it's stored as 1.0 times 10 to the 4th power -- or something close to that. When it's cast as an integer, the result just happens to come out a little closer to 9999 than to 10000, so you get 9999. The same thing happens with 10**7 (but not 10**6 or 10**5).

select power(10,4);

This result is shown as 10000 (no trailing '.0'), so it is almost certainly an integer. Casting to an integer does not change it. Most likely the "power" function has been overloaded so that it can accept integer arguments and perform the exponentiation without converting to floating point and back.


I hope this helps.



     
  <Prev Next>   <<First <Prev Next> Last>>  
 
 
 
 
 
 
 
 
  
  Top Home Privacy Feedback  
 
 
Copyright for the TeradataForum (TDATA-L), Manta BlueSky    
Copyright 2016 - All Rights Reserved    
Last Modified: 15 Jun 2023