Home Page for the TeradataForum
 

Archives of the TeradataForum

Message Posted: Mon, 16 Feb 2015 @ 15:29:24 GMT


     
  <Prev Next>   <<First <Prev Next> Last>>  


Subj:   Re: Casting to integer with ** operator resulting incorrect
 
From:   Dieter Noeth

Geoffrey Rommel wrote:

  select 10**4;
In Teradata Studio, this result is shown as 10000.0. The '.0' on the end tips us off that this is actually a floating-point number
 


TYPE(10**4) returns FLOAT.

I agree that, 10**4 as a float might be truncated to 9999 due to known issues for floating point numbers.


  select power(10,4);
This result is shown as 10000 (no trailing '.0'), so it is almost certainly an integer
 


TYPE(POWER(10,4)) returns NUMBER.


And now for something (not) completely different:

FLOAT has 15 digits accuracy while NUMBER should have 38.

select 2**63, power(2,32) * power (2,31), power(2,63);

     *** Query completed. One row found. 2 columns returned.
     9.22337203685478E 018   -- 2**63 -- FLOAT, of course wrong
     9223372036854775808    -- power(2,32) * power (2,31) -- NUMBER, correct
     9223372036854776000    -- power(2,63)  -- NUMBER, wrong

The internal calculation for POWER seems to based on FLOATs, too, it returns only a fake NUMBER, ouch.


Dieter



     
  <Prev Next>   <<First <Prev Next> Last>>  
 
 
 
 
 
 
 
 
  
  Top Home Privacy Feedback  
 
 
Copyright for the TeradataForum (TDATA-L), Manta BlueSky    
Copyright 2016 - All Rights Reserved    
Last Modified: 15 Jun 2023