rounding error?

K

Kevin B

Regardless of how the cell has been formatted, cell A1 still contains the
value of 5.0568725. To get a zero you would have to round the number in cell
A1 to 4 decimal positions"

=ROUND(5.0568725,4)
 
S

Steve

In my C1 formula why doesn't "round(a1,4)" calculate a value of 5.0569 so
when I subtract 5.0569 I get zero?
 
K

Kevin B

Initially I overlooked your formula when I responded the first time but I've
not been able to replicate your answer. I'm using Excel 2003 and regardless
of where I do the rounding I'm getting 0
 
J

Jerry W. Lewis

My guess is that B1 is the result of a calculation an that =(B1-5.0569) is
not zero.

Computers do finite precision math, so rounding errors can accumulate.
Consequently, with floating point values you should test whether
ABS(x-y)<epsilon, instead of whether x=y.
 
Top