Simple Math Producing Errors

L

Lythandra

Hi,

I have some simple Math going on; just Adding and such yet I am getting some
errors where the result will be 40.0000000037287 where the result should just
be 40.

I am getting the same thing in simple multiplication where sometimes it will
inject lots of decimal places for some reason.

I have double checked everything that it is adding and there is no reason
why it should be getting any decimal places.

Anyone have any ideas why this is happening?

I can force it to 2 decimal places if I need to but I would ratherfix it at
the root of the problem if I can.

Thanks
 
L

Lythandra

'DERE BE GREMLINS IN THE ENGINE CAPTAIN, DA PLANE IS GOING DOWN!!!

AND THE GREMLINS CAN'T ADD RIGHT!!!!

:p
 
M

Marshall Barton

Lythandra said:
I have some simple Math going on; just Adding and such yet I am getting some
errors where the result will be 40.0000000037287 where the result should just
be 40.

I am getting the same thing in simple multiplication where sometimes it will
inject lots of decimal places for some reason.

I have double checked everything that it is adding and there is no reason
why it should be getting any decimal places.

Anyone have any ideas why this is happening?

I can force it to 2 decimal places if I need to but I would ratherfix it at
the root of the problem if I can.


There are no gremlins! This is a normal effect of using
floating point numbers and occurs whether you do the math
with pencil and paper or on a computer.

If you are really using only a few decimal places, you can
retain greater accuracy by using the fixed point Currency
data type instead of Double (Single is notoriously weak on
precision).
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top