Store Page
Clickteam Fusion 2.5
All Discussions Screenshots Artwork Broadcasts Videos Workshop News Guides Reviews

Clickteam Fusion 2.5

Store Page
Squatter 1 Jan, 2015 @ 12:08pm
Understanding Floating Point Precision and 0.1
This is just info for users with Floating Point Precision and how to avoid this issue.

I came across this many times in excel

Like all programs using this, errors can happen and some discussions on its limitations are posted below.

Essentially 0.1 is a really bad number!

It cannot be represented precisely in binary and repeats infinitely and therefore is rounded.

Try to understand the limitiation and work around it.

Here is an example that you can put into any program that returns incorrectly

=1*(0.5-0.4-0.1)

Should be 0 !!!!

Excel returns: -2.77556E-17

Fusion 2.5 Developer with Double-Precision Calculator returns: 1.49012e-009

Fusion 2.5 Standard returns: -2.77556e-17

To fix Excel: use Round() or change precision options in excel

To fix in Fusion 2.5 Developer with Double-Precision Calculator: Set format of Double-Precision Calculator to N Digits after decimal point to 2 (stops working when it was set to 7)

To fix in Fusion 2.5 Standard: use Round()

Websites with this discussion:

http://blogs.office.com/2008/04/10/understanding-floating-point-precision-aka-why-does-excel-give-me-seemingly-wrong-answers/

http://support.microsoft.com/kb/78113

https://docs.python.org/2/tutorial/floatingpoint.html
Last edited by Squatter; 11 Jan, 2015 @ 12:27am
< >
Showing 1-2 of 2 comments
Squatter 8 Oct, 2016 @ 8:18pm 
I gotta add more to this, but 0.1 is not the only culprit. 0.2 and 0.3 have this issue as well. I also assume there might be more out there but essentially it is any decimal number that cannot be represented by a set finite binary. In otherwords, the binary to represent would be infinite (similar to how 1/3 = 0.333333333 to infinity)
#1
JAGIELSKI 9 Oct, 2016 @ 6:07am 
Originally posted by Squatter:
I gotta add more to this, but 0.1 is not the only culprit. 0.2 and 0.3 have this issue as well. I also assume there might be more out there but essentially it is any decimal number that cannot be represented by a set finite binary. In otherwords, the binary to represent would be infinite (similar to how 1/3 = 0.333333333 to infinity)
That's why I was always proponent of fixed point arithmethic. Let's say we have 2 32bit unsigned integers. One represents integer part of the number and another shows decimal one. The highest number we can represent with it is 4294770687.4294770687. With two unsigned 64bit values, the maximum value we could represent, given the "point" is straight in the middle, bit-wise would be 18446744073709158398.18446744073709158398

Of course we could move "point" to be more on the decimal or integer part, so it's more precise, e.g. 32bit integer with 96bit decimal portion so we can achieve higher decimal precision at the cost of integer side of things. The only reason why fixed point math couldn't really be used well in early consoles and computers was because of the memory and architecture (cpu) limitations, when used correctly, it can be as precise as you need without problems shown in this article.
#2
< >
Showing 1-2 of 2 comments
Per page: 1530 50

Clickteam Fusion 2.5 > General Discussions > Topic Details
Date Posted: 1 Jan, 2015 @ 12:08pm
Posts: 2

Discussions Rules and Guidelines