Mimicking loss of precision - the problem: 0.0 – 1.0 != 0/255 – 255/255 ??

OpenGL OpenGL: Advanced Coding
1

Mimicking loss of precision - the problem: 0.0 – 1.0 != 0/255 – 255/255 ??

I’m having a lot of difficulty understanding the loss of precision that data stored into a texture is undergoing. I’m really disgusted with myself for not being able to figure this out on my own, but hopefully someone can enlighten me.

I’m using the gpu for data processing and understand the numbers going into the gpu will experience some precision loss and that numbers stored in textures will experience even more, however it’s not exactly happening the way I imagined it would. After having issues with a frag program I stripped the program down to expose the code that was causing my problem.

Here is the issue; I’m taking an integer from 0 – 255 and loading it into a vertex program (same number for each vertex), within the program dividing it by 255, and storing it into the color component that’s heading off to the frag program. Within the first frag program I pass the number into a texture. Now in theory I should be able to lookup the texture (the texture is set to nearest filtering) and multiply it by 255 to get my original number (seeing as the 0 – 255 integer mimics the loss of precision), but this not the case. Somewhere around 29 the number drifts enough that I’m getting 28 back. What gives?

As I said before I understand that there will be loss of precision, I just need to understand the way it’s happening so I can compensate for it.

Thanks!

-ToolChest

2

Try adding 0.5 before dividing with 255.

3

Don’t have time to think through your problem, but here is something you may not be taking into account:

Note:
values 0-255 = 256 possible values

If you convert to 0…1 float range to a 0…255 byte by multiplying by 255 the distribution of values is not even. (ie. only if the value is exactly 1.0 will you get 255 (assuming no rounding up and just truncation 0.99999 will go to 254).) In these cases I usually do:

intValue = floatvalue * 256.0f
clamp(intvalue, 0, 255)

So that if the value was exactly 1.0 it equals 255 in the clamp.

4

Thanks Guys,

I’m using rounding (don’t think I made that clear), which should be taking care of the expected drift that would cause number to end up being slightly less than or greater than the whole number. The numbers I saw were WAY off, but Humus’ idea worked, although I had to add 0.49 to prevent the numbers from swinging up to the next whole number. I’ll write an app to test out all of the numbers between 0 and 255 just to make sure.

Thanks!

-ToolChest

5

When you do texture coordinate look-up, remember that 1.0 == 0.0 (if clamp is off), so if your input is 0-255, you want to divide by 256 to actually get a 1:1 value:texel relation.