Mimicking loss of precision - the problem: 0.0 – 1.0 != 0/255 – 255/255 ??
I’m having a lot of difficulty understanding the loss of precision that data stored into a texture is undergoing. I’m really disgusted with myself for not being able to figure this out on my own, but hopefully someone can enlighten me.
I’m using the gpu for data processing and understand the numbers going into the gpu will experience some precision loss and that numbers stored in textures will experience even more, however it’s not exactly happening the way I imagined it would. After having issues with a frag program I stripped the program down to expose the code that was causing my problem.
Here is the issue; I’m taking an integer from 0 – 255 and loading it into a vertex program (same number for each vertex), within the program dividing it by 255, and storing it into the color component that’s heading off to the frag program. Within the first frag program I pass the number into a texture. Now in theory I should be able to lookup the texture (the texture is set to nearest filtering) and multiply it by 255 to get my original number (seeing as the 0 – 255 integer mimics the loss of precision), but this not the case. Somewhere around 29 the number drifts enough that I’m getting 28 back. What gives?
As I said before I understand that there will be loss of precision, I just need to understand the way it’s happening so I can compensate for it.
Thanks!
-ToolChest