After releasing transparency capabilities in GlowScript (glowscript.org) I discovered that neither transparency nor mouse picking was working on Ubuntu and the MacBook Pro running Snow Leopard (works okay on Lion). I had to work through various strange shader problems. Consider the pick machinery, as simpler than but related to the transparency case (they share the same vertex shader). In picking, each object is given a false color that is the id of that object. No lighting is invoked, so that the color of all pixels of an object represent the id of that object, and readPixels yields the id of the object that currently lies under the mouse.
Start with the vertex and fragment shaders used to make a simple opaque scene, which works on all platforms. That is, replace the pick shaders with those used for normal scenes.
In pick mode the CPU sends false color data to the GPUs. The vertex shader need not change, but the fragment shader needs to be changed to eliminate lighting calculations, since all pixels of an object need to have the same false color, representing the id of that object. A minimal change is to leave the call to the lighting calculation in place, and just change the assignment to gl_FragColor to use the color coming from the vertex shader, leaving in place the lighting calculation. (Note that the color is a varying whose value in fact doesn’t change for an object because the color is the same for all vertices in a GlowScript object.) This fails on Ubuntu, even though it’s just a change in the gl_FragColor assignment statement. I had to give up on Ubuntu, at least for now.
Because I want the pick calculation to be as fast as possible, I tried to eliminate as much unnecessary code as possible from both the vertex and fragment shaders. I fairly quickly reduced the fragment shader to the following, which works on Snow Leopard:
#ifdef GL_ES
precision highp float;
#endif
varying vec4 mat_color;
void main(void) {
gl_FragColor = mat_color;
}
I then started trying to eliminate unnecessary code from the vertex shader used for picking. The process was very frustrating. Just about any scheme continued to work on Windows, but Snow Leopard was very unforgiving. It took many hours of trial and error testing to reduce the code in the vertex shader and still have picking work on the Mac. Because depth peeling uses the same vertex shader, once I got picking to work I was also able to get depth peeling to work.
Originally I thought that the transparency problem had to do with writing and reading lots of textures, but this turned out not to be correct. The problem rather seems to have something having to do with shader compilation.
You can see demos of mouse picking and transparency here: