Hmm I am not sure I understand exactly what you ask but in case it might be that, you should know that openGL counts X and Y coordinates as strarting from the bottom left corner of the viewport as opposed to most GUI libraries so maybe you would need to invert the Y axis?
I want to check whether a triangle is blocked by other triangles or not through using Z-Buffer in OpenGL. However, it is the fact that this method is not good because of the error that converting Vertex to Pixel produces.
I know that i can use ray method to check it and accelerate it by using Octree.
But what I want to do is to check all the triangles in a dynamic rectangle by dragging mouse.
The solid triangle in the image above which is in the frontest is what i what.
by the way, the number of triangles in the rectangle is about 10^4.
http://www.opengl.org/wiki/Vertex_Transformation
explains how vertices are transformed to window space.
The final conversion is that windowCoordinate[2] gets converted to a integer and written to the depth buffer.
The conversion depends on the format of the z-buffer : 16 or 24 or 32 bit.
One for instance is: render your entire opaque scene with depth tests (LEQUAL) and writes enables. Then re-draw with depth test EQUAL and occlusion query for each object. Any object you get a non-zero occlusion query back for, you can see. Anyone you don’t, you can’t.
I can’t use occlusion query because of the version of OpenGL my hardware can support. Does other method which using some graceful algorithms on object space exist?
to find all visible triangles in opengl 1.1: you can assign a unique color value for each triangle using glColor3ub, render the screen, read the pixels. RGB value of each pixel then corresponds to the visible triangles. but be careful: the fixed pipeline should not modify the color of the triangle: so you have to disable lighting, texturing, blending etc. But this would be very slow since you would have to read the pixels from the back buffer and you would have to check each pixel (probably at least a million of them depending on your viewport size) in every iteration.
also this would not solve the issue of finding the fully visible triangles. to do that you would have to come up with a way to compute the number of pixels of a fully visible triangle (this might not be trivial since you would have to take clipping into account as well) and compare this number to the number of pixels visible for that triangle.
But you should seriously consider upgrading to a recent gpu to be able to use occlusion queries in order to solve your problem: it is the best solution offered in this page.
The method of coloring the triangle sounds good, however, as you said, it is could be trivial when solving the issue of finding the fully visible triangles.
I try to find some papers to look for other methods.
If i can not find it, i have to upgrade GPU and use occlusion queries.
