How is OpenGL ES conformance awarded?

I’m really curious as to how the conformance testing is done and how certification is awarded. Why ?

Well, I just got a Nexus 4 with an Adreno 320 which is supposedly OpenGL ES 3 ready to find out that it doesn’t support 2 features from the standard (it could have other quirks too, these are just the ones I have found) :

  1. Something weird happens when using small non power of two render targets (this ONLY happens when the render targets are smaller than the default screen resolution of 1280x768).
  2. RGB16F render targets, it only does RGBA16F, here’s the specs on that ( post gets denied for pointing to the specs, but you can easily google them out )
  3. glCompressedTexSubImage3D doesn’t work at all . It throws an error ( <core_glCompressedTexSubImage3DOES:711>: GL_INVALID_OPERATION ) for ETC1 textures. It is the only way to update compressed 2D Texture Arrays, so you can figure out that now I can’t use this GL ES 3 feature and uploading things uncompressed (which does work) would require something like 500MB of RAM, so I don’t see that as an option.

So can someone tell me how is the Adreno 320 OpenGL ES 3 conformant when it lacks these mentioned features ? Or how is the certification being awarded if it doesn’t meet all the standard’s requirements ?

Dolphin Emulator developer here. We were one of the first GL ES 3.0 applications on the Android Play Store (maybe the first, actually), and our experience with the GL ES 3.0 drivers has been similar or even worse. After trying to get GPU drivers to fix bugs for several months and getting mostly ignored, we had to go the public shaming route with “Dolphin Emulator and OpenGL drivers - Hall of Fame/Shame” (Google it, apparently I’m not allowed to post links). Since then a few of our bugs got fixed, but there seems to be a huge problem with how drivers get certified as ES 3.0 compliant.

To stay on topic here I’m going to mostly talk about Qualcomm, but ARM Mali devices showed similar issues in their ES 3.0 drivers.

While some of the bugs were hard to reproduce and only happened in special conditions, we found a surprising amount of ES 3.0 features which were simply not implemented or completely broken. For example:

  1. Calling the textureSize function in a shader causes the Qualcomm driver to abort() when compiling the shader.
  2. Dynamically indexing a UBO causes the Qualcomm driver to fail with an “internal error” when compiling the shader.
  3. “int foo = int(float_vertex_attr)” gives consistent garbage values in foo.
  4. glGetShaderInfoLog is limited to 1023 bytes of output (which is easily exceeded because the Qualcomm shader compiler spams warnings as soon as you use a macro) and clamps compilation logs after that. Qualcomm said they were aware of this bug and that it was introduced as a “customer request”.
  5. glGetShaderiv(shader, GL_INFO_LOG_LENGTH, &length); always returns a 0 length (FIXED, but was there when the driver got certified)
  6. In some conditions glBlitFramebuffer performs a 90° rotation on the texture (bug reported by several people)
  7. Using centroid in/out in shaders completely breaks rendering and gives a black screen (FIXED, but was there when the driver got certified)

A few more (including Mali bugs) here: https://docs.google.com/spreadsheet/ccc?key=0AmmTNE31f5JTdC1CbUV0cWtObmx4dFhmNU85eHdRNFE&usp=drive_web#gid=0

Remember that we’re not Crytek or DICE: we’re a small team of open source developers working on some GPU code that’s not incredibly complex. But still we managed to hit 5+ critical bugs in Qualcomm drivers completely breaking our rendering. At this point the only things I can say about the OpenGL ES 3.0 specification is that 1) it seems completely useless; 2) it looks more like a marketing award than a proper compliance certification; 3) Qualcomm should not have it.

Could someone from Khronos enlighten us about this so-called certification process and why some ES 3.0 features seem to be completely missing from it? (and I’m not talking about extensive fuzzing or random testing - just compiling a shader with “textureSize(…)” in it doesn’t look like something that should be missed)

Sorry I missed cippyboy’s original post back in September.

Not to bore you with details, but conformance testing works like this: Adopters port the test to their platform, run it, and submit the log files to a review committee. The committee has 30 days to flag any problems. If no flags are raised, the Adopter can say the implementation is conformant. In practice, maybe 5% of submissions get rejected at this stage. In theory, someone could submit a fake log file, but that would violate the Adopter’s Agreement (which is a contract), and might constitute fraud. I seriously doubt that anyone’s doing that.

As to what we’re trying to achieve: The CTS isn’t meant to test the robustness of implementations - it doesn’t do randomized testing, and it doesn’t do exhaustive testing of weird combinations of state. It does aim to verify that every entry point is present and accepts all legal arguments, and that it does the right thing up to the level of precision mandated by the spec, on a reasonable selection of cases. As of ES 3.0, it also tries to verify that the implementation throws errors when it should - previous versions didn’t do that.

How good is it? It’s non-trivial (about 400K LOC), and it definitely catches a lot of bugs. But it’s far from comprehensive. Shading language coverage is quite weak, not at all systematic. The next revision (coming very soon) includes a huge improvement in language coverage, contributed by Khronos member drawElements. (No, I don’t work for them, just grateful for their help.)

Let’s talk about cippyboy’s issues:

  1. Small NPOT render targets: The CTS tests four render target sizes, one being 113x47. Since we do test this case, I’d call this an implementation bug (robustness issue) rather than a CTS coverage hole.

  2. RGB16F render targets: See ES 3 spec, section 3.8.3, “Required Texture Formats”. RGB16F is a required texture format, but is not renderable. So Qualcomm’s driver is correct on this one, unless they support an extension that claims to make it renderable, such as EXT_color_buffer_half_float. Note that EXT_color_buffer_float does NOT make RGB16F renderable.

  3. CompressedTexSubImage3D with ETC1 textures: I assume you mean ETC2? It looks like there’s a test case for CompressedTexSubImage3D and ETC2 with array textures created by glTexStorage3D, but I don’t see one for textures created with CompressedTexImage3D. So yes, this might be a coverage hole. We definitely prioritized testing the TexStorage API since that is now the preferred way to create a texture.

And delroth’s:

1), 2), 3), 7) are language bugs, where I admit our coverage is currently terrible. We hope the new drawElements tests will catch these.

  1. sounds pretty annoying, but it isn’t a conformance failure - the ES 3 spec doesn’t mandate the contents of info logs.

  2. is legal by itself, but if you then call GetShaderInfoLog and it gives you a different length, I would agree that it’s a bug. I’m pretty sure we don’t test for that.

  3. is very interesting, would really like to know why we aren’t catching this one. There test cases for BlitFramebuffer, but I have not stepped through to see how they validate orientation of the resulting image. Did you report this to Qualcomm? Maybe we can get a test case from them.

OK, winding this up, I’ll say this on behalf of the working group: We do understand that implementation variability and bugs are a serious problem for developers. We take this very seriously, and making it better is a very high priority. The best thing you can do to help us is to file bug reports on issues like this at the Khronos public bugzilla . Reports of out-of-spec behaviors of supposedly conformant implementations are definitely fair game for bug reports.

When you file bugs, the usual rules apply: generalities like “shading language coverage is terrible” don’t help much. Specific bugs like “On allegedly conformant device <x>, the following FramebufferBlit call produces a result that is rotated by 90 degrees” help a lot, especially if accompanied by (say) a minimal code example.

One last thing - both of you reported issues with Qualcomm drivers, and I’m sure they have their share of bugs. But they also deserve credit - they have been strong supporters of the CTS program, and have put a lot of resources into managing test development and improving the tests. (No, I don’t work for them either, but I’m grateful for their help too.)

First of all, thanks a lot for your reply.

[QUOTE=eswg_chair;30020]Sorry I missed cippyboy’s original post back in September.

Not to bore you with details, but conformance testing works like this: Adopters port the test to their platform, run it, and submit the log files to a review committee. The committee has 30 days to flag any problems. If no flags are raised, the Adopter can say the implementation is conformant. In practice, maybe 5% of submissions get rejected at this stage. In theory, someone could submit a fake log file, but that would violate the Adopter’s Agreement (which is a contract), and might constitute fraud. I seriously doubt that anyone’s doing that.

As to what we’re trying to achieve: The CTS isn’t meant to test the robustness of implementations - it doesn’t do randomized testing, and it doesn’t do exhaustive testing of weird combinations of state. It does aim to verify that every entry point is present and accepts all legal arguments, and that it does the right thing up to the level of precision mandated by the spec, on a reasonable selection of cases. As of ES 3.0, it also tries to verify that the implementation throws errors when it should - previous versions didn’t do that.

How good is it? It’s non-trivial (about 400K LOC), and it definitely catches a lot of bugs. But it’s far from comprehensive. Shading language coverage is quite weak, not at all systematic. The next revision (coming very soon) includes a huge improvement in language coverage, contributed by Khronos member drawElements. (No, I don’t work for them, just grateful for their help.)[/QUOTE]

Thanks for that information. Regarding updates to the CTS: does that mean new driver releases will have to pass the new CTS to keep ES 3.0 conformance status? Is there any pressure on Adopters to prioritize fixing new CTS bugs when a new CTS is released?

[QUOTE=eswg_chair;30020]Let’s talk about cippyboy’s issues:

  1. Small NPOT render targets: The CTS tests four render target sizes, one being 113x47. Since we do test this case, I’d call this an implementation bug (robustness issue) rather than a CTS coverage hole.

  2. RGB16F render targets: See ES 3 spec, section 3.8.3, “Required Texture Formats”. RGB16F is a required texture format, but is not renderable. So Qualcomm’s driver is correct on this one, unless they support an extension that claims to make it renderable, such as EXT_color_buffer_half_float. Note that EXT_color_buffer_float does NOT make RGB16F renderable.

  3. CompressedTexSubImage3D with ETC1 textures: I assume you mean ETC2? It looks like there’s a test case for CompressedTexSubImage3D and ETC2 with array textures created by glTexStorage3D, but I don’t see one for textures created with CompressedTexImage3D. So yes, this might be a coverage hole. We definitely prioritized testing the TexStorage API since that is now the preferred way to create a texture.

And delroth’s:

1), 2), 3), 7) are language bugs, where I admit our coverage is currently terrible. We hope the new drawElements tests will catch these.

  1. sounds pretty annoying, but it isn’t a conformance failure - the ES 3 spec doesn’t mandate the contents of info logs.

  2. is legal by itself, but if you then call GetShaderInfoLog and it gives you a different length, I would agree that it’s a bug. I’m pretty sure we don’t test for that.

  3. is very interesting, would really like to know why we aren’t catching this one. There test cases for BlitFramebuffer, but I have not stepped through to see how they validate orientation of the resulting image. Did you report this to Qualcomm? Maybe we can get a test case from them.[/quote]

ACK. Regarding 6), fished for more details: what happens is that blitting a framebuffer to the backbuffer does not work properly (rotated output) when the phone is oriented in landscape mode (vs. portait mode). See https://developer.qualcomm.com/comment/6997#comment-6997 for a few people reporting that issu

[QUOTE=eswg_chair;30020]OK, winding this up, I’ll say this on behalf of the working group: We do understand that implementation variability and bugs are a serious problem for developers. We take this very seriously, and making it better is a very high priority. The best thing you can do to help us is to file bug reports on issues like this at the Khronos public bugzilla . Reports of out-of-spec behaviors of supposedly conformant implementations are definitely fair game for bug reports.

When you file bugs, the usual rules apply: generalities like “shading language coverage is terrible” don’t help much. Specific bugs like “On allegedly conformant device <x>, the following FramebufferBlit call produces a result that is rotated by 90 degrees” help a lot, especially if accompanied by (say) a minimal code example.[/quote]

We have been reporting these issues to Adopters directly in the past (with variable success), but we will try reporting these issues to KHR as well when appropriate.

What exactly do you consider out-of-spec? For example, on Mali and Adreno, using glBufferSubData repeatedly causes OOM issues. On Mali, using glClear to clear color can take up to 10-20ms to complete because the driver tries to enforce vsync (even when vsync is explictly disabled). These are not directly conformance issues but still make chunks of the standard unusable.

That’s good, but as an ES 3.0 developer Qualcomm was and is still by far the worst offender regarding ES 3.0 driver bugs. The fact that it’s incredibly hard to get them to ACK and fix bugs does not really help. It’s kind of incredible that our main source of support regarding Qualcomm drivers was not Qualcomm themselves (we tried!) but Rob Clark from the Freedreno project, an alternative Adreno driver.

This topic was automatically closed 183 days after the last reply. New replies are no longer allowed.