Why COLLADA failed...

Well, after implementing FBX import to my humble software, I’m now implementing COLLADA import. And since I wasted so much time with it, why not waste some more time talking a bit why COLLADA failed…
First, it’s a format with questionable complexity and choice of XML schema. It has also questionable choices in the schema itself. However, I don’t think any of this was the main reason for the failure. Any software has some level of these issues.
The main part of the failure is that neither the original authors, nor Khronos it seems understood that the single most important factor to succeed was to properly make the project open source. Yes, there are at least 3 open source implementations of COLLADA - OpenCOLLADA, COLLADA-DOM, FCOLLADA. IMO, all of these are bad.

Why open source is so important.

  • because it’s a lot of work to get the importer/exporter framework right. Yes this is one big reason, but not the main one.
  • because COLLADA is supposed to be an interchange format so all applications must ideally use the same framework. This should have been your main goal! COLLADA has an extension mechanism, and everybody complains about it. The problem is not that mechanism, the problem is that nobody will actually implement it for other software! You need one open source library, that is good enough so everybody is using it, and want to add any custom profiles directly there, so everybody else will get the same level of support.

Why the current libraries failed.

  • FCOLLADA - abandoned long time ago
  • COLLADA-DOM - abandoned. Has different headers for 1.4 and 1.5 profiles. Uses too much external libraries (like boost). Uses DOM which is not ideal.
  • OpenCOLLADA - mainly targets export. Semi-abandoned. The import functionality is unfinished, it creates it’s own profiles. Who will read it? Even OpenCOLLADA itself doesn’t… It has just a callback - here is some data that we don’t know what do to with. Do you realize, this is the main problem, the library itself should handle these extension - this is the whole point to make things work in the same way everywhere. But the library currently doesn’t import even core things like animation clips, different effect profiles, asset descriptions, etc… It crashes on import errors, doesn’t have custom allocators, custom IO. The SAX implementation itself is questionable. It’s far from production quality. Except Blender, not sure who else is using it. Not Autodesk which have their own implementation… Not OpenSceneGraph which is using COLLADA-DOM…

Well, I don’t know the whole history, but this is the main issue - no feature complete single open source library that makes all software developers want to use it, and extend it. Ok, I understand there are at least two big issues here:

  • most developers are lazy. They want things done for them
  • most companies prefer to invest in their own closed source solutions. They also change focus, close divisions, etc. I don’t know why Sony abandoned this project or just don’t care to develop it publicly as open source. But this is a fact - no other big company will go and support it, at least to the extend it requires.

So, what is the solution - IMO - use more natural open source development, decentralize, but still provide some sort of funding and guidance. There are individual or small developers that will want to work on COLLADA, but doesn’t have the financing. I still think this will be much cheaper then getting a big company involved. There must be still someone there that actively manages the project and shows some constant progress, so people know the library is not dead and has vision for the feature. The main difficulty is how to manage proper direction of the implementation. There are already 3 big libraries, IMO we don’t need yet another one. Maybe get people involved to discuss which is the most promising one, which has the best direction, fix it and start using it as much as possible. Using different implementations everywhere is terrible specifically for COLLADA. It’s not just wasting huge effort, it’s failing to achieve the main goal - really rich interchange format.

Until this happens, I will do what the rest of the industry does - put COLLADA just to say they support it, and otherwise work mostly on their FBX implementation. For me, most promising is OpenCOLLADA, although I would probably fork it to make it a bit more usable. For example - it can create perfectly fine DOM, it’s just missing proper interface. Why this is not there? I will need also to make the SAX part more flexible. For example - I want to say - SAX parse only the meshes, for the rest I want DOM access. Or tell the library where to store the heavy source data, and access everything else in DOM fashion. The library currently loads complete objects. What happens if I want to load a scene with one huge mesh? It will basically load everything like DOM but with the terrible (for COLLADA) IWritter interface. Why it’s terrible - because COLLADA doesn’t enforce an order that is easy to load. I would prefer either a low level library that just gives me what was just parsed without allocating any memory on its own, or cleverly designed hybrid library which has very flexible way to offload memory allocations to the application, and provide easy DOM model.

[QUOTE=ivalylo;40364]Well, after implementing FBX import to my humble software, I’m now implementing COLLADA import. And since I wasted so much time with it, why not waste some more time talking a bit why COLLADA failed…
First, it’s a format with questionable complexity and choice of XML schema. It has also questionable choices in the schema itself. However, I don’t think any of this was the main reason for the failure. Any software has some level of these issues.
The main part of the failure is that neither the original authors, nor Khronos it seems understood that the single most important factor to succeed was to properly make the project open source. Yes, there are at least 3 open source implementations of COLLADA - OpenCOLLADA, COLLADA-DOM, FCOLLADA. IMO, all of these are bad.
[/QUOTE]

This shows a mis-understanding of the fundamental difference between a document format, and an API.
Fbx, FCOLLADA, COLLADA-DOM, OpenCOLLADA are APIs
COLLADA on the other hand is a document format.

APIs provide language specific (C++ for example) programmatic interface to interface with formats. Some APIs can interface with more than one format.

There is also a fbx format, but it is proprietary, un-documented and can change at any time. Some have tried to get a specificaion of FBX fomat (see FBX binary file format specification — Developer Blog for example), but the obvious intent is not to specify the format itself, but just the API. Compare with the published COLLADA specifications.

It is perfectly fine to have opinions about everything, but you should compare COLLADA format (XML, schema and all) with FBX format on one hand, and compare FBX API with the many APIs that exist to interface with the COLLADA Format. There are several APIs you have not mentioned for different languages (C#, javascript) and plenty of other tools that can be used to take advantage of the published Collada (format) specification. Yes, there is more complexity, and lots of choices, but this also enable usage outside of what is permitted by the availability of the API.

What I do not understand, is if you like the FBX API, why are you not using FBX to load / save COLLADA documents ? Especially if you already have implemented it in your application?

Why open source is so important.

  • because it’s a lot of work to get the importer/exporter framework right. Yes this is one big reason, but not the main one.
  • because COLLADA is supposed to be an interchange format so all applications must ideally use the same framework. This should have been your main goal! COLLADA has an extension mechanism, and everybody complains about it. The problem is not that mechanism, the problem is that nobody will actually implement it for other software! You need one open source library, that is good enough so everybody is using it, and want to add any custom profiles directly there, so everybody else will get the same level of support.

Why the current libraries failed.

  • FCOLLADA - abandoned long time ago
  • COLLADA-DOM - abandoned. Has different headers for 1.4 and 1.5 profiles. Uses too much external libraries (like boost). Uses DOM which is not ideal.
  • OpenCOLLADA - mainly targets export. Semi-abandoned. The import functionality is unfinished, it creates it’s own profiles. Who will read it? Even OpenCOLLADA itself doesn’t… It has just a callback - here is some data that we don’t know what do to with. Do you realize, this is the main problem, the library itself should handle these extension - this is the whole point to make things work in the same way everywhere. But the library currently doesn’t import even core things like animation clips, different effect profiles, asset descriptions, etc… It crashes on import errors, doesn’t have custom allocators, custom IO. The SAX implementation itself is questionable. It’s far from production quality. Except Blender, not sure who else is using it. Not Autodesk which have their own implementation… Not OpenSceneGraph which is using COLLADA-DOM…

Khronos was not involved in the development of the various APIs you are mentioning here, Khronos is the home for the format specification only.

Collada is more than 10 years, and had several versions, so its not a big surprise that several APIs were created and abandoned, in the same amount of time entire companies were created and disappeared, or where acquired by other companies, and their line of products cancelled. Lucky for us fbx was not a product of Softimage for instance, otherwise would be in trouble as there FBX API would have been discontinued, and the files saved in the undocumented fbx format lost. Looking at what happened in the industry I’m glad I can rely on COLLADA and other published formats to still have access to my data, and will still for many years to come.

This said, you are right that its a bit confusing, and definitely does not help adoption. So to be clear, FCOLLADA was abandonned years ago, and I had not heard of it in a while.
DOM and SAX parsers are different architectures for interfacing with XML. So those two style of APIs have their own merits.
AFAIK COLLADA-DOM is still very much in used by the robotic industry, and in particular 1.5 with the very good IK specification that fits perfectly robots (not so much video games). So if you are interested in robots that ca be defined in an an open format - GitHub - rdiankov/collada_robots: Robot Files in COLLADA 1.5 Specification

On the other hand OpenCOLLADA is a SAX implementation, and has been mostly used in importe/exporter implementation. OpenCOLLADA and the 3dsMAX and Maya import/export open source project was adopted by Khronos, and actively maintained since.

Well, I don’t know the whole history, but this is the main issue - no feature complete single open source library that makes all software developers want to use it, and extend it. Ok, I understand there are at least two big issues here:

  • most developers are lazy. They want things done for them
  • most companies prefer to invest in their own closed source solutions. They also change focus, close divisions, etc. I don’t know why Sony abandoned this project or just don’t care to develop it publicly as open source. But this is a fact - no other big company will go and support it, at least to the extend it requires.

So, what is the solution - IMO - use more natural open source development, decentralize, but still provide some sort of funding and guidance. There are individual or small developers that will want to work on COLLADA, but doesn’t have the financing. I still think this will be much cheaper then getting a big company involved. There must be still someone there that actively manages the project and shows some constant progress, so people know the library is not dead and has vision for the feature. The main difficulty is how to manage proper direction of the implementation. There are already 3 big libraries, IMO we don’t need yet another one. Maybe get people involved to discuss which is the most promising one, which has the best direction, fix it and start using it as much as possible. Using different implementations everywhere is terrible specifically for COLLADA. It’s not just wasting huge effort, it’s failing to achieve the main goal - really rich interchange format.

OpenCOLLADA - GitHub - KhronosGroup/OpenCOLLADA - seems to be what you are looking for. Looks like it is far from your expectations, but its the place you want to be looking at.

Until this happens, I will do what the rest of the industry does - put COLLADA just to say they support it, and otherwise work mostly on their FBX implementation. For me, most promising is OpenCOLLADA, although I would probably fork it to make it a bit more usable. For example - it can create perfectly fine DOM, it’s just missing proper interface. Why this is not there? I will need also to make the SAX part more flexible. For example - I want to say - SAX parse only the meshes, for the rest I want DOM access. Or tell the library where to store the heavy source data, and access everything else in DOM fashion. The library currently loads complete objects. What happens if I want to load a scene with one huge mesh? It will basically load everything like DOM but with the terrible (for COLLADA) IWritter interface. Why it’s terrible - because COLLADA doesn’t enforce an order that is easy to load. I would prefer either a low level library that just gives me what was just parsed without allocating any memory on its own, or cleverly designed hybrid library which has very flexible way to offload memory allocations to the application, and provide easy DOM model.

The community is looking forward to your contributions to the OpenCOLLADA project.

Not sure how I don’t understand the difference between a document format and API, since I explicitly talk about implementations and explicitly talk about issues of the COLLADA format itself and it’s open source implementations.

Why I even tried to use COLLADA at all? Since the reasons it was created are still valid, and rich open format with good open source implementation is still needed. Also, the FBX DAE importer has the common issue to work good only when it imports its own exports, but fails with other exporters. So it’s pointless, I would use directly FBX.

Yes, COLLADA is more open then FBX, but not by far. Since most of us are small players, that can’t really affect the format direction. If this is Autodesk or Khronos, what is the difference if you are not in these groups? It is a format that was not developed with years, so in practice - it’s worse then FBX when it comes to features. What people like me can do is just extend the schema, but we return to the original problem I described - who else will support importing it? I can go and implement my extensions in an open source implementation, and fix any kind of bugs related to that import. But to do this, it must be widely used and complete library. This is how every developer would think. I know it’s normal to have a lot of COLLADA implementations, but who will support them? This is a common issue with open source projects. I don’t mind having 10 implementations, but in most cases - 9 of them will fail, so one can succeed to be really adopted. Wasting work on 3 big project, none of which become dominant is a pointer of bad open source development to me.

“OpenCOLLADA and the 3dsMAX and Maya import/export open source project was adopted by Khronos, and actively maintained since.”

Personally, I don’t see the point for such focus in the project. Autodesk products have FBX which supports better their native formats. Any commercial application IMO, will decide to have native support via the FBX SDK, especially since OpenCOLLADA doesn’t have full feature importer. This is proven by Unity and Unreal Engine for example. Also, like you said, The FBX library already has DAE export/import. Its problem is that it’s not open source. The biggest win here would be to create good enough library that everyone (including Autodesk) will use natively and achieve consistent results across applications. Currently, you must just load and hope that different implementations are compatible. Tell me that I don’t understand what the difference is… I’m talking here pages to explain why you really need ONE dominant implementation and why the format really requires it. How Alembic succeed to be adopted for so little time and has so good support in Maya for example? I know this format has a different purpose, and yet it’s not so different. My internal format can do perfectly fine what both these formats can do.

Also I know what DOM/SAX means… To be more specific what I meant, imagine a COLLADA file with one mesh inside. Your SAX implementation will load 99% of the data at once in this case. But since the COLLADA elements may come in any order, you may need to parse the file a lot of times. See the dae2ma implementation in OpenCOLLADA for example. So you get the worse from both worlds - no memory is saved, no easy data access. Probably this is not the common case, but still, a good library should not have weak spots. Good API should be flexible.

Like you said, everyone has it’s opinion. Mine is that COLLADA has issues. I would not cry like most people against the spec and how complicated it is. IMO the format is fine and the issues come mainly from not realizing what kind of implementation and development the format requires. Adoption requires mainly good importing library, and I tried to explain what a good library means for me - actively developed, feature complete, dominant, widely used with consistent result. Things will come in that order and maybe the people responsible for COLLADA, don’t realize how important and connected is the format with the implementation that will be developed to support it… Everyone can develop a format, the difficult part is to make people like, use and support it.

Sorry for the harsh posts, but I say things like I see them. Personally, it’s not such big deal for me (FBX and Alembic do the job), it’s big deal for the industry as a whole. I would gladly contribute small things, since I’m a small guy. But like I said - this requires implementation that I would like to work with, but OpenCOLLADA is not there yet. I would prefer to hack something in-house, specifically for me, since this will be less time and less money of course.

I’ve been at work on a complete overhaul of COLLADA-DOM for a number of months now. I am surprised to see activity in this forum, and especially Remi chiming in.

I think I am making that common library. I think all of peoples’ misgivings on Collada are correct, but that it is not a bad idea at all; just so far no one has had enough at stake to carry it through, in terms of implementing it as it should be. And if there is not even a single implementation, then there is not a reason to discuss a new specification; which I believe is also very much in order. But I am rolling up my sleeves to do the job that no one else has done in 10 years first.

Anyway, I think if the Blender.org dev community really wants Collada for Blender, then when I am finished with the new COLLADA-DOM, it will be perfect for Blender. I don’t know what the titan of industry companies have in their products Collada wise, but I’d be surprised if it’s more than the rest of us. So far it’s seen as mere import/export, and to a very limited degree at that. That’s only the tip of the iceberg in terms of what is described by the specification.

What I am doing with COLLADA-DOM is really not specific to COLLADA. It’s more of a C++ oriented databinding framework, with light DOM, that is ideal for anything really. It’s actually a good way to represent semantics in software, because semantics isn’t something that should be bogged down in data-structure, it should IOW simply be described, and then based on the description, C++ configuration files can be generated, and generated C++ can do wildly different things that hand-written C++ because the generator can simply produce code that would be impossible, or absurd to do by hand. I don’t mean mere volume, but I am doing strange things with COLLADA-DOM, like every child-group and attribute is a template, with a pointer-to-member to itself, so that it knows where its object is. That is, the attribute, on its own, can find the element that it belongs to, its offset into the element, and this is a compile time calculation, a reverse this-pointer. Anyway, see the COLLADA-DOM Discussion Forum (COLLADA Document Object Model (DOM) / Discussion / Open Discussion) where I am blogging about the work.

I came to this work earlier this year, thinking COLLADA would be very mature after 10 years, to find it instead a nonviable solution. But faced with no other option, and liking the idea very much in concept, I found myself the first person to really take it seriously. I was going to write about the last few days working on the URL design, in the SF.net board just now, but I remembered replying to some posts in this forum a little while back, and thought I’d check here first, to see if someone had replied.

Remi, while you are here debating COLLADA, can you look at this (Using collada as a default format for our game engine - COLLADA - Khronos Forums) old thread. I was seeing if I could find a reference from the Sailing the Gulf book you co-wrote, and while I couldn’t, while clicking around in the old archives, I read some threads there.

I wrote Mark Barnes to ask if he remembered anything about the reference in the book (just a URL, no Title) and asked him about what you said in this other post, at the bottom, because it throws me for a loop. I think the attitude taking there, can also be the problem with COLLADA not developing into a useful tool suite, because it sets a very limited goal I think, that encourages the development of very limited, short-sighted frameworks. You speak of COLLADA as if its a one way communication protocol, where the signal is disposable. It’s a very different view and application of COLLADA than I have. (I view it as a highly extensible web of multi-representational documents, much like the WWW, that represent more-or-less permanent artifacts first and foremost, and secondarily as a handshaking mechanism. Taking the latter view, its easy to see why no one would carry it further than they need to.)