It’s been bit hard to grasp the full concept though, so I second the request for a 101-tutorial documentation. That would make the learning curve a bit easier as the Collada-specification is mostly just a reference, and leaves many things unanswered.
Yes, we do our best, but more is needed indeed.
We wrote a book to give more information about the concept and more implementation details that we are not able to have in the specification, since it has to be a reference document by design.
We are currently working on tutorials that we are planning to add in a wiki section of collada.org. But we need help with this effort.
- What is the current status with external binary files?
COLLADA users all have their own preferred binary format, and most of the time one binary format per target platform. Defining a unique format is outside of the scope of the COLLADA specification.
It’s encouraging to see, that there is at least is some sort of effort to “binarize” COLLADA and make file sizes and loading times more acceptable for larger models. However, I’m bit worried that the existing raw binary method would generate many binary files - one for each array and therefore increasing external file count per .dae a lot. I’m not 100% sure about this, so correct me if I’m wrong, but it just doesn’t sound very scalable solution.
Encouraging is the right word ! We want to encourage game developers to use COLLADA design to address their needs. To answer your question, the COLLADA DOM binary sample will create a single raw file, containing all the values stored in arrays in the document, using the ‘offset’ and ‘stride’ values of accessors.
Note that this does not address issues such as alignment, little/big endian, NaN and so forth, so the binary raw file is only usable on the same kind of platform/OS that has created it.
Maybe it would be better to solve the issue by creating a real binary-based COLLADA specification, which could be translated to XML when needed, than introducing external files and raw-data outside COLLADA. What do you think?
I am not sure what would be the advantage of this ?
This is a huge amount of work to do this. A binary format would have to be bidirectional (XML->binary->XML), lossless (64 or 128 bits floats?), cross platform (endianess, alignment, IEEE vs other encoding), and require a lot of tool writing to create a validation for which we currently can use any XML validation tools.
Maybe you are wondering about performance and size issues ?
We already made some measurements and we do not see any significant different between XML and binary implementation.
In fact, if you compare COLLADA (XML) and the interchange format used by Autodesk between Max and Maya (FBX - binary format), we have observed that the same content is twice as large saved by FBX than with COLLADA.
If you compare with Maya .ma files for large files, the COLLADA file is 1.2x to 1.8x larger than the source file. So no big difference there. Then if you use zip to make the files even smaller, you will observe that the COLLADA files compress better than the binary file, and in all our tests the zipped COLLADA file is smaller than the zipped binary file.
If you look at performance, most of the time spent in the COLLADA loader or exporter in 3dsMax or Maya is spent in the internal SDK, so even if you completely eliminate the serialization time (either in binary or XML), the export/import time will be the same.
- How is the COLLADA content pipeline supposed to work?
Most of the time COLLADA-documents are created by exporting data from a modeling tool and when exporting a new COLLADA file is created. As one of the main points of COLLADA is that external data is retained in the file and it can be processed, modified and saved by various tool, I don’t see how this fits with the fact that all the data is lost when the model is re-exported.
Indeed, if you erase the previous document with the new one, all information that were in the previous document will be lost. That’s the nature of the file system, and has nothing to do with COLLADA per say.
If you are at the origin of your content pipeline, this is not a problem. But if you are in the middle of your content pipeline, then you’ll have to import, and then export back in the COLLADA document.
The rule of thumb is that your source (format) is the place where you store all the information you need, so you do not lose the information the next time you ‘export’ from the source.
If you decide that your source is one specific tool, then you have to make sure to all your date to this tool (plug-ins), make sure that the given source format can handle all your extra data, and make sure that you always edit the data within this tool.
Another choice is to use COLLADA as your source, so you make sure that you always import/export (and not load/save) from all the tools. This way you do not have to concentrate all your data creation in one single tool, you are not limited by how much a single tool can handle, you gain a lot of flexibility and productivity. You enable simpler and more specialized tools to be used in your pipeline, you can take advantage of new releases of tools in the middle of game development (since you do not rely on their source format, and do not depend on a lot of plug-ins)
It’s your choice.
I figure that there should be some intelligent merging of files going on, but as far as I see there’s no support for merging in the COLLADA DOM or FCOLLADA. I think it would be useful have in the base libraries as that’s what everyone probably needs to have anyway.
Another way to take advantage of COLLADA is to use multiple sources and external references. You would be creating a ‘root’ document, which contains instances of geometry, materials (itself an instance of FX), animations, skin… Each of those elements are external COLLADA files that can be created by the same, or different application. This workflow has several advantages, since it allows for much faster exports, concurrent/collaborative development, utilization of several tools on the same content, ‘real-time’ partial updates and so forth.
Merging and Diff would indeed be really useful tools for COLLADA. Those would need to be graphical tools. Imagine that you can highlight the meshes that have been changed between two version, switch between the two. Imagine using this capability to merge files, just like this is done for source code development.
COLLADA and its asset tag that is available on an element basis can be used to drastically improve asset management. Unfortunately there has been little effort from the tool companies. The only reference to this I have see so far is the Pinecoast CLManager.
Automatic merging is yet another issue that has not been completely solved even for source code, so we are probably not there yet for 3D data.
This of course wouldn’t be a problem if COLLADA was used as the source format in modeling tools, but that’s unlikely to be the case in the near future or?
Yes, that is most probably unlikely for the main DCC tools that they will drop their own format and switch to COLLADA! But this is not unlikely for smaller specialized tools, and already been done for Equinox-3D for example. The author used the <extra> extensibility mechanism to add all the information needed for the tool (menu and window positions, extra geometry/shader definition…), and ditched out the original source format.
Note that it actually does not make a difference, since you are the one that make the choice of what will be the source format in your pipeline, as explained above.
In the future, we are looking at a very different model: getting rid of files.
Instead, we want to use a real-time 3D database system, where we will send updates, and get notification from. This provides the ultimate speed with:
- real-time update of the data from the modeler to the game engine
- ultimate security of data, since even if an application crashes the data is safe in the server
- collaborative development, since several artists can act on the same data at the same time.
We are actively collaborating with the Verse project and already did a prototype demonstration during GDC’07.