Collada DOM Integration Templates

We’re approaching the next release of the Collada DOM, and we’re considering taking out the integration templates. They’re not particularly useful and I’ve always recommended that people avoid them. In fact it’s somewhat unfortunate that they were ever included in the DOM. If you use these classes, please reply to this post.

:shock: I am currently using them to write a Blender Importer, is integration going to be phased out altogether or is there going to be a replacement for it?

Sorry for taking so long to reply. I’ve been away on vacation a while.

The current thought is to remove the integration objects entirely. In fact I’ve already removed them from the latest code in the svn repo on SourceForge, but that doesn’t reflect a final decision or anything; it’d be easy to put them back in.

The only useful thing the integration templates provide that I’m aware of is the ability to map from a dom* object to your custom object. So if you have an integration class for <node>s, you can easily take a domNode object after loading a document and lookup your corresponding intNode integration object. But it turns out that this is really easy to do yourself. It’s really just as easy as using the integration objects was, and allows greater flexibility and control over when things get converted.

The most common use case for the integration templates is to register your integration objects, load a document (after which your integration objects will be created), then traverse the dom* element hierarchy pulling in the data in your custom integration objects. I’ve written a sample program (part of the DOM’s automated test suite) that demonstrates how you can do the same thing without the integration objects.

I suggest you take a look at the code in the link. The point I’m trying to make is that the integration objects really weren’t all the helpful in writing an importer. What do you think?

Steve

I am currently using them to write a Blender Importer
Is this associated with the importer currently maintained by Illusoft, or is this a completely independent effort?

It is a completely independent effort, I plan on making it available under a MIT license or similar once it is mature enough, its publicly accessible from my SVN server at http://svn.aeongames.com/svn/BlenderCollada/trunk/.

If you’re wondering why I didn’t extended Illusoft’s plug in, its because I am not that proficient as a Python programmer and they manually parse the DAE XML using Python’s built in expat, I rather not deal with DOM code . :smiley:

[nevermind this]
So, I take it you’re taking out the templates but not integration, meaning I would have to implement the integration interface from scratch, may I ask what is it that makes the templates so bad so I can avoid it?
[/nevermind this]

Edit: I read only your last post before realizing there was already a previous more extensive one.

Thanks :slight_smile:

I agree, I started copying over files for pretty much any element I needed, and in the end I had to take out a lot of them and move their code to the topmost parent element (this after I found that I had a mesh object but no vertices in it at the time of processing a skin element).

Right now I do think I should be able to move everything out of the templates without too much of a problem, the thing I did find useful though was the way you keep references to the application objects in the void* _object property, so no need to look them up or keep a separate database.

Perhaps a similar property (just a setter and a getter for it) could be added to domElement just to keep the application object and element linked (if there isn’t already one there).

I agree, I started copying over files for pretty much any element I needed, and in the end I had to take out a lot of them and move their code to the topmost parent element (this after I found that I had a mesh object but no vertices in it at the time of processing a skin element).
That’s another weird thing about the integration templates. There’s one integration class provided for each Collada element, and it encourages programmers to use an integration class for each element they want to import. Then you end up with a mess of integration classes with complex interdependencies that make initialization difficult.
Right now I do think I should be able to move everything out of the templates without too much of a problem, the thing I did find useful though was the way you keep references to the application objects in the void* _object property, so no need to look them up or keep a separate database.

Perhaps a similar property (just a setter and a getter for it) could be added to domElement just to keep the application object and element linked (if there isn’t already one there).
That could be done. A developer would still do the hierarchy traversal and user-object conversion work himself, but he wouldn’t have to do the lookup tables. But then object lifetime becomes an issue. Would the application object just be a void*? Then users have to manage object lifetime themselves, and make sure they free up any allocated application objects at the appropriate time. Traditionally the integration objects have always been reference counted, so developers didn’t have to think about it. When a developer writes their own lookup table (like I did in the sample program) it’s obvious that he has to take care of cleaning up the objects when he’s done. Or instead of a void* we might have a pointer to a user-settable daeRefCountedObj in each daeElement. That way users wouldn’t need to consider object lifetime, but they’d have to derive all their objects from daeRefCountedObj.

Writing a lookup table really isn’t difficult though, so I’m not sure if it’s necessary to have a pointer to a user-object at all.

I just think it would be a “nice to have” for the times where you need URIs, for example skins reference geometry objects, it would be nice to do things like (not accurate, but you get the idea):

localgeomobject* geom = (localgeomobject*) skin->getGeometry()->getUserObject();

rather than go through a lookup table after you’ve dealt with the geometry and mesh retrieval, void* does seem a little crude, but it really lets the user make it anything they need, of course, I am not advocating code like:

domElement->setObject(new localgeom);

but rather something like say:

std::vector <localgeom> geoms;

localgeom geom;
/* fill geom */
geoms.push_back(geom);
domElement->setObject(&geoms.back())

so it really works out as a link between the two.

I am just throwing ideas here. :smiley:

In your example, if you change

std::vector <localgeom> geoms;

to

std::map<domGeometry*, localgeom> geomTable;

then you basically have the lookup table. Then the line

localgeom* geom = (localgeom*) skin->getGeometry()->getUserObject();

would instead be

localgeom& geom = geomTable[skin->getGeometry()];

That doesn’t seem any more or less difficult then using a void* pointer embedded with the daeElement.

But still, I can understand why some people would want to do it with custom data attached to each daeElement. I think I’ll put in some setUserData/getUserData methods which just deal with void*s.

You’re right, and perhaps I am being too pedantic :smiley: but what I am really thinking is that the std::map lookup is O(log n) which is very good, but keeping the reference in the object itself is O(1), which is better. There may not be much performance gain there, or there could be a lot, we need benchmarks for that.

Anyway, I’ll just wait to see that the final saying is for the next stable version and change my code accordingly then the time comes.

Thanks :D.

There may not be much performance gain there, or there could be a lot, we need benchmarks for that.
I remember at my last company I used the Collada DOM to write an importer. Originally I used the integration templates, then I realized they were stupid and decided to rip them out and replace them with my own lookup tables. I was very worried about the lookup tables having O(log N) lookups whereas the integration templates had O(1) lookups. I did some before/after benchmarking and, to my amazement, my loader was (very) slightly faster without the integration templates. There must’ve been some extra overhead associated with the integration objects… I never really figured that out. The point though is that I think there’s really very little to worry about regarding the O(log N) vs O(1) object lookups. Performance is almost certainly going to be dominated by things like reading from disk, parsing strings, and data copying.

But I’m fine with the setUserData/getUserData, so I’m going to put them in.

If you’re wondering why I didn’t extended Illusoft’s plug in, its because I am not that proficient as a Python programmer and they manually parse the DAE XML using Python’s built in expat, I rather not deal with DOM code.
I missed this post originally but just found it as I was reviewing the thread. I don’t mean to dissuade you, but I’d hope that you wouldn’t let a lack of proficiency in Python prevent you from working with their plugin. Python is a very easy language to learn, and is IMO much nicer to program in than C. In any case, I wish you luck with the project. It’d be great if you can help improve Blender’s Collada support.

Well, I didn’t wanted to go on a rant about the Illusoft plugin/Blender lack of a binary plug-in API, but you leave me no choise :wink:.

Actually as of today, the Illusoft plug-in is broken on Blender 2.45, I attribute this to a couple of factors, the first is that Blender lacks a binary plugin api, this is probably a design decision made by the Blender Foundation so plugins are compatible across platforms, however, their Python API is a moving target, it is not uncommon for plugins to break from one minor version to the next, and since the scripts are not required to work for compilation to succeed, stale scripts remain stale.

Illusoft’s plugin on SourceForge has seen no advance in 5 months, since there were talks about support for skin and animation I suppose the code in the SVN in there is likely not the latest, so if anyone decides to work in it, that person would have to start by fixing all the broken code due to changes in the Python Blender API, which may or may not have been fixed by Illusoft already.

Finally, I already wrote a DOM implementation using expat for an exporter I have (although in C), I rather not go through the pain of dealing with all the COLLADA intricacies if I don’t have to, enter the COLLADA_DOM.

Off topic: By the way, you know it would be nice to have special getters for accessors, for example accesor->getParamValue(param_name,position)

so if you’re navigating vertices, you could do something like(pseudocode):


float pos[3];
foreach(vertexindex)
{
pos[0] = accesor->getParamValue("X",vertexindex);
pos[1] = accesor->getParamValue("Y",vertexindex);
pos[2] = accesor->getParamValue("Z",vertexindex);
...
}

and not have to deal with strides and offsets manually.

Fair enough. I don’t disagree with your decision, I just wanted to make sure it wasn’t out of fear of learning Python.

Off topic: By the way, you know it would be nice to have special getters for accessors, for example accesor->getParamValue(param_name,position)
Yeah, this is a good point that touches on a larger topic. There’s a huge gap between how geometry is described in Collada and how geometry is described in a rendering engine that communicates directly with the graphics card. The DOM handles the text parsing of a Collada document for you but really does little to help get the geometry into a form suitable for a rendering engine. That type of thing has never been the focus of the DOM, which was originally conceived as an extremely low-level wrapper over a Collada document that shields the programmer from having to do manual parsing of the XML.

I think this lack of emphasis on bridging the gap between Collada and the graphics card was a big mistake on the part of the original DOM designers, but it’s unlikely to change much at this point.

Its definitely not that, my proprietary format exporter for Blender is a Python Script :smiley:

I think this lack of emphasis on bridging the gap between Collada and the graphics card was a big mistake on the part of the original DOM designers, but it’s unlikely to change much at this point.

Oh well, shouldn’t be so hard to implement, though it would be nice if it was already there, perhaps something to keep in mind for a later iteration of the DOM.

In revision 369 I added the setUserData/getUserData methods to daeElement. I also rewrote the integrationExample.cpp sample program to use those functions instead of maintaining separate lookup tables. I’m rather pleased with the result. The code is much simpler without the lookup tables, the only drawback being that I have to explicitly free the objects I attached to the daeElements when I’m done.

Thanks for suggesting the void* pointer with each daeElement. It worked out nicely.

No problem, I am glad to help.

Now I am almost done moving my code outside the integration templates, but I found materials to be quite a pain, is there a nicer way to get to them from the material attribute in polygons/triangles/polylist/etc element than querying the database and then iterate? I am thinking something similar to the SID uri resolver would be nice.

I’ll grab a copy of the code from SVN, I got to check if a tiny patch I wrote is still relevant.

Cheers!

is there a nicer way to get to them from the material attribute in polygons/triangles/polylist/etc element than querying the database and then iterate?
I’m not sure what exactly you mean when you talk about iterating or querying the database. To resolve a material symbol specified in a <triangles> (or similar) element, you need the <instance_material> from when the geometry gets used in the node hierarchy via <instance_geometry>. <instance_material> specifies exactly which <material> element a material symbol binds to, so you shouldn’t need any iteration or database queries.

I got to check if a tiny patch I wrote is still relevant.
I guess you saw that I closed the patch request in the SourceForge tracker. I don’t think the code is relevant anymore, but let me know if I missed something.

Steve

Well, to give you an idea my code is really ugly, and it looks like this:

	
for(size_t m = 0; m<Mesh->getPolygons_array().getCount();++m)
	{
		domPolygons* Polygons = (domPolygons*)(domElement*)Mesh->getPolygons_array()[m];
		// Find Material
		for(size_t i = 0;i<Polygons->getDocument()->getDatabase()->getElementCount(NULL,"instance_material",NULL);++i)
		{
			Polygons->getDocument()->getDatabase()->getElement(&InstanceMaterial,i,NULL,"instance_material",NULL);
			if((InstanceMaterial->getElementType()==COLLADA_TYPE::INSTANCE_MATERIAL)&&(strcmp(((domInstance_material*)InstanceMaterial)->getSymbol(),Polygons->getMaterial())==0))
			{
				daeURI materialURI(((domInstance_material*)InstanceMaterial)->getTarget());
				if (materialURI.getState() != daeURI::uri_success) 
				{
					materialURI.resolveElement();
				}
				if(materialURI.getElement()->getElementType()==COLLADA_TYPE::MATERIAL)
				{
					Material=(domMaterial*)(domElement*)materialURI.getElement();
					assert(Material->getIntObject()->getObject());
					PyObject* PyMeshMaterials = PyObject_GetAttrString(PyMesh,"materials");
					// Check here if the material is already in the list, if not add it, if so, grab it's index
					bool foundmaterial = false;
					int matcount = PyList_Size(PyMeshMaterials);
					for(int j=0;j<matcount;++j)
					{
						PyObject* CurMaterial = PyList_GetItem(PyMeshMaterials,j);
						PyObject* PyMaterial = (PyObject*)Material->getIntObject()->getObject();
						if(PyObject_Compare(PyMaterial,CurMaterial)==0)
						{
							MaterialIndex = j;
							foundmaterial=true;
							break;
						}
					}
					if((!foundmaterial)&&(matcount<16))
					{
						MaterialIndex = PyList_Size(PyMeshMaterials);
						PyObject* DaeMaterial = Py_BuildValue("[O]",(PyObject*)Material->getIntObject()->getObject());
						if(PyNumber_InPlaceAdd(PyMeshMaterials,DaeMaterial)==NULL)
						{
							PyErr_Print();
						}
						if(PyObject_SetAttrString(PyMesh,"materials",PyMeshMaterials)<0)
						{
							PyErr_Print();
						}						
						Py_DECREF(DaeMaterial);
					}
					Py_DECREF(PyMeshMaterials);
				}
				break;
			}
		}
// ... lots more code
}

The Py* calls and variables are Python API calls to Blender Functions.

Anyway, yes I use instance_material, my mistake :).

I see what you’re saying, I was getting to <skin> elements directly before, now I am using <scene> and iterating nodes in a visual_scene (I had an issue with joints because I was disregarding the <skeleton> element), so I should be able to get those there.

A thing I just noticed is that I have no instance_geometry, but the material info is in the instance_controller as a <bind_material>.

Yep, I got a note of the update, I don’t think the code is relevant anymore either, but I’ll check to see if there are any other issues.

In that code you’re not traversing Collada’s data structures correctly, particularly in the part where you’re finding which materials to apply to a mesh. Instead of starting with the meshes and looking through all <instance_materials> in the document for a matching material symbol, you’re supposed to traverse the <node> hierarchy looking for <instance_geometry>/<instance_controller> elements, which will have a <bind_material> that tells you how to map the material symbols to <material> elements. The way you have it coded now, your program could find an <instance_material> with a matching material symbol from a <instance_geometry>/<instance_controller> that has nothing to do with the mesh you’re processing.

One other thing to note is that you should be careful about when the DOM returns reference values. In many cases the DOM returns values by reference to avoid extra object copies. For example, in the code where you’re getting the material’s URI, instead of

daeURI materialURI(...

try

daeURI& materialURI(...

to avoid creation of a temporary daeURI. daeURI isn’t a particularly big structure so in this case it’s not going to make a huge speed difference in your code, but especially when working with arrays you should be careful.

Steve

Alright, now that you sent me in the right direction, my code looks like this:


	for(size_t i = 0; i<Mesh->getTriangles_array().getCount();++i)
	{
		domTriangles* Triangles = (domTriangles*)(domElement*)Mesh->getTriangles_array()[i];		
		if(NULL!=bind_material)
		{
			for(size_t i = 0;i<bind_material->getTechnique_common()->getInstance_material_array().getCount();++i)
			{
				if(strcmp(bind_material->getTechnique_common()->getInstance_material_array()[i]->getSymbol(),Triangles->getMaterial())==0)
				{
					PyObject* PyMaterial = ProcessMaterial((domMaterial*)(domElement*)bind_material->getTechnique_common()->getInstance_material_array()[i]->getTarget().getElement());
					PyObject* PyMeshMaterials = PyObject_GetAttrString(PyMesh,"materials");
					// Check here if the material is already in the list, if not add it, if so, grab it's index
					bool foundmaterial = false;
					int matcount = PyList_Size(PyMeshMaterials);
					for(int j=0;j<matcount;++j)
					{
						PyObject* CurMaterial = PyList_GetItem(PyMeshMaterials,j);
						if(PyObject_Compare(PyMaterial,CurMaterial)==0)
						{
							MaterialIndex = j;
							foundmaterial=true;
							break;
						}
					}
					if((!foundmaterial)&&(matcount<16))
					{
						MaterialIndex = PyList_Size(PyMeshMaterials);
						PyObject* PyMaterialAsList = Py_BuildValue("[O]",PyMaterial);
						if(PyMaterialAsList==NULL)
						{
							PyErr_Print();
						}
						if(PyNumber_InPlaceAdd(PyMeshMaterials,PyMaterialAsList)==NULL)
						{
							PyErr_Print();
						}
						if(PyObject_SetAttrString(PyMesh,"materials",PyMeshMaterials)<0)
						{
							PyErr_Print();
						}
						Py_DECREF(PyMaterialAsList);
					}
					Py_DECREF(PyMeshMaterials);
					break;
				}
			}
		}
...

Much better IMO :slight_smile:
I no longer make use of a daeURI on that particular piece of code, so no worries there, I’ll keep your advice in mind though, Thanks.