Page 3 of 5 FirstFirst 12345 LastLast
Results 21 to 30 of 47

Thread: Memory and Speed issues

  1. #21
    Guest

    memory and speed - and consistency

    Referencing this data externally seems to be the best solution; it keeps the file clean, consistent and easy to read, but allows massive amounts of data to be stored and recalled efficiently. If you decide to store binary data, then what is the point in wrapping it in xml tags? Imagine a file that contains five objects, each with 100,000 polys - it seems ridiculous to embed the binary data within the file, between xml tags, when the file is predominantly binary. On the other hand, if you have a ton of smaller objects in the file, and the vert lists are relatively short then the benefit of embedding this data in binary form is not really apparent.

    Referencing data externally is not a new idea by any stretch of the imagination - #include <etc>. One of the great benefits of an intermediate format is that it can be abstract and flexible. This also comes into play when you start to consider how large files will be dealt with in the future. For one, it is generally not possible to edit all of the polygons in a complex scene in a meaningful way. Generally, an artist will work subsection by subsection, exporting each subsection separately. These exports can be referenced by a top level file that is really quite small. When an edit is made to one of the source files subsequently, only that area must be exported, nothing in the top level file has changed. If the level grows, then two exports need to be done: 1) The new section and 2) the top level file - which, if it is only references should take more time to think about than to export - in which case it might make more sense for it to be edited by hand

    -Judah

  2. #22
    Senior Member
    Join Date
    Jul 2004
    Location
    Santa Clara
    Posts
    356
    Quote Originally Posted by greggman
    You shouldn't be editing middle format files because your edits will get lost every time your artists re-export.
    That is not true for COLLADA. And that is one of the major breakthrough, and design goal. One can create tools outside of the modeler, modify the COLLADA file, and be able to load the files back in the modeler.
    In other words, COLLADA is designed to be the source data. The Asset tag is there to enable asset management to work on a per element basis, and not on a per file basis, as it is the case today. So the importer can understand what has been changed outside of the modeler.

  3. #23
    Guest

    memory and speed - and consistency

    Yeah, and because of this, there will be a great need for intermediary tools to edit these exported files. For instance, once you have exported a number of files, and defined some assets, you might want to try a substitution of sorts. Instead of re-exporting, I think a tool in the midground would be very helpful. Something that would allow you to edit the flow of data, so to speak. This would not be a tool for editing vertex location or values, or performing other tasks best handled by art tools, but a tool that would allow you to edit data sources and destinations, such as external references, etc. I'm thinking of something like the Maya hypergraph editor - any takers?

    This is once again, a great example of why not to flatten the exported data - once it's flat, you can't go back.

    -Judah

  4. #24
    Senior Member
    Join Date
    Jul 2004
    Location
    Santa Clara
    Posts
    356
    Quote Originally Posted by Panajev2001a
    Quote Originally Posted by remi

    You can already store this information in the most compact binary form you want by using external references in COLLADA if you need to.
    Which would mean that he would have to write again a partial exporter.

    Am I right ?

    I fear that he would be not the only developers that would want to do that and then decide they do not need COLLADA.

    Making everyone happy is not possible, but an industry stabdard should aim to please at least a large chunk of developers to assure acceptance.
    This has nothing to do with COLLADA specification. As I said, it is already possible today to reference external binary data with COLLADA.

    You are right that this option would have to be added to the exporter, and if it is not supported by the exporter provided by the DCC tool you are using you would have to do it yourself. The alternative would be to contact them to ask for this feature to be supported.

    But in regard to this specific feature, exporting binary representations of floating point numbers would require that the same coding standard would need to be used on every platform it needs to be loaded back. It may be true for all the platforms you are using, but this is not true in the general case. This means that one would have to provide a conversion when loading the data.
    For example floating point data used in shaders, or in smaller devices like mobile phones 3D graphics, are not IEEE compatible.

  5. #25
    Member
    Join Date
    Aug 2004
    Location
    SCEJ - Tokyo, Japan
    Posts
    34
    The collada spec already explicitly states that collada is not a game engine format. It's a middle format targeted at tools that will then generate game ready formats. Therefore having any of the data in it be binary specific is not a problem since the tools themselves can handle the conversion and they can handle it much faster than they can handle parsing 2-5 times the data in ascii floats.

    The external binary format idea in the collada spec is not even an option and is illconceived as if it's not in a format that the collada spec specifies then a collada complient reader would not be able to read the file.

    If you want to add to the spec the format of those external files that would solve the problem since then any collada reader could read any file with external references but in that case, all i'm suggesting is that instead of the file being external it be allowed to be internal as CDATA.

  6. #26
    Guest
    Quote Originally Posted by greggman
    The collada spec already explicitly states that collada is not a game engine format. It's a middle format targeted at tools that will then generate game ready formats.
    Right, so we needn't worry about the data sizes of a specific platform at this point. For now, a float is a float.

    Quote Originally Posted by greggman
    The external binary format idea in the collada spec is not even an option and is illconceived as if it's not in a format that the collada spec specifies then a collada complient reader would not be able to read the file.

    If you want to add to the spec the format of those external files that would solve the problem since then any collada reader could read any file with external references but in that case, all i'm suggesting is that instead of the file being external it be allowed to be internal as CDATA.
    If I made it sound as though the binary specification should not be in the Collada spec, then that was a mistake. The whole idea here is to come up with something that is universal, I think that's why we're all participating. If it is possible to embed the data cleanly, then so be it, but it seems like there are some issues with this data being represented properly within the xml file. Personally, I'd rather not have large blocks of binary data in these files, but that is my preference.

  7. #27
    Guest
    Okay, I'm coming into this a little late, but here's my take on it:

    * I'd keep all scene data in a single file. Tif files can still be external because they're external in the source data as well. Trying to keep a bunch of binary files together with the .dae file seems ugly.
    * An optional faster, more compressed format for large data would be nice for all the reasons greggman iterates.
    * But CDATA seems like a bad idea for reasons also already covered -- http://lists.xml.org/archives/xml-dev/1 ... 00388.html
    * I think it's also legitimate to live within the limitations of an existing library like libxml2; I poked around a bit in the source and couldn't figure out if it had native base64 support.

    Why does libxml2 link with zlib? Does it support reading gzipped xml files directly? If that's the case, perhaps that would be a good middle ground?

    -Dave

  8. #28
    Junior Member
    Join Date
    Sep 2004
    Posts
    6
    I agree that storing raw binary data directly inside a xml file is not a solution. I also see the point in trying to work inside the xml constraint's and not inventing a new format.

    I agree with danny that a 3-4 fold performance increase can be the difference between nice to use and unbearable eg. waiting 5 or 20 seconds for a file to load.

    To solve the issue at hand you could just base 64 the binary data and put that data into the CDATA tag. The file size would increase by 33%, base 64 decoding/encoding can be quite fast and it is definately a lot faster than reading ascii float values.

    I see that this solution is ultimately not optimal and the only way to really solve it is by not using XML.

    I don't think embedding binary data in external files is a good solution either since you are just putting work from the programmer to the artist who is using collada and now has to keep 2 files around instead of one. And what happens if he wants to rename a file? It just doesn't cut it.

    For the time being Base64'ing vertex and index data seems like a good solution to me.

    I am sure i am not telling you (gabor) anything new here, so what do you think is wrong with base 64ing the binary data.

  9. #29
    Quote Originally Posted by joeante
    I am sure i am not telling you (gabor) anything new here, so what do you think is wrong with base 64ing the binary data.
    It's not inherently wrong, but there are two issues:
    - as Remi mentioned, storing floats in a native binary (or base64 encoded binary) form breaks interchangeability, because not all platforms have IEEE compliant floats
    - I'm still not convinced that it would be significantly faster than our current decimal-ASCII <-> float converters, but we should do some tests to determine that (see earlier message)

  10. #30
    Junior Member
    Join Date
    Sep 2004
    Posts
    6
    I thought collada is a interchange/intermediate format.

    Consoles/Mobile games might not have IEEE compliant floats but since collada is an intermediate/interchange format it will not be used to load meshes directly into the game which might be running on a platform without IEEE compliant floats.
    Rather the format will be used in a conversion process, which converts collada files to the native format of that game/platform. Thats my assumption anyway, is anyone planning to use collada as the final format in their game which will be shipped with the game to user?
    If someone does it anyway, it's not very hard to conver to non-IEEE floats anyway.

    It sounds like a very good idea to do some speed tests of ASCII vs binary. Please post the results.

Page 3 of 5 FirstFirst 12345 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •