[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Public WebGL] A different approach to interleaved typed arrays

The TC39 folks are making a valiant effort at defining structured data in JavaScript. But I don't think their efforts will be able to bear fruit in a timeframe that would be convenient for us. But still I feel their pain. Their main complaint of our current Typed Arrays proposal is that it exposes endian issues to the author. Not only that, but it makes it easy to accidentally write endian dependent code. Since the majority of machines today are little endian, this is a ticking time bomb waiting for the day that someone tries it on a big endian machine.

I'd like to propose changes to the current Typed Array proposal which would solve the problem of exposing endianness, but would still be efficient:


We've discussed in the past the idea of making the views non-overlapping. Mapping a range of an ArrayBuffer to one view would make it inaccessible to any other view. Furthermore, when mapping a range of an ArrayBuffer, that range is clear, preventing the author from, for instance:

1) Mapping a range to bytes
2) Filling that range with data, which happens to be floating point values in little endian order
3) removing that mapping
4) Mapping that same range to floats to access the previously loaded data

This would all be well and good, and the implementation isn't really that complex. It would simply require remembering which views are mapped to which buffer and doing a validation whenever a new mapping is made.

But it has a problem because it would not be practical to do interleaved mapping. You'd essentially need a new view for each component of each element. For instance, if you had 1000 vertices, made up of 3 floats followed by 4 bytes, you'd need 1000 Float32Arrays and 1000 Int8Arrays.

But we could add an explicit interleaved mapping capability to the Typed Arrays. It would require something like this:

	var buf  = new ArrayBuffer(1000 * 16 /* 3 floats + 4 bytes */);
	var floats = new Float32Array(buf, 0, 1000, 3, 4);
	var colors = new UInt8Array(buf, 12, 1000, 4, 12);

The extra 2 parameters added to the Typed Array constructors are:

	elementsPerGroup - number of elements in each group of elements
	bytesToNextGroup - number of bytes to skip to get from the end of a group of elements to the start of the next

There are other ways to phrase those parameters. For instance you might want the second one to be 'stride', which is the number of bytes from the start of one group to the next. But the result is the same. In the above example the views don't overlap because the extra parameters define which parts of the buffer belong to which view. Checking for validity is essentially the same as for the non-stride case, although perhaps a bit more complex. But I don't believe creating views into a buffer is likely to happen frequently, so it can be a relatively expensive operation without hurting performance.

The next question is, how does access change? In the current scheme, writing to an interleaved buffer involves computing offsets to each group in JavaScript. With this scheme, each view would appear to be a contiguous array of the given type. Loading a view is simply a matter of:

	var values = [ ... array of 3000 floating point values ... ];

and have the values "scattered" to the appropriate groups of elements. Doing this in native code would be significantly faster that doing a JavaScript loop, computing offsets and loading single values at a time.

How would you change mappings? How would you disassociate a Typed Array with a range in an ArrayBuffer and then associate a different one? One simple answer is that you can't. Once an association is made, no other association with those bytes of the ArrayBuffer can ever be made. This seems reasonable given the automatic garbage collection that occurs in JavaScript. ArrayBuffers are not precious resources that need to be shared. It would be just as easy to create a new ArrayBuffer when new mappings are needed. This is especially true if the data in the ArrayBuffer doesn't survive across mappings. It also makes it unnecessary to clear the data in a newly mapped range. If you assume the ArrayBuffer was cleared on creation, the data in the newly mapped range is guaranteed to be valid.

We also discussed the notion of separating ArrayBuffers that are used for incoming data from those that are used to prepare data for uploading to the GPU. I think we should repurpose the DataView concept for this functionality. We can call it a DataBuffer, or NetworkBuffer, or whatever. It would have the API's from the DataView, but it would actually be backed by an internal buffer. This object would be returned from XMLHttpRequest, a BLOb object or any other object that accesses data in some external (not machine specific) endianness. When getting data from this buffer, you specify the endianness you know the data to be in and it is returned to you in the native machine endianness. When writing data to the buffer (for eventual output to an external destination) you specify endianness and the data is converted from internal endianness to the one specified.

We should probably add methods to this object which can deal with arrays of data and we might even want to make it aware of interleaved Typed Arrays. For instance, if you have a DataBuffer 'inData' with floats and bytes interleaved the same as in the above example, but in little endian order, you might say:

	inData.setArray(floats, 0, 1000, true);
	inData.setArray(colors, 12, 1000, true);

These calls would handle the byte swapping for matching endianness (if needed) and the interleaving the values into groups according to the layout of Typed Arrays.

This solves the problem of undesired endian errors and provides a fast API for interleaving data.



You are currently subscribed to public_webgl@khronos.org.
To unsubscribe, send an email to majordomo@khronos.org with
the following command in the body of your email: