OpenML Overview

Problems with cross-platform dynamic media authoring and playback and how they are addressed by the OpenML framework

Content creators, whether working on a video production, developing a game, or constructing an interactive disc, are currently forced to use many different APIs and tools from many different vendors and organizations to simply tie their multiple applications and hardware peripherals together.

Similarly, application developers in this situation currently have to make some very difficult decisions about the APIs to use in order to access the various pieces of hardware in the system. Much of the work is liable to be hardware- and operating system-specific and require hand-tuning to achieve acceptable performance.

The goal of OpenML is to rectify this situation. OpenML offers a common specification for interoperability across platforms, operating systems, hardware devices and software applications. The existence of such a standard will allow content developers to mix and match applications and hardware and develop their multimedia content without regard to the hardware that might or might not be available during playback. It will allow application developers to spend more time and energy on application development and much less time on mundane details such as device control, synchronization, and buffering issues. It will also free them from the unenviable task of dealing with different APIs for video, graphics, and audio for each operating environment.

OpenML provides specific solutions using proven technologies to a range of integration and control problems.

Problem		Solution
#1: Configure and control the Input/Output and processing of video and digital audio		SGI's dmSDK Digital Media Handling
#2: Configure and control display devices		SGI's Xdc extension to X11 Display Control
#3: Enable OpenGL hardware to add more value to the media authoring workflow		OpenGL Extensions Integrating graphics and video
#4: Synchronization of diverse media streams (graphics, audio, video)		UST/MSC Synchronization

Application Programmers View of the OpenML Programming Environment

The diagram below shows the major components of the OpenML environment from a programmer's perspective. Items shown with a blue background already exist, and the OpenML programming environment does not change how they are accessed in any way. Items with a red background are new in OpenML.

Window System
Rather than attempt to achieve cross-platform standardization in this area, the OpenML environment will simply coexist peacefully with the native windowing environment.

OpenML and OpenGL Integration

OpenGL and OpenML tight and synergistic integration

• OpenGL 1.2.1 with GLX 1.3 (Linux/UNIX) or WGL (Windows)
  • OpenML extends OpenGL with video primitives and DCC rendering features
• Defines a set of OpenGL extensions that OpenML compliant hardware must implement
  • Pbuffers are the fundamental method of integrating graphics and video
• Enables buffering and off-screen processing of video data
  • OpenGL imaging extensions are used to process video streams
• Filtering, color correction, blending, image enhancement, color space conversion
  • Significant opportunity for OpenGL hardware vendors to add more value
  • Use OpenGL graphics hardware for 3D and video acceleration

OpenGL Extensions Bring Together the Worlds of 3D and Video
Extensions to OpenGL add capabilities for synchronization with other parts of the system, for improved rendering performance and quality, and for improved treatment of video formats.
Extensions include:

• Synchronization control
  • Primitives to control UST/MSC synchronization from OpenGL
  • Functions added: WaitForMSC, SwapBuffersMSC, GetSyncValues
• Interlace support
  • Based on two previous extensions: SGIX_interlace, INGR_interlace_read
  • Enables reading and writing of the frame buffer while skipping every other line
  • Enables processing (e.g., convolution) on interlaced video streams
• Texture color mask management to enable advanced compositing
  • Enables independent loading of RGB images and Alpha masks in texture memory
  • Recently promoted from an SGI extension to an ARB extension
  • Works for textures like the OpenGL color mask works for the color buffer
• Chroma resampling to support pixel formats such as YCrCb
  • Upsampling and downsampling when converting between YCrCb and RGB
  • Sampling options are: Replicate, Zero fill, Average

OpenML's Mldc API

MLdc API for abstracted display control
The OpenML Display Control API (MLdc) provides an interface to initialize, set and change parameters to control video output devices MLdc also facilitates communication to and from monitors capable of sending and receiving commands.

• Allows an application to control what is displayed on monitors
  • Sophisticated level of control beyond most windowing systems
• MLdc sets up channels to drive each display screen
  • Channels that are driven from sections of physical frame buffers
  • Number and type of supported channels depends on the display device
• Properties that can be controlled include:
  • The number of channels and where their display data is derived
  • Interrogating monitor information
  • Precise video positioning
  • Video format query and settings
  • Vertical retrace rate
  • Interlaced vs. progressive
  • Gamma ramp control
  • Video sync sources
  • Genlock state notification

OpenML's Open Source "ML" API

ML Digital Media I/O
The Open Source OpenML Media Library (ML) provides an application programming interface to the digital media I/O, digital media data conversion, and digital media synchronization facilities in an OpenML programming environment. ML is a low-level API in the same sense that OpenGL is considered a low-level API: it exposes the capabilities of the underlying hardware in a way that imposes little policy.

The primary functions of ML are to:

• Support asynchronous communication between an application and media devices such as video input/output, audio input/output, and graphics.
• Provide synchronization primitives that give applications the ability to correlate multiple digital media streams and coordinate their presentation to an end user
• Provide processing capabilities (transcoders) for digital media streams
• Provide device control and device capability queries
• Provide buffering mechanisms that support the smooth delivery of digital media and obtain the best possible performance on a given system

Synchronizing Audio, Video and Graphics with UST/MSC/SBC

The UST/MSC/SBC Solution for Synchronizing Audio, Video and Graphics
Underneath all of this are system facilities that provide precise timing and synchronization information.

• UST - Unadjusted System Time - time reference for software and devices
  • 64-bit monotonic counter with nanosecond resolution, microsecond system-wide accuracy
• MSC - Media Stream Counter - incremented for each channel sample
  • i.e. Video - per frame, Graphics - per retrace, audio - per audio sample, e.g. 44.1KHz
• SBC - Swap Buffer Count - incremented on graphics buffer swap
  • Not necessarily in synch with retrace

Using UST/MSC/SBC

• Robust Synchronization - even on non real-time OS
• Every media sample is time-stamped
  • Timestamp can be accessed through OpenGL and ML APIs for temporal analysis and control
• Application can synchronize output events to UST
  • Enabling precise timing and synchronization in non-real-time operating systems
• Applications can detect dropped samples
  • And take corrective action

Licensing and OpenML

There are three level of participation in OpenML development and use:

1. Application developers - any developer or company can freely use the OpenML specification & SDK to integrate OpenML functionality into their hardware or software products. OpenML is a cross-platform, free, open standard so developers may implement and ship products using OpenML completely free of charge, royalty or licensing. OpenML developers may also use the OpenML logo on their web site.
2. Contributors - open membership, nominal annual fee, working group participation & voting
3. Promoters - founders of the group, driving the Khronos organization and the OpenML standards and having final ratification voting rights