The OpenVX Specification  r28647
Design Overview

# Software Landscape

OpenVX is intended to be used either directly by applications or as the acceleration layer for higher-level vision frameworks, engines or platform APIs.

OpenVX Usage Overview

# Design Objectives

OpenVX is designed as a framework of standardized computer vision functions able to run on a wide variety of platforms and potentially to be accelerated by a vendor's implementation on that platform. OpenVX can improve the performance and efficiency of vision applications by providing an abstraction for commonly-used vision functions and an abstraction for aggregations of functions (a “graph”), thereby providing the implementer the opportunity to minimize the run-time overhead.

The functions in OpenVX 1.0 are intended to cover common functionality required by many vision applications.

## Hardware Optimizations

This specification makes no statements as to which acceleration methodology or techniques may be used in its implementation. Vendors may choose any number of implementation methods such as parallelism and/or specialized hardware offload techniques.

This specification also makes no statement or requirements on a “level of performance” as this may vary significantly across platforms and use cases.

## Hardware Limitations

The OpenVX 1.0 focuses on vision functions that can be significantly accelerated by diverse hardware. Future versions of this specification may adopt additional vision functions into the core standard when hardware acceleration for those functions becomes practical.

# Assumptions

## Portability

OpenVX 1.0 has been designed to maximize functional and performance portability wherever possible, while recognizing that the API is intended to be used on a wide diversity of devices with specific constraints and properties. Tradeoffs are made for portability where possible: for example, portable Graphs constructed using this API should work on any OpenVX implementation and return similar results within the precision bounds defined by the OpenVX conformance tests.

## Opaqueness

To avoid forcing hardware-specific requirements onto any particular implementation, the API is designed to be opaque.

OpenVX is intended to address a very broad range of devices and platforms - from deeply embedded systems to desktop machines, and even distributed computing architectures.

The range of implementations is quite discreet, and as such, the API shall only address all these spaces through opaqueness.

All data, except client-facing structures, are opaque and hidden behind a reference that may be as thin or thick as an implementation needs. Each implementation provides the standardized interfaces for accessing data that takes care of specialized hardware, platform, or allocation requirements. Memory that is imported or shared from other APIs is not subsumed by OpenVX and is still maintained and accessible by the originator.

OpenVX does not dictate any requirements on memory allocation methods or the layout of opaque memory objects and it does not dictate byte packing or alignment for structures on architectures.

# Object-Oriented Behaviors

OpenVX objects are both strongly typed at compile-time for safety critical applications and are strongly typed at run-time for dynamic applications. Each object has its typedef'd type and its associated enumerated value in the vx_type_e list. Any object may be down-cast to a vx_reference safely to be used in functions that require this, specifically vxQueryReference, which can be used to get the vx_type_e value using an vx_enum.

# OpenVX Framework Objects

This specification defines the following OpenVX framework objects.

• Object: Context - The OpenVX context is the object domain for all OpenVX objects. All data objects live in the context as well as all framework objects. The OpenVX context keeps reference counts on all objects and must do garbage collection during its deconstruction to free lost references. While multiple clients may connect to the OpenVX context, all data are private in that the references that refer to data objects are given only to the creating party. The results of calling an OpenVX function on data objects created in different contexts are undefined.
• Object: Kernel - A Kernel in OpenVX is the abstract representation of a computer vision function, such as a “Sobel Gradient” or “Lucas Kanade Feature Tracking”. A vision function may implement many similar or identical features from other functions, but it is still considered a single, unique kernel as long as it is named by the same string and enumeration and conforms to the results specified by OpenVX. Kernels are similar to function signatures in this regard.
• Object: Parameter - An abstract input, output, or bidirectional data object passed to a computer vision function. This object contains the signature of that parameter's usage from the kernel description. This information includes:
• Signature Index - The numbered index of the parameter in the signature.
• Object Type - e.g. VX_TYPE_IMAGE, or VX_TYPE_ARRAY, or some other object type from vx_type_e.
• Usage Model - e.g. VX_INPUT, VX_OUTPUT, or VX_BIDIRECTIONAL.
• Presence State - e.g. VX_PARAMETER_STATE_REQUIRED, or VX_PARAMETER_STATE_OPTIONAL.
• Object: Node - A node is an instance of a kernel that will be paired with a specific set of references (the parameters). Nodes are created from and associated with a single graph only. When a vx_parameter is extracted from a Node, an additional attribute can be accessed:
• Reference - The vx_reference assigned to this parameter index from the Node creation function (e.g., vxSobel3x3Node).
• Object: Graph - A set of nodes connected in a directed (only goes one-way) acyclic (does not loop back) fashion. A Graph may have sets of Nodes that are unconnected to other sets of Nodes within the same Graph. See Graph Formalisms.

# OpenVX Data Objects

Data objects are object that are processed by graphs in nodes.

• Object: Array An opaque array object that could be an array of primitive data types or an array of structures.
• Object: Convolution An opaque object that contains $$MxN$$ matrix of vx_int16 values. Also contains a scaling factor for normalization. Used specifically with vxuConvolve and vxConvolveNode.
• Object: Delay An opaque object that contains a manually controlled, temporally-delayed list of objects.
• Object: Distribution An opaque object that contains a frequency distribution (e.g., a histogram).
• Object: Image An opaque image object that may be some format in vx_df_image_e.
• Object: LUT An opaque lookup table object used with vxTableLookupNode and vxuTableLookup.
• Object: Matrix An opaque object that contains $$MxN$$ matrix of some scalar values.
• Object: Pyramid An opaque object that contains multiple levels of scaled vx_image objects.
• Object: Remap An opaque object that contains the map of source points to destination points used to transform images.
• Object: Scalar An opaque object that contains a single primitive data type.
• Object: Threshold An opaque object that contains the thresholding configuration.

# Error Objects

Error objects are specialized objects that may be returned from other object creator functions when serious platform issue occur (i.e., out of memory or out of handles). These can be checked at the time of creation of these objects, but checking also may be put-off until usage in other APIs or verification time, in which case, the implementation must return appropriate errors to indicate that an invalid object type was used.

vx_<object> obj = vxCreate<Object><Method>(context, ...);
vx_status status = vxGetStatus((vx_refernce)obj);
if (obj && status == VX_SUCCESS) {
// object is good
}

# Graphs Concepts

The graph is the central computation concept of OpenVX. The purpose of using graphs to express the Computer Vision problem is to allow for the possibility of any implementation to maximize its optimization potential because all the operations of the graph and its dependencies are known ahead of time, before the graph is processed.

Graphs are composed of one or more nodes that are added to the graph through node creation functions. Graphs in OpenVX must be created ahead of processing time and verified by the implementation, after which they can be processed as many times as needed.

Graph Nodes are linked together via data dependencies with no explicitly-stated ordering. The same reference may be linked to other nodes. Linking has a limitation, however, in that only one node in a graph may output to any specific data object reference. That is, only a single writer of an object may exist in a given graph. This prevents indeterminate ordering from data dependencies. All writers in a graph shall produce output data before any reader of that data accesses it.

## Virtual Data Objects

Graphs in OpenVX depend on data objects to link together nodes. When clients of OpenVX know that they do not need access to these intermediate data objects, they may be created as virtual. Virtual data objects can be used in the same manner as non-virtual data objects to link nodes of a graph together; however, virtual data objects are different in the following respects.

• Inaccessible - No calls to an Access/Commit API shall succeed given a reference to an object created through a virtual create function from a Graph external perspective. Calls to Access/Commit from within client-defined functions may succeed as they are Graph internal.
• Dimensionless or Formatless - Virtual data objects may be declared to have no dimensions or format and they may return zeros or generic values for formats when queried.
• Scoped - Virtual data objects are scoped within the Graph in which they are created; they cannot be shared outside their scope.
• Intermediates - Virtual data objects should be used only for intermediate operations within Graphs, because they are fundamentally inaccessible to clients of the API.
• Optimizations - Virtual data objects do not have to be created during Graph validation and execution and therefore may be of zero size.

These restrictions enable vendors the ability to optimize some aspects of the data object or its usage. Some vendors may not allocate such objects, some may create intermediate sub-objects of the object, and some may allocate the object on remote, inaccessible memories. OpenVX does not proscribe which optimization the vendor does, merely that it may happen.

## Node Parameters

Parameters to node creation functions are defined as either atomic types, such as vx_int32, vx_enum, or as objects, such as vx_scalar, vx_image. The atomic variables of the Node creation functions shall be converted by the framework into vx_scalar references for use by the Nodes. A node parameter of type vx_scalar can be changed during the graph execution; whereas, a node parameter of an atomic type (vx_int32 etc.) require at least a graph revalidation if changed. All node parameter objects may be modified by retrieving the reference to the vx_parameter via vxGetParameterByIndex, and then passing that to vxQueryParameter to retrieve the reference to the object.

If the type of the parameter is unknown, it may be retrieved with the same function.

vx_enum type;
vxQueryParameter(param, VX_PARAMETER_ATTRIBUTE_TYPE, &type, sizeof(type));
/* cast the ref to the correct vx_<type>. Atomics are now vx_scalar */

## Graph Parameters

Parameters may exist on Graphs, as well. These parameters are defined by the author of the Graph and each Graph parameter is defined as a specific parameter from a Node within the Graph using vxAddParameterToGraph. Graph parameters communicate to the implementation that there are specific Node parameters that may be modified by the client between Graph executions. Additionally, they are parameters that the client may set without the reference to the Node but with the reference to the Graph using vxSetGraphParameterByIndex. This allows for the Graph authors to construct Graph Factories. How these factories work falls outside the scope of this document.

Framework: Graph Parameters

## Execution Model

Graphs must execute in both:

• Synchronous blocking mode (in that vxProcessGraph will block until the graph has completed), and in
• Asychronous single-issue-per-reference mode (via vxScheduleGraph and vxWaitGraph).

### Asychronous Mode

In asynchronous mode, Graphs must be single-issue-per-reference. This means that given a constructed graph reference $$G$$, it may be scheduled multiple times but only executes sequentially with respect to itself. Multiple graphs references given to the asynchronous graph interface do not have a defined behavior and may execute in parallel or in series based on the behavior or the vendor's implementation.

## Graph Formalisms

To use graphs several rules must be put in place to allow deterministic execution of Graphs. The behavior of a processGraph( $$G$$) call is determined by the structure of the Processing Graph $$G$$. The Processing Graph is a bipartite graph consisting of a set of Nodes $$N_1 \ldots N_n$$ and a set of data objects $$d_1 \ldots d_i$$. Each edge ( $$N_x$$, $$D_y$$) in the graph represents a data object $$D_y$$ that is written by Node $$N_x$$ and each edge ( $$D_x$$, $$N_y$$) represents a data object $$D_x$$ that is read by Node $$N_y$$. Each edge $$e$$ has a name Name( $$e$$), which gives the parameter name of the node that references the corresponding data object. Each Node Parameter also as a type Type(node, name) in {INPUT, OUTPUT, INOUT}. Some data objects are Virtual, and some data objects are Delay. Delay data objects are just collections of data objects with indexing (like an image list) and known linking points in a graph. A node may be classified as a head node, which has no backward dependency. Alternatively, a node may be a dependent node, which has a backward dependency to the head node. In addition, the Processing Graph has several restrictions:

1. Output typing - Every output edge ( $$N_x$$, $$D_y$$) requires Type( $$N_x$$, Name( $$N_x$$, $$D_y$$)) in {OUTPUT, INOUT}
2. Input typing - Every input edge ( $$N_x$$, $$D_y$$) requires Type( $$N_y$$, Name( $$D_x$$, $$N_y$$)) in {INPUT} or {INOUT}
3. Single Writer - Every data object is the target of at most one output edge.
4. Broken Cycles - Every cycle in $$G$$ must contain at least input edge ( $$D_x$$, $$N_y$$) where $$D_x$$ is Delay.
5. Virtual images must have a source - If $$D_y$$ is Virtual, then there is at least one output edge that writes $$D_y$$ ( $$N_x$$, $$D_y$$)
6. Bidirectional data objects shall not be virtual - If Type( $$N_x$$, Name( $$N_x$$, $$D_y$$)) is INOUT implies $$D_y$$ is non-Virtual.
7. Delay data objects shall not be virtual - If $$D_x$$ is Delay then it shall not be Virtual.

The execution of each node in a graph consists of an atomic operation (sometimes referred to as firing) that consumes data representing each input data object, processes it, and produces data representing each output data object. A node may execute when all of its input edges are marked present. Before the graph executes, the following initial marking is used:

• All input edges ( $$D_x$$, $$N_y$$) from non-Virtual objects Dx are marked (parameters must be set).
• All input edges ( $$D_x$$, $$N_y$$) with an output edge ( $$N_z$$, $$D_x$$) are unmarked.
• All input edges ( $$D_x$$, $$N_y$$) where $$D_x$$ is a Delay data object are marked.

Processing a node results in unmarking all the corresponding input edges and marking all its output edges; marking an output edge ( $$N_x$$, $$D_y$$) where $$D_y$$ is not a Delay results in marking all of the input edges ( $$D_y$$, $$N_z$$). Following these rules, it is possible to statically schedule the nodes in a graph as follows: Construct a precedence graph $$P$$, including all the nodes $$N_1 \ldots N_x$$, and an edge ( $$N_x$$, $$N_z$$) for every pair of edges ( $$N_x$$, $$D_y$$) and ( $$D_y$$, $$N_z$$) where $$D_y$$ is not a Delay. Then unconditionally fire each node according to any topological sort of $$P$$.

The following assertions should be verified:

• $$P$$ is a Directed Acyclic Graph (DAG), implied by 4 and the way it is constructed.
• Every data object has a value when it is executed, implied by 5, 6, 7, and the marking.
• Execution is deterministic if the nodes are deterministic, implied by 3, 4, and the marking.
• Every node completes its execution exactly once.

The execution model described here just acts as a formalism. For example, independent processing is allowed across multiple depended and depending nodes and edges, provided that the result is invariant with the execution model described here.

## Node Execution Independence

In the following example a client computes the gradient magnitude and gradient phase from a blurred input image. The vxMagnitudeNode and vxPhaseNode are independently computed, in that each does not depend on the output of the other. OpenVX does not mandate that they are run simultaneously or in parallel, but it could be implemented this way by the OpenVX vendor.

A simple graph with some independent nodes.

The code to construct such a graph can be seen below.

vx_image images[] = {
vxCreateImage(context, 640, 480, VX_DF_IMAGE_UYVY),
vxCreateImage(context, 640, 480, VX_DF_IMAGE_U8),
vxCreateImage(context, 640, 480, VX_DF_IMAGE_U8),
};
vx_graph graph = vxCreateGraph(context);
vx_image virts[] = {
};
vxChannelExtractNode(graph, images[0], VX_CHANNEL_Y, virts[0]),
vxGaussian3x3Node(graph, virts[0], virts[1]),
vxSobel3x3Node(graph, virts[1], virts[2], virts[3]),
vxMagnitudeNode(graph, virts[2], virts[3], images[1]),
vxPhaseNode(graph, virts[2], virts[3], images[2]),
status = vxVerifyGraph(graph);
if (status == VX_SUCCESS)
{
status = vxProcessGraph(graph);
}
vxReleaseContext(&context); /* this will release everything */

## Verification

Graphs within OpenVX must go through a rigorous validation process before execution to satisfy the design concept of eliminating run-time overhead (parameter checking) that guarantees safe execution of the graph. OpenVX must check for (but is not limited to) these conditions:

• Parameters To Nodes:
• Each required parameter is given to the node (vx_parameter_state_e). Optional parameters may not be present and therefore are not checked when absent. If present, they are checked.
• Each parameter given to a node must be of the right direction (a value from vx_direction_e).
• Each parameter given to a node must be of the right object type (from the object range of vx_type_e).
• Each parameter attribute or value that has algorithmic significance must be verified. In the case of a scalar value, it may need to be range checked (e.g.,  $$0.5 <= k <= 1.0$$). The implementation is not required to do run-time range checking of scalar values. If the value of the scalar changes at run time to go outside the range, the results are undefined. The rationale is that the potential performance hit for run-time range checking is too large to be enforced. It will still be checked at graph verification time as a time-zero sanity check. If the scalar is an output parameter of another node, it must be initialized to a legal value. In the case of vxScaleImageNode, the relation of the input image dimensions to the output image dimensions determines the scaling factor. These values or attributes of data objects must be checked for compatibility on each platform.
• Graph Connectivity - the vx_graph must be a Directed Acyclic Graph (DAG). No cycles or feedback is allowed. The vx_delay object has been designed to explicitly address feedback between Graph executions.
• Resolution of Virtual Data Objects - Any changes to Virtual data objects from unspecified to specific format or dimensions, as well as the related creation of objects of specific type that are observable at processing time, takes place at Verification time.

# Callbacks

Callbacks are a method to control graph flow and to make decisions based on completed work. The vxAssignNodeCallback call takes as a parameter a callback function. This function will be called after the execution of the particular node, but prior to the completion of the graph. If nodes are arranged into independent sets, the order of the callbacks is unspecified. Nodes that are arranged in a serial fashion due to data dependencies perform callbacks in order. The callback function may use the node reference first to extract parameters from the node, and then extract the data references. Data outputs of Nodes with callbacks shall be available (via Access/Commit methods) when the callback is called.

# User Kernels

OpenVX supports the concept of client-defined functions that shall be executed as Nodes from inside the Graph or are Graph internal. The purpose of this paradigm is to:

• Further exploit independent operation of nodes within the OpenVX platform.
• Allow componentized functions to be reused elsewhere in OpenVX.
• Formalize strict verification requirements (i.e., Contract Programming).

A graph with User Kernel nodes which are independent of the “base” nodes.

In this example, to execute client-supplied functions, the graph does not have to be halted and then resumed. These nodes shall be executed in an independent fashion with respect to independent base nodes within OpenVX. This allows implementations to further minimize execution time if hardware to exploit this property exists.

## Parameter Validation

User Kernels must aid in the Graph Verification effort by providing explicit validation functions for each vision function they implement. Each parameter passed to the instanced Node of a User Kernel are validated using the client-supplied validation functions. The client must check these attributes and/or values of each parameter:

• If an attribute or value of the parameter has algorithmic significance, it must be checked. For example, the size of array, or the value of a scalar to be within a range, or a dimensionality constraint of an image such as width divisibility. (Some implementations may have restrictions, such as an image width be evenly divisible by some fixed number).
• If the output parameters depend on attributes or values from input parameters, those relationships must be checked (within the output validator).

Input validators execute before output validators. This allows any or all inputs to be used as dependents of output parameter validation.

### The Meta Format Object

The Meta Format Object is an opaque object used to collect requirements about the output parameter, which then the OpenVX implementation will check. The Client must manually set relevant object attributes to be checked against output parameters, such as dimensionality, format, scaling, etc.

### Delta Rectangles

There is a special case with vx_image output parameters where the User Kernel output validation function can specify a positional and/or size-related change of the valid region of the output image relative to the input image during verification time. This is intended to give the optimizer more information about memory usage, and could lead to better outcomes or different strategies. Delta rectangles (specified using the vx_delta_rectangle_t parameter) are used to update a valid region for the user kernels with a call to vxSetMetaFormatAttribute from the output validator.

For example, for a 5x5 box filter where 2 border pixels of the output are lost (invalid), and with no center shift, use:

vx_delta_rectangle_t delta = {2, 2, -2, -2};

For the same 5x5 box filter, except with a center-shift into the upper-left corner:

vx_delta_rectangle_t delta = {0, 0, -4, -4};

If this attribute has not been set prior to graph verification, the graph manager must determine the new valid region based on vxCommitImagePatch calls during the execution time.

// User Kernel Function is a 5x5 filter with no explicit border handling code.
// So, 2 pixels on each edge of the image will be lost.
vx_delta_rectangle_t delta = {2,2,-2,-2};

## User Kernels Naming Conventions

User Kernels must be exported with a unique name (see Naming Conventions for information on OpenVX conventions) and a unique enumeration. Clients of OpenVX may use either the name or enumeration to retrieve a kernel, so collisions due to non-unique names will cause problems. The kernel enumerations may be extended by following this example:

#define VX_KERNEL_NAME_KHR_XYZ "org.khronos.example.xyz"
#define VX_LIBRARY_XYZ (0x3) // assigned from Khronos, vendors control their own
enum vx_kernel_xyz_ext_e {
VX_KERNEL_KHR_XYZ = VX_KERNEL_BASE(VX_ID_KHRONOS, VX_LIBRARY_XYZ) + 0x0,
// up to 0xFFF kernel enums can be created.
};

Each vendor of a vision function or an implementation must apply to Khronos to get a unique identifier (up to a limit of $$2^{12}-1$$ vendors). Until they obtain a unique ID vendors must use VX_ID_DEFAULT.

To construct a kernel enumeration, a vendor must have both their ID and a library ID. The library ID's are completely vendor defined (however when using the VX_ID_DEFAULT ID, many libraries may collide in namespace).

Once both are defined, a kernel enumeration may be constructed using the VX_KERNEL_BASE macro and an offset. (The offset is optional, but very helpful for long enumerations.)

# Immediate Mode Functions

OpenVX also contains an interface defined within <VX/vxu.h> that allows for immediate execution of vision functions. These interfaces are prefixed with vxu to distinguish them from the Node interfaces, which are of the form vx<Name>Node. Each of these interfaces replicates a Node interface with some exceptions. Immediate mode functions are defined to behave as Single Node Graphs, which have no leaking side-effects (e.g., no Log entries) within the Graph Framework after the function returns. The following tables refer to both the Immediate Mode and Graph Mode vision functions. The Module documentation for each vision function draws a distinction on each API by noting that it is either an immediate mode function with the tag [Immediate] or it is a Graph mode function by the tag [Graph].

# Base Vision Functions

OpenVX comes with a standard or base set of vision functions. The following table lists the supported set of vision functions, their input types (first table) and output types (second table), and the version of OpenVX in which they are supported.

## Inputs

Vision Function U8 U16 S16 S32 U32 F32 4CC
AbsDiff 1.0
Accumulate 1.0
AccumulateSquared 1.0
AccumulateWeighted 1.0
And 1.0
Box3x3 1.0
CannyEdgeDetector 1.0
ChannelCombine 1.0
ChannelExtract 1.0
ColorConvert 1.0
ConvertDepth 1.0 1.0
Convolve 1.0
Dilate3x3 1.0
EqualizeHistogram 1.0
Erode3x3 1.0
FastCorners 1.0
Gaussian3x3 1.0
HarrisCorners 1.0
HalfScaleGaussian 1.0
Histogram 1.0
IntegralImage 1.0
TableLookup 1.0
Magnitude 1.0
MeanStdDev 1.0
Median3x3 1.0
MinMaxLoc 1.0 1.0
Multiply 1.0 1.0
Not 1.0
OpticalFlowLK 1.0
Or 1.0
Phase 1.0
GaussianPyramid 1.0
Remap 1.0
ScaleImage 1.0
Sobel3x3 1.0
Subtract 1.0 1.0
Threshold 1.0
WarpAffine 1.0
WarpPerspective 1.0
Xor 1.0

## Outputs

Vision Function U8 U16 S16 U32 S32 F32 4CC
AbsDiff 1.0
Accumulate 1.0
AccumulateSquared 1.0
AccumulateWeighted 1.0
And 1.0
Box3x3 1.0
CannyEdgeDetector 1.0
ChannelCombine 1.0
ChannelExtract 1.0
ColorConvert 1.0
ConvertDepth 1.0 1.0
Convolve 1.0 1.0
Dilate3x3 1.0
EqualizeHistogram 1.0
Erode3x3 1.0
FastCorners 1.0
Gaussian3x3 1.0
HarrisCorners 1.0
HalfScaleGaussian 1.0
Histogram 1.0
IntegralImage 1.0
TableLookup 1.0
Magnitude 1.0
MeanStdDev 1.0
Median3x3 1.0
MinMaxLoc 1.0 1.0 1.0
Multiply 1.0 1.0
Not 1.0
OpticalFlowLK 1.0
Or 1.0
Phase 1.0
GaussianPyramid 1.0
Remap 1.0
ScaleImage 1.0
Sobel3x3 1.0
Subtract 1.0 1.0
Threshold 1.0
WarpAffine 1.0
WarpPerspective 1.0
Xor 1.0

# Lifecycles

## OpenVX Context Lifecycle

The lifecycle of the context is very simple.

The lifecycle model for an OpenVX Context.

## Graph Lifecycle

OpenVX has four main phases of graph lifecycle:

• Construction - Graphs are created via vxCreateGraph, and Nodes are connected together by data objects.
• Verification - The graphs are checked for consistency, correctness, and other conditions. Memory allocation may occur.
• Execution - The graphs are executed via vxProcessGraph or vxScheduleGraph. Between executions data may be updated by the client or some other external mechanism. The client of OpenVX may change reference of input data to a graph, but this may require the graph to be validated again by checking vxIsGraphVerified.
• Deconstruction - Graphs are released via vxReleaseGraph. All Nodes in the Graph are released.

Graph Lifecycle

## Data Object Lifecycle

All objects in OpenVX follow a similar lifecycle model. All objects are

• Created via vxCreate<Object><Method> or retreived via vxGet<Object><Method> from the parent object if they are internally created.
• Used within Graphs or immediate functions as needed.
• Then objects must be released via vxRelease<Object> or via vxReleaseContext when all objects are released.

### OpenVX Image Lifecycle

This is an example of the Image Lifecycle using the OpenVX Framework API. This would also apply to other data types with changes to the types and function names.

Image Object Lifecycle

# Host Memory Data Object Access Patterns

For objects retrieved from OpenVX that are 2D in nature, such as vx_image, vx_matrix, and vx_convolution, the manner in which the host-side has access to these memory regions is well-defined. OpenVX uses a row-major storage (that is each unit in a column is memory-adjacent to its row adjacent unit). Two-dimensional objects are always created (using vxCreateImage or vxCreateMatrix) in width (columns) by height (rows) notation, with the arguments in that order. When accessing these structures in “C” with two-dimensional arrays of declared size, the user must therefore provide the array dimensions in the reverse of the order of the arguments to the Create function. This layout ensures row-wise storage in C on the host. A pointer could also be allocated for the matrix data and would have to be indexed in this row-major method.

## Matrix Access Example

const vx_size columns = 3;
const vx_size rows = 4;
vx_matrix matrix = vxCreateMatrix(context, VX_TYPE_FLOAT32, columns, rows);
if (matrix)
{
vx_int32 j, i;
#if defined(OPENVX_USE_C99)
vx_float32 mat[rows][columns]; /* note: row major */
#else
vx_float32 *mat = (vx_float32 *)malloc(rows*columns*sizeof(vx_float32));
#endif
if (vxAccessMatrix(matrix, mat) == VX_SUCCESS) {
for (j = 0; j < rows; j++)
for (i = 0; i < columns; i++)
#if defined(OPENVX_USE_C99)
mat[j][i] = (vx_float32)rand()/(vx_float32)RAND_MAX;
#else
mat[j*columns + i] = (vx_float32)rand()/(vx_float32)RAND_MAX;
#endif
vxCommitMatrix(matrix, mat);
}
#if !defined(OPENVX_USE_C99)
free(mat);
#endif
}

## Image Access Example

Images and Array differ slightly in how they are accessed due to more complex memory layout requirements.

void *base_ptr = NULL;
vx_uint32 width = 640, height = 480, plane = 0;
vx_image image = vxCreateImage(context, width, height, VX_DF_IMAGE_U8);
rect.start_x = rect.start_y = 0;
rect.end_x = rect.end_y = PATCH_DIM;
status = vxAccessImagePatch(image, &rect, plane,
if (status == VX_SUCCESS)
{
vx_uint32 x,y,i,j;
vx_uint8 pixel = 0;
/* a couple addressing options */
/* use linear addressing function/macro */
*ptr2 = pixel;
}
*ptr2 = pixel;
}
}
* for subsampled planes, scale will change
*/
vx_uint8 *tmp = (vx_uint8 *)base_ptr;
tmp[i] = pixel;
}
}
/* more efficient direct addressing by client.
* for subsampled planes, scale will change.
*/
vx_uint8 *tmp = (vx_uint8 *)base_ptr;
tmp[i] = pixel;
}
}
/* this commits the data back to the image. If rect were 0 or empty, it
* would just decrement the reference (used when reading an image only)
*/
status = vxCommitImagePatch(image, &rect, plane, &addr, base_ptr);
}
vxReleaseImage(&image);

## Array Access Example

Arrays only require a single value, the stride, instead of the entire addressing structure that images need.

vx_size i, stride = 0ul;
void *base = NULL;
/* access entire array at once */
vxAccessArrayRange(array, 0, num_items, &stride, &base, VX_READ_AND_WRITE);
for (i = 0; i < num_items; i++)
{
vxArrayItem(mystruct, base, i, stride).some_uint += i;
vxArrayItem(mystruct, base, i, stride).some_double = 3.14f;
}
vxCommitArrayRange(array, 0, num_items, base);

Access/Commit pairs can also be called on individual elements of array using a method similar to this:

/* access each array item individually */
for (i = 0; i < num_items; i++)
{
mystruct *myptr = NULL;
vxAccessArrayRange(array, i, i+1, &stride, (void **)&myptr, VX_READ_AND_WRITE);
myptr->some_uint += 1;
myptr->some_double = 3.14f;
vxCommitArrayRange(array, i, i+1, (void *)myptr);
}

# Extending OpenVX

Beyond User Kernels there are other mechanisms for vendors to extend features in OpenVX. These mechanisms are not available to User Kernels.

## Extending Attributes

When extending attributes, vendors must use their assigned ID from vx_vendor_id_e in conjunction with the appropriate macros for creating new attributes with VX_ATTRIBUTE_BASE. The typical mechanism to extend a new attribute for some object type (for example a vx_node attribute from VX_ID_TI) would look like this:

enum {
VX_NODE_ATTRIBUTE_TI_NEWTHING = VX_ATTRIBUTE_BASE(VX_ID_TI, VX_TYPE_NODE) + 0x0,
}

## Vendor Custom Kernels

Vendors wanting to add more kernels to the base set supplied to OpenVX should provide a header of the form

#include <VX/vx_ext_<vendor>.h>

that contains definitions of each of the following.

• New Node Creation Function Prototype per function.
vx_node vxXYZNode(vx_graph graph, vx_image input, vx_uint32 value, vx_image output, vx_array temp);
• A new Kernel Enumeration(s) and Kernel String per function.
#define VX_KERNEL_NAME_KHR_XYZ "org.khronos.example.xyz"
#define VX_LIBRARY_XYZ (0x3) // assigned from Khronos, vendors control their own
enum vx_kernel_xyz_ext_e {
VX_KERNEL_KHR_XYZ = VX_KERNEL_BASE(VX_ID_KHRONOS, VX_LIBRARY_XYZ) + 0x0,
// up to 0xFFF kernel enums can be created.
};
• A new VXU Function per function.
vx_status vxuXYZ(vx_context context, vx_image input, vx_uint32 value, vx_image output, vx_array temp);
This should come with good documentation for each new part of the extension. Ideally, these sorts of extensions should not require linking to new objects to facilitate usage.

## Vendor Custom Extensions

Some extensions affect base vision functions and thus may be invisible to most users. In these circumstances, the vendor must report the supported extensions to the base nodes through the VX_CONTEXT_ATTRIBUTE_EXTENSIONS attribute on the context.

vx_char *tmp, *extensions = NULL;
vx_size size = 0ul;
extensions = malloc(size);
extensions, size);

Extensions in this list are dependent on the extension itself; they may or may not have a header and new kernels or framework feature or data objects. The common feature is that they are implemented and supported by the implementation vendor.

## Hinting

The specification defines a Hinting API that allows Clients to feed information to the implementation for optional behavior changes. See Framework: Hints. It is assumed that most of the hints will be vendor- or implementation-specific. Check with the OpenVX implementation vendor for information on vendor-specific extensions.

## Directives

The specification defines a Directive API to control implementation behavior. See Framework: Directives. This may allow things like disabling parallelism for debugging, enabling cache writing-through for some buffers, or any implementation-specific optimization.

# Known Extensions to OpenVX

## User Kernel Tiling

The User Kernel Tiling facility enables optimizations of the user kernels (e.g., locality of execution or parallelism) when performing computation on the image data. Modern processors have a diverse memory hierarchy that varies from relatively small but fast and expensive memory to relatively large but slow and inexpensive memory. Image data are typically too large to fit into the fast but small memory. The ability to break the image data into smaller sized units allows for optimized computation on these smaller units with fast memory access or parallel execution of a user kernel on multiple image tiles simultaneously. The OpenVX Graph Manager possesses the knowledge about the memory hierarchy of the platform and is hence in a position to break the image data into smaller units for memory optimization. Knowledge of the memory access pattern of an algorithm is key for the graph manager to enable optimizations.

The Khronos OpenVX Working Group will include this extension as part of the OpenVX 1.1 specification, contingent on community feedback.