The OpenVX Specification  dba1aa3
Optical Flow Pyramid (LK)

## Detailed Description

Computes the optical flow using the Lucas-Kanade method between two pyramid images.

The function is an implementation of the algorithm described in [1]. The function inputs are two vx_pyramid objects, old and new, along with a vx_array of vx_keypoint_t structs to track from the old vx_pyramid. Both pyramids old and new pyramids must have the same dimensionality. VX_SCALE_PYRAMID_HALF pyramidal scaling must be supported.
The function outputs a vx_array of vx_keypoint_t structs that were tracked from the old vx_pyramid to the new vx_pyramid. Each element in the vx_array of vx_keypoint_t structs in the new array may be valid or not. The implementation shall return the same number of vx_keypoint_t structs in the new vx_array that were in the older vx_array.

In more detail: The Lucas-Kanade method finds the affine motion vector $$V$$ for each point in the old image tracking points array, using the following equation:

$\begin{bmatrix} V_x \\ V_y \end{bmatrix} = \begin{bmatrix} \sum_{i}{I_x}^2 & \sum_{i}{I_x*I_y} \\ \sum_{i}{I_x*I_y} & \sum_{i}{I_y}^2 \end{bmatrix}^{-1} \begin{bmatrix} -\sum_{i}{I_x*I_t} \\ -\sum_{i}{I_y*I_t} \end{bmatrix}$

Where $$I_x$$ and $$I_y$$ are obtained using the Scharr gradients on the input image:

$G_x = \begin{bmatrix} +3 & 0 & -3 \\ +10 & 0 & -10 \\ +3 & 0 & -3 \end{bmatrix}$

$G_y = \begin{bmatrix} +3 & +10 & +3 \\ 0 & 0 & 0 \\ -3 & -10 & -3 \end{bmatrix}$

$$I_t$$ is obtained by a simple difference between the same pixel in both images. $$I$$ is defined as the adjacent pixels to the point $$p(x,y)$$ under consideration. With a given window size of $$M$$, $$I$$ is $$M^2$$ points. The pixel $$p(x,y)$$ is centered in the window. In practice, to get an accurate solution, it is necessary to iterate multiple times on this scheme (in a Newton-Raphson fashion) until:

• the residual of the affine motion vector is smaller than a threshold
• And/or maximum number of iteration achieved. Each iteration, the estimation of the previous iteration is used by changing $$I_t$$ to be the difference between the old image and the pixel with the estimated coordinates in the new image. Each iteration the function checks if the pixel to track was lost. The criteria for lost tracking is that the matrix above is invertible. (The determinant of the matrix is less than a threshold : $$10^{-7}$$ .) Or the minimum eigenvalue of the matrix is smaller then a threshold ( $$10^{-4}$$ ). Also lost tracking happens when the point tracked coordinate is outside the image coordinates. When vx_true_e is given as the input to use_initial_estimates, the algorithm starts by calculating $$I_t$$ as the difference between the old image and the pixel with the initial estimated coordinates in the new image. The input vx_array of vx_keypoint_t structs with tracking_status set to zero (lost) are copied to the new vx_array.

Clients are responsible for editing the output vx_array of vx_keypoint_t structs array before applying it as the input vx_array of vx_keypoint_t structs for the next frame. For example, vx_keypoint_t structs with tracking_status set to zero may be removed by a client for efficiency.

This function changes just the x, y, and tracking_status members of the vx_keypoint_t structure and behaves as if it copied the rest from the old tracking vx_keypoint_t to new image vx_keypoint_t.

## Functions

vx_node VX_API_CALL vxOpticalFlowPyrLKNode (vx_graph graph, vx_pyramid old_images, vx_pyramid new_images, vx_array old_points, vx_array new_points_estimates, vx_array new_points, vx_enum termination, vx_scalar epsilon, vx_scalar num_iterations, vx_scalar use_initial_estimate, vx_size window_dimension)
[Graph] Creates a Lucas Kanade Tracking Node. More...

vx_status VX_API_CALL vxuOpticalFlowPyrLK (vx_context context, vx_pyramid old_images, vx_pyramid new_images, vx_array old_points, vx_array new_points_estimates, vx_array new_points, vx_enum termination, vx_scalar epsilon, vx_scalar num_iterations, vx_scalar use_initial_estimate, vx_size window_dimension)
[Immediate] Computes an optical flow on two images. More...

## Function Documentation

 vx_node VX_API_CALL vxOpticalFlowPyrLKNode ( vx_graph graph, vx_pyramid old_images, vx_pyramid new_images, vx_array old_points, vx_array new_points_estimates, vx_array new_points, vx_enum termination, vx_scalar epsilon, vx_scalar num_iterations, vx_scalar use_initial_estimate, vx_size window_dimension )

[Graph] Creates a Lucas Kanade Tracking Node.

Parameters
 [in] graph The reference to the graph. [in] old_images Input of first (old) image pyramid in VX_DF_IMAGE_U8. [in] new_images Input of destination (new) image pyramid VX_DF_IMAGE_U8. [in] old_points An array of key points in a vx_array of VX_TYPE_KEYPOINT; those key points are defined at the old_images high resolution pyramid. [in] new_points_estimates An array of estimation on what is the output key points in a vx_array of VX_TYPE_KEYPOINT; those keypoints are defined at the new_images high resolution pyramid. [out] new_points An output array of key points in a vx_array of VX_TYPE_KEYPOINT; those key points are defined at the new_images high resolution pyramid. [in] termination The termination can be VX_TERM_CRITERIA_ITERATIONS or VX_TERM_CRITERIA_EPSILON or VX_TERM_CRITERIA_BOTH. [in] epsilon The vx_float32 error for terminating the algorithm. [in] num_iterations The number of iterations. Use a VX_TYPE_UINT32 scalar. [in] use_initial_estimate Use a VX_TYPE_BOOL scalar. [in] window_dimension The size of the window on which to perform the algorithm. See VX_CONTEXT_OPTICAL_FLOW_MAX_WINDOW_DIMENSION
Returns
vx_node.
Return values
 vx_node A node reference. Any possible errors preventing a successful creation should be checked using vxGetStatus
 vx_status VX_API_CALL vxuOpticalFlowPyrLK ( vx_context context, vx_pyramid old_images, vx_pyramid new_images, vx_array old_points, vx_array new_points_estimates, vx_array new_points, vx_enum termination, vx_scalar epsilon, vx_scalar num_iterations, vx_scalar use_initial_estimate, vx_size window_dimension )

[Immediate] Computes an optical flow on two images.

Parameters
 [in] context The reference to the overall context. [in] old_images Input of first (old) image pyramid in VX_DF_IMAGE_U8. [in] new_images Input of destination (new) image pyramid in VX_DF_IMAGE_U8 [in] old_points an array of key points in a vx_array of VX_TYPE_KEYPOINT those key points are defined at the old_images high resolution pyramid [in] new_points_estimates an array of estimation on what is the output key points in a vx_array of VX_TYPE_KEYPOINT those keypoints are defined at the new_images high resolution pyramid [out] new_points an output array of key points in a vx_array of VX_TYPE_KEYPOINT those key points are defined at the new_images high resolution pyramid [in] termination termination can be VX_TERM_CRITERIA_ITERATIONS or VX_TERM_CRITERIA_EPSILON or VX_TERM_CRITERIA_BOTH [in] epsilon is the vx_float32 error for terminating the algorithm [in] num_iterations is the number of iterations. Use a VX_TYPE_UINT32 scalar. [in] use_initial_estimate Can be set to either vx_false_e or vx_true_e. [in] window_dimension The size of the window on which to perform the algorithm. See VX_CONTEXT_OPTICAL_FLOW_MAX_WINDOW_DIMENSION
Returns
A vx_status_e enumeration.
Return values
 VX_SUCCESS Success * An error occurred. See vx_status_e.