BilateralFilter function

[egorovp]

Problem

1. The input tensor may have 2 or 3 dimensions.
   In case of 3 dimensions the size of the first dimension can be 1 or 2.
   Why it can be changed from 1 to 20 in CTS?
2. According to documentation supported border mode shall be replicate or constant. Why it is set to undefined in CTS? Since it is undefined in CTS, then why the resulting values of borders are still checked?
3. In case of 3 dimensions the first dimension is "radiometric" and the filtering goes by summing averaged weighted pixels in the window.
   Weight for every pixel in the window depends on the distance to the center of the window and on the difference in radiometric space between these two pixels.
   Why is radiometric space ignored in CTS?

Links

1. Standard
2. Conformance Tests

[fbrill3]

Here is some feedback on each of your three questions from one of our group members:

1. In the referred code line, the tensor_dims[] is to store the size of the each dimension of the tensor. The dimension of tensor is specified by dims. 1 to 20 is the range that a given dimension of the tensor will take (as generated randomly within this range). See the following vxCreateTensor() call to understand the usage of dims and tensor_dims.
2. Both Sample Implementation and CTS does not take into account border pixels at all (border pixels are ignored). This is noted in the OpenVX internally as bug to be fixed with the next version of OpenVX (1.3) CTS.
3. Could you please refer to the CTS code for your comment? Radiometric space is not ignored in the CTS.
   See in test_bilateralfilter.c function ownCheckBilateralFilterResult() that both the distance from the center as well as the difference in pixel value (color) are taken into account to calculate the resulting weight of the pixel.
   w = space_weight[j+radius]*color_weight_8[abs(nei - in)];
   Additionally, following is a known issue in the CTS for Bilateral Filter, which has been noted by the group:
   “Bilateral filter conformance test requires that the reference implementation under test uses a kernel that is one dimensional only (1x5) although the spec requires a 2D kernel indirectly (assuming the parameter diameter indicates a 2D square kernel). Any implementation performing bilateral filtering with a two-dimensional kernel (e.g. 5x5) will fail conformance.”

[egorovp]

@fbrill3,
1)
Quote from standard:

In case of 3 dimensions the 1st dimension of the vx_tensor. Which can be of size 1 or 2.

The size of the first dimension should be randomly chosen from the range [1,2].

Why is it chosen from the range [1,20]?

---

Out_count is number of all cells in the tensor.
All pixels except some (some = the value of radius) of the first and some of the last are processed sequentially one by one. It means that in the CTS input tensor is processed as it is a line (tensor of 1 dimension).
If it will be so, then border mode is undefined and the values in borders are not checked.

But why you have this implementation for input tensors of 2 or 3 dimensions?

Example: radius = 1, tensor_dims = {3, 3}               Or if now it's implemented 1-D filtering
CTS: x o o          Border mode is undefined: x x x     Border mode is undefined: x o x
     o o o                                    x o x                               x o x
     o o x                                    x x x                               x o x
x - ignored pixels
o - processed pixels

---

In addition to 2nd paragraph, this loop means that the whole tensor is processed identically.
Why the radiometric space is filtered similary as filtering goes into the color space?

Why in calculation of weights instead of values from radiometric space there are always taken values of color space?

Example:
In calculation of weight for color_weight_16 should be passed values of nei and in from radiometric space,
and in summation the value for nei from color space.
(nei and in are coordinates from image, and input tensor has for each coordinate 2 values: color value and radiometric value)

[fbrill3]

@egorovp , thanks for the follow-up; we'll review and get back.

[fbrill3]

@egorovp , on further review, we agree that the sample implementation and CTS do not fully/correctly implement the spec, and they need to be updated. We will reply to the specific issues you point out shortly.

[fbrill3]

@egorovp,

Here are some responses to your latest questions.

The size of the first dimension should be randomly chosen from the range [1,2].
Why is it chosen from the range [1,20]?

You are correct, the first dimension can only have a value of 1 or 2 as per the spec. CTS and Sample code is incorrect in choosing a range of [1,20]. We will correct it in the next release.

But why you have this implementation for input tensors of 2 or 3 dimensions?

The handling of 3-dimensional tensor is not complete and accurate. First, as you noted the first dimension is limited to size of 1 or 2 only. Second, the bilateral filtering of each plane, that is [1, y, z] or [2, y, z], should be done by processing the relevant pixels/data in the second and third dimension of the tensor. The pixels should not be combined across the planes (Not sure if there could be variations of this in other implementations. E.g. OpenCV for RGB could be combining RGB value into a combination for processing – to be verified). See the explanation to your item 3 below for more on this.

This is just clarification. In the diagram you have provided, for a Border mode of ‘undefined’ it does not matter whether the edge pixels are processed or not. The value of these pixels is not relevant for testing the output. In present implementation, the 1-D (1x5) kernel ignores the first few columns (size of radius) of the first row and the last few columns of the last row. It processes the first and last few columns for each row in between. It could have been chosen to ignore these columns as well but for a border mode ‘undefined’ these pixels are irrelevant. We are in process of updating the code to make this ‘kernel’ 2-D (5x5) and process the 2D tensor the same way as currently.

In addition to 2nd paragraph, this loop means that the whole tensor is processed identically.
Why the radiometric space is filtered similary as filtering goes into the color space?
Why in calculation of weights instead of values from radiometric space there are always taken values of color space?

We will check if the processing of 3D tensor is being performed correctly and fix any shortcomings. To clarify the shortcoming here is the explanation: The spec states that the first dimension containing radiometric info can be used for multiple types of info (color, depth or movement). But it does not clarify what the expected behavior is for each of this radiometric space, which theoretically can differ based on implementation or the purpose of the application. The planes along the first dimension could be treated entirely independent and bilateral filter applied on each of them separately. Alternatively, an implementation may use some combination of the values in different planes and use a combination value as radiometric/spatial value.

As a reference, for color space (e.g. RGB), OpenCV has an implementation that can be used to model the OpenVX handling of the color image planes for bilateral filtering. Additional formats for which there are no well-defined usage (e.g. depth, movement etc) an optional/no guidance should be provided by spec.

It must be confirmed though if OpenCV RGB image/3-channel layout can be seamlessly adopted by OpenVX given its tensor structure has a required dimension order [radiometric, width, height]. For the above the OpenVX group will work on providing a clear implementation and conformance test guideline and make it available in future releases (post 1.3 spec version as the 1.3 version is already frozen for release)

[egorovp]

@fbrill3, thank you for all your responses. Looking forward to new releases.

[kiritigowda]

Thank you for the detailed report. The OpenVX specification is the source of truth. All three issues identified are CTS bugs:

First dimension range — The CTS incorrectly allows values 1–20 for the radiometric dimension; this should be restricted per spec.
Border mode inconsistency — The spec clearly states only VX_BORDER_REPLICATE and VX_BORDER_CONSTANT are supported. The CTS setting undefined border mode while still checking border pixel values is contradictory and should be fixed.
3D radiometric handling — The CTS does not properly account for the radiometric dimension in weight calculations for 3D tensors.
All three will be addressed in the 1.3.2 CTS release.

A CTS tracking issue has been filed: KhronosGroup/OpenVX-cts#34

The specification is correct as written — no documentation changes required.