Type Coercion Exception while performing a MAC operation with signed values over unsigned buffers

[crmartin781]

Hi there -

I've re-worked my RGB -> YUV420 conversion into a Generator class defined as follows:

using conv_mat_row = std::array<int, 3>;

// Define a Generator to convert RGB to planar YUV420
class RGBtoYUV420 : public Halide::Generator<RGBtoYUV420> {
  Var x,y,c;

  /* note:  if you're wondering why the 'excessive' use of curly braces, 
   * look here:  https://stackoverflow.com/questions/12844475/why-cant-simple-initialize-with-braces-2d-stdarray 
   */
  std::array<conv_mat_row, 3> conv1 = { {
         { {  66,  129,  25 } },
	 { { -38, -74,  112 } },
	 { { 112, -94, -18   } } 
    } };

//etc 
  
public:
  Input<Buffer<uint8_t, 3>> input{"rgb_input"};
  Output<Buffer<uint8_t>> outY{"i420_Y_output"};
  Output<Buffer<uint8_t>> outU{"i420_U_output"};
  Output<Buffer<uint8_t>> outV{"i420_V_output"};


  void generate() {
    
    Func y_part("Luma"), u_part("Chroma-U"), v_part("Chroma-V");
    Func avg_rgb("avg_rgb"), avg_rgb_x("avg_rgb_x");

    Expr yval = input(x, y, c);
    Expr uval = input(x, y, c);
    Expr vval = input(x, y, c);
    
    //more code, that is not of issue to the discussion...

    //apply yval(r, g, b) transform, 1-1 sampled from image
    yval = conv1[0][0] * input(x, y, 0) + conv1[0][1] * input(x, y, 1) + conv1[0][2] * input(x, y, 2);

    //etc...

    //apply uval(r,g,b) transform, with averaged rgb samples, over 2x2 sub-areas
    uval = conv1[1][0] * avg_rgb(2 * x, 2 * y, 0) - conv1[1][1] * avg_rgb(2 * x, 2 * y, 1) + conv1[1][2] * avg_rgb(2 * x, 2 * y, 2);

   //similar code for vval

 // Output planar YUV420
    outY(x,y) = y_part(x,y);
    outU(x,y) = u_part(x,y);
    outV(x,y) = v_part(x,y);
  } //end generate() method

 void schedule() {
  }
};

HALIDE_REGISTER_GENERATOR(RGBtoYUV420, rgb2i420)

Where things start to go sideways, is during the evaluation of uval Expr. Due to the fact that conv1[1][0] is < 0, the generator throw the following exception message:

Unhandled exception: Error: Integer constant -38 will be implicitly coerced to type uint8, which changes its value to (uint8)218.

I tried Halide::cast<int8_t>(input(x,y,c)) the Expr's, but that didn't seem to work.
So, how do I get past this coercion problem, properly?

Thanks, Charles.

[crmartin781]

So, I got past my immediate problems with type-coercion, with careful use of Halide::cast<>.
However, the halide-generator that results from my build throws again.
Upon isolating, I discovered this code fragment is the issue:

Expr roundup_and_add_bias(Expr val, int8_t bias)
  {
    val = val + 127;
    //val = val / cast<int16_t>(256); //the generator likes this...
    val = val >> 8; //the generator throws...WHY?
    val = val + bias + 1;
    return val;
  }

I get this exception:

Unhandled exception: Error: Signed integer overflow occurred during constant-folding. Signed integer overflow for int32 and int64 is undefined behavior in Halide.

To me, this smells like a bug, since the Expr class has trouble dealing with certain type-combinations of the right-shift operator (>>), and possibly left-shift (<<).

Is there a document explaining these exception objects?

[mcourteaux]

It's important to know the type of the val argument. When that is still uint8, then shifting 8 bits left or right, fully wipes the number. This is checked in Simplify_Call.cpp at line 148.