TJ: Explicitly reject OOB lossless crop regions

tj*Transform() relied upon the underlying transupp API to check the
cropping region.  However, transupp uses unsigned integers for the
cropping region, whereas the tjregion structure uses signed integers.
Thus, casting negative values from a tjregion structure produced very
large unsigned values.  In the case of the left and upper boundary, this
was innocuous, because jtransform_request_workspace() rejected the
values as being out of bounds.  However, jtransform_request_workspace()
did not always reject very large width and height values, because it
supports expanding the destination image by specifying a cropping region
larger than the source image.  In certain cases, it allowed those
values, and the libjpeg memory manager subsequently ran out of memory.

NOTE: Prior to this commit, image expansion technically worked with
tj*Transform() as long as the cropping width and height were valid and
automatic JPEG buffer (re)allocation was used.  However, that behavior
is not a documented feature of the TurboJPEG API, nor do we have any way
of testing it at the moment.  Official support for image expansion can
be added later, if there is sufficient demand for it.

Similarly, this commit modifies tj3SetCroppingRegion() so that it
explicitly checks for left boundary values exactly equal to the scaled
image width and upper boundary values exactly equal to the scaled image
height.  If the specified cropping width or height was 0 (which is
interpreted as {scaled image width} - {left boundary} or
{scaled image height} - {upper boundary}), then such values caused a
cropping width or height of 0 to be passed to the libjpeg API.  In the
case of the width, this was innocuous, because jpeg_crop_scanline()
rejected the value.  In the case of the height, however, this caused
unexpected and hard-to-diagnose errors farther down the pipeline.

turbojpeg.c

@@ -1947,7 +1947,7 @@ DLLEXPORT int tj3SetCroppingRegion(tjhandle handle, tjregion croppingRegion)
     croppingRegion.w = scaledWidth - croppingRegion.x;
   if (croppingRegion.h == 0)
     croppingRegion.h = scaledHeight - croppingRegion.y;
-  if (croppingRegion.w < 0 || croppingRegion.h < 0 ||
+  if (croppingRegion.w <= 0 || croppingRegion.h <= 0 ||
       croppingRegion.x + croppingRegion.w > scaledWidth ||
       croppingRegion.y + croppingRegion.h > scaledHeight)
     THROW("The cropping region exceeds the scaled image dimensions");
@@ -2712,6 +2712,8 @@ DLLEXPORT int tj3Transform(tjhandle handle, const unsigned char *jpegBuf,
     else xinfo[i].slow_hflip = 0;
 
     if (xinfo[i].crop) {
+      if (t[i].r.x < 0 || t[i].r.y < 0 || t[i].r.w < 0 || t[i].r.h < 0)
+        THROW("Invalid cropping region");
       xinfo[i].crop_xoffset = t[i].r.x;  xinfo[i].crop_xoffset_set = JCROP_POS;
       xinfo[i].crop_yoffset = t[i].r.y;  xinfo[i].crop_yoffset_set = JCROP_POS;
       if (t[i].r.w != 0) {
@@ -2764,20 +2766,23 @@ DLLEXPORT int tj3Transform(tjhandle handle, const unsigned char *jpegBuf,
   srccoefs = jpeg_read_coefficients(dinfo);
 
   for (i = 0; i < n; i++) {
-    int w, h;
+    int dstWidth = dinfo->image_width, dstHeight = dinfo->image_height;
     int dstSubsamp = (t[i].options & TJXOPT_GRAY) ? TJSAMP_GRAY : srcSubsamp;
 
-    if (!xinfo[i].crop) {
+    if (t[i].op == TJXOP_TRANSPOSE || t[i].op == TJXOP_TRANSVERSE ||
-      w = dinfo->image_width;  h = dinfo->image_height;
+        t[i].op == TJXOP_ROT90 || t[i].op == TJXOP_ROT270) {
-      if (t[i].op == TJXOP_TRANSPOSE || t[i].op == TJXOP_TRANSVERSE ||
+      dstWidth = dinfo->image_height;  dstHeight = dinfo->image_width;
-          t[i].op == TJXOP_ROT90 || t[i].op == TJXOP_ROT270) {
+    }
-        w = dinfo->image_height;  h = dinfo->image_width;
+
-      }
+    if (xinfo[i].crop) {
-    } else {
+      if (t[i].r.x >= dstWidth || t[i].r.x + xinfo[i].crop_width > dstWidth ||
-      w = xinfo[i].crop_width;  h = xinfo[i].crop_height;
+          t[i].r.y >= dstHeight || t[i].r.y + xinfo[i].crop_height > dstHeight)
+        THROW("The cropping region exceeds the destination image dimensions");
+      dstWidth = xinfo[i].crop_width;  dstHeight = xinfo[i].crop_height;
     }
     if (this->noRealloc) {
-      alloc = FALSE;  dstSizes[i] = tj3JPEGBufSize(w, h, dstSubsamp);
+      alloc = FALSE;
+      dstSizes[i] = tj3JPEGBufSize(dstWidth, dstHeight, dstSubsamp);
     }
     if (!(t[i].options & TJXOPT_NOOUTPUT))
       jpeg_mem_dest_tj(cinfo, &dstBufs[i], &dstSizes[i], alloc);