TJ: Calc. xformed buf sizes based on dst. subsamp

With respect to tj3Transform(), this addresses an oversight from
bb1d540a80.

Note to self: A convenience function/method for computing the worst-case
transformed JPEG size for a particular transform would be nice.

doc/html/group___turbo_j_p_e_g.html

@@ -3173,7 +3173,7 @@ If you choose option 1, then <code>*jpegSize</code> should be set to the size of
     <tr><td class="paramname">dstBufs</td><td>pointer to an array of n byte buffers. <code>dstBufs[i]</code> will receive a JPEG image that has been transformed using the parameters in <code>transforms[i]</code>. TurboJPEG has the ability to reallocate the JPEG destination buffer to accommodate the size of the transformed JPEG image. Thus, you can choose to:<ol type="1">
 <li>pre-allocate the JPEG destination buffer with an arbitrary size using <a class="el" href="group___turbo_j_p_e_g.html#ga1a2c96d8b47530b6e6050ba6f10b7c57" title="Allocate a byte buffer for use with TurboJPEG.">tj3Alloc()</a> and let TurboJPEG grow the buffer as needed,</li>
 <li>set <code>dstBufs[i]</code> to NULL to tell TurboJPEG to allocate the buffer for you, or</li>
-<li>pre-allocate the buffer to a "worst case" size determined by calling <a class="el" href="group___turbo_j_p_e_g.html#gac6285e58e35a35d871d7162ec5a929c4" title="The maximum size of the buffer (in bytes) required to hold a JPEG image with the given parameters.">tj3JPEGBufSize()</a> with the transformed or cropped width and height and the level of subsampling used in the destination image. Under normal circumstances, this should ensure that the buffer never has to be re-allocated. (Setting <a class="el" href="group___turbo_j_p_e_g.html#ggaa0f6be63ba78278299c9f5c12031fe82ae64ffb358bc7b194fd48e0f27750b29b" title="JPEG destination buffer (re)allocation [compression, lossless transformation].">TJPARAM_NOREALLOC</a> guarantees that it won't be.) Note, however, that there are some rare cases (such as transforming images with a large amount of embedded Exif or ICC profile data) in which the transformed JPEG image will be larger than the worst-case size, and <a class="el" href="group___turbo_j_p_e_g.html#ggaa0f6be63ba78278299c9f5c12031fe82ae64ffb358bc7b194fd48e0f27750b29b" title="JPEG destination buffer (re)allocation [compression, lossless transformation].">TJPARAM_NOREALLOC</a> cannot be used in those cases.</li>
+<li>pre-allocate the buffer to a "worst case" size determined by calling <a class="el" href="group___turbo_j_p_e_g.html#gac6285e58e35a35d871d7162ec5a929c4" title="The maximum size of the buffer (in bytes) required to hold a JPEG image with the given parameters.">tj3JPEGBufSize()</a> with the transformed or cropped width and height and the level of subsampling used in the destination image (taking into account grayscale conversion and transposition of the width and height.) Under normal circumstances, this should ensure that the buffer never has to be re-allocated. (Setting <a class="el" href="group___turbo_j_p_e_g.html#ggaa0f6be63ba78278299c9f5c12031fe82ae64ffb358bc7b194fd48e0f27750b29b" title="JPEG destination buffer (re)allocation [compression, lossless transformation].">TJPARAM_NOREALLOC</a> guarantees that it won't be.) Note, however, that there are some rare cases (such as transforming images with a large amount of embedded Exif or ICC profile data) in which the transformed JPEG image will be larger than the worst-case size, and <a class="el" href="group___turbo_j_p_e_g.html#ggaa0f6be63ba78278299c9f5c12031fe82ae64ffb358bc7b194fd48e0f27750b29b" title="JPEG destination buffer (re)allocation [compression, lossless transformation].">TJPARAM_NOREALLOC</a> cannot be used in those cases unless the embedded data is discarded using <a class="el" href="group___turbo_j_p_e_g.html#ga153b468cfb905d0de61706c838986fe8" title="Do not copy any extra markers (including Exif and ICC profile data) from the source image to the dest...">TJXOPT_COPYNONE</a>.</li>
 </ol>
 If you choose option 1, then <code>dstSizes[i]</code> should be set to the size of your pre-allocated buffer. In any case, unless you have set <a class="el" href="group___turbo_j_p_e_g.html#ggaa0f6be63ba78278299c9f5c12031fe82ae64ffb358bc7b194fd48e0f27750b29b" title="JPEG destination buffer (re)allocation [compression, lossless transformation].">TJPARAM_NOREALLOC</a>, you should always check <code>dstBufs[i]</code> upon return from this function, as it may have changed.</td></tr>
     <tr><td class="paramname">dstSizes</td><td>pointer to an array of n size_t variables that will receive the actual sizes (in bytes) of each transformed JPEG image. If <code>dstBufs[i]</code> points to a pre-allocated buffer, then <code>dstSizes[i]</code> should be set to the size of the buffer. Upon return, <code>dstSizes[i]</code> will contain the size of the transformed JPEG image (in bytes.)</td></tr>

fuzz/transform.cc

@@ -102,11 +102,11 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
                           TJXOPT_OPTIMIZE;
   dstBufs[0] =
     (unsigned char *)tj3Alloc(tj3JPEGBufSize((height + 1) / 2, (width + 1) / 2,
-                                             jpegSubsamp));
+                                             TJSAMP_GRAY));
   if (!dstBufs[0])
     goto bailout;
 
-  maxBufSize = tj3JPEGBufSize((height + 1) / 2, (width + 1) / 2, jpegSubsamp);
+  maxBufSize = tj3JPEGBufSize((height + 1) / 2, (width + 1) / 2, TJSAMP_GRAY);
 
   if (tj3Transform(handle, data, size, 1, dstBufs, dstSizes,
                    transforms) == 0) {

java/TJBench.java

@@ -629,6 +629,22 @@ final class TJBench {
       }
 
       tsubsamp = subsamp;
+      if ((xformOpt & TJTransform.OPT_GRAY) != 0)
+        tsubsamp = TJ.SAMP_GRAY;
+      if (xformOp == TJTransform.OP_TRANSPOSE ||
+          xformOp == TJTransform.OP_TRANSVERSE ||
+          xformOp == TJTransform.OP_ROT90 ||
+          xformOp == TJTransform.OP_ROT270) {
+        if (tsubsamp == TJ.SAMP_422)
+          tsubsamp = TJ.SAMP_440;
+        else if (tsubsamp == TJ.SAMP_440)
+          tsubsamp = TJ.SAMP_422;
+        else if (tsubsamp == TJ.SAMP_411)
+          tsubsamp = TJ.SAMP_441;
+        else if (tsubsamp == TJ.SAMP_441)
+          tsubsamp = TJ.SAMP_411;
+      }
+
       if (doTile || xformOp != TJTransform.OP_NONE || xformOpt != 0 ||
           customFilter != null) {
         if (xformOp == TJTransform.OP_TRANSPOSE ||
@@ -641,40 +657,22 @@ final class TJBench {
         if (xformOp != TJTransform.OP_NONE &&
             xformOp != TJTransform.OP_TRANSPOSE && subsamp == TJ.SAMP_UNKNOWN)
           throw new Exception("Could not determine subsampling level of JPEG image");
-        if ((xformOpt & TJTransform.OPT_GRAY) != 0)
-          tsubsamp = TJ.SAMP_GRAY;
         if (xformOp == TJTransform.OP_HFLIP ||
+            xformOp == TJTransform.OP_TRANSVERSE ||
+            xformOp == TJTransform.OP_ROT90 ||
             xformOp == TJTransform.OP_ROT180)
           tw = tw - (tw % TJ.getMCUWidth(tsubsamp));
         if (xformOp == TJTransform.OP_VFLIP ||
-            xformOp == TJTransform.OP_ROT180)
+            xformOp == TJTransform.OP_TRANSVERSE ||
-          th = th - (th % TJ.getMCUHeight(tsubsamp));
+            xformOp == TJTransform.OP_ROT180 ||
-        if (xformOp == TJTransform.OP_TRANSVERSE ||
-            xformOp == TJTransform.OP_ROT90)
-          tw = tw - (tw % TJ.getMCUHeight(tsubsamp));
-        if (xformOp == TJTransform.OP_TRANSVERSE ||
             xformOp == TJTransform.OP_ROT270)
-          th = th - (th % TJ.getMCUWidth(tsubsamp));
+          th = th - (th % TJ.getMCUHeight(tsubsamp));
         tntilesw = (tw + ttilew - 1) / ttilew;
         tntilesh = (th + ttileh - 1) / ttileh;
 
-        if (xformOp == TJTransform.OP_TRANSPOSE ||
-            xformOp == TJTransform.OP_TRANSVERSE ||
-            xformOp == TJTransform.OP_ROT90 ||
-            xformOp == TJTransform.OP_ROT270) {
-          if (tsubsamp == TJ.SAMP_422)
-            tsubsamp = TJ.SAMP_440;
-          else if (tsubsamp == TJ.SAMP_440)
-            tsubsamp = TJ.SAMP_422;
-          else if (tsubsamp == TJ.SAMP_411)
-            tsubsamp = TJ.SAMP_441;
-          else if (tsubsamp == TJ.SAMP_441)
-            tsubsamp = TJ.SAMP_411;
-        }
-
         TJTransform[] t = new TJTransform[tntilesw * tntilesh];
         jpegBufs =
-          new byte[tntilesw * tntilesh][TJ.bufSize(ttilew, ttileh, subsamp)];
+          new byte[tntilesw * tntilesh][TJ.bufSize(ttilew, ttileh, tsubsamp)];
 
         for (y = 0, tile = 0; y < th; y += ttileh) {
           for (x = 0; x < tw; x += ttilew, tile++) {
@@ -737,7 +735,7 @@ final class TJBench {
       } else {
         if (quiet == 1)
           System.out.print("N/A     N/A     ");
-        jpegBufs = new byte[1][TJ.bufSize(ttilew, ttileh, subsamp)];
+        jpegBufs = new byte[1][TJ.bufSize(ttilew, ttileh, tsubsamp)];
         jpegSizes = new int[1];
         jpegBufs[0] = srcBuf;
         jpegSizes[0] = srcSize;

java/doc/org/libjpegturbo/turbojpeg/TJTransformer.html

@@ -379,7 +379,8 @@ extends <a href="TJDecompressor.html" title="class in org.libjpegturbo.turbojpeg
  transformed using the parameters in <code>transforms[i]</code>.  Use
  <a href="TJ.html#bufSize(int,int,int)"><code>TJ.bufSize()</code></a> to determine the maximum size for each
  buffer based on the transformed or cropped width and height and the level
- of subsampling used in the destination image.</dd>
+ of subsampling used in the destination image (taking into account
+ grayscale conversion and transposition of the width and height.)</dd>
 <dd><code>transforms</code> - an array of <a href="TJTransform.html" title="class in org.libjpegturbo.turbojpeg"><code>TJTransform</code></a> instances, each of
  which specifies the transform parameters and/or cropping region for the
  corresponding transformed JPEG image</dd>

java/org/libjpegturbo/turbojpeg/TJTransformer.java

@@ -89,7 +89,8 @@ public class TJTransformer extends TJDecompressor {
    * transformed using the parameters in <code>transforms[i]</code>.  Use
    * {@link TJ#bufSize TJ.bufSize()} to determine the maximum size for each
    * buffer based on the transformed or cropped width and height and the level
-   * of subsampling used in the destination image.
+   * of subsampling used in the destination image (taking into account
+   * grayscale conversion and transposition of the width and height.)
    *
    * @param transforms an array of {@link TJTransform} instances, each of
    * which specifies the transform parameters and/or cropping region for the
@@ -130,14 +131,33 @@ public class TJTransformer extends TJDecompressor {
     byte[][] dstBufs = new byte[transforms.length][];
     if (getWidth() < 1 || getHeight() < 1)
       throw new IllegalStateException("JPEG buffer not initialized");
-    checkSubsampling();
+    int srcSubsamp = get(TJ.PARAM_SUBSAMP);
     for (int i = 0; i < transforms.length; i++) {
       int w = getWidth(), h = getHeight();
+      int dstSubsamp = srcSubsamp;
+
+      if ((transforms[i].options & TJTransform.OPT_GRAY) != 0)
+        dstSubsamp = TJ.SAMP_GRAY;
+      if (transforms[i].op == TJTransform.OP_TRANSPOSE ||
+          transforms[i].op == TJTransform.OP_TRANSVERSE ||
+          transforms[i].op == TJTransform.OP_ROT90 ||
+          transforms[i].op == TJTransform.OP_ROT270) {
+        w = getHeight();  h = getWidth();
+        if (dstSubsamp == TJ.SAMP_422)
+          dstSubsamp = TJ.SAMP_440;
+        else if (dstSubsamp == TJ.SAMP_440)
+          dstSubsamp = TJ.SAMP_422;
+        else if (dstSubsamp == TJ.SAMP_411)
+          dstSubsamp = TJ.SAMP_441;
+        else if (dstSubsamp == TJ.SAMP_441)
+          dstSubsamp = TJ.SAMP_411;
+      }
+
       if ((transforms[i].options & TJTransform.OPT_CROP) != 0) {
         if (transforms[i].width != 0) w = transforms[i].width;
         if (transforms[i].height != 0) h = transforms[i].height;
       }
-      dstBufs[i] = new byte[TJ.bufSize(w, h, get(TJ.PARAM_SUBSAMP))];
+      dstBufs[i] = new byte[TJ.bufSize(w, h, dstSubsamp)];
     }
     TJDecompressor[] tjd = new TJDecompressor[transforms.length];
     transform(dstBufs, transforms);

tjbench.c

@@ -738,6 +738,15 @@ static int decompTest(char *fileName)
       THROW_UNIX("allocating JPEG size array");
     memset(jpegSizes, 0, sizeof(size_t) * ntilesw * ntilesh);
 
+    tsubsamp = (xformOpt & TJXOPT_GRAY) ? TJSAMP_GRAY : subsamp;
+    if (xformOp == TJXOP_TRANSPOSE || xformOp == TJXOP_TRANSVERSE ||
+        xformOp == TJXOP_ROT90 || xformOp == TJXOP_ROT270) {
+      if (tsubsamp == TJSAMP_422) tsubsamp = TJSAMP_440;
+      else if (tsubsamp == TJSAMP_440) tsubsamp = TJSAMP_422;
+      else if (tsubsamp == TJSAMP_411) tsubsamp = TJSAMP_441;
+      else if (tsubsamp == TJSAMP_441) tsubsamp = TJSAMP_411;
+    }
+
     if (noRealloc &&
         (doTile || xformOp != TJXOP_NONE || xformOpt != 0 || customFilter)) {
       for (i = 0; i < ntilesw * ntilesh; i++) {
@@ -745,9 +754,9 @@ static int decompTest(char *fileName)
 
         if (xformOp == TJXOP_TRANSPOSE || xformOp == TJXOP_TRANSVERSE ||
             xformOp == TJXOP_ROT90 || xformOp == TJXOP_ROT270)
-          jpegBufSize = tj3JPEGBufSize(tileh, tilew, subsamp);
+          jpegBufSize = tj3JPEGBufSize(tileh, tilew, tsubsamp);
         else
-          jpegBufSize = tj3JPEGBufSize(tilew, tileh, subsamp);
+          jpegBufSize = tj3JPEGBufSize(tilew, tileh, tsubsamp);
         if (jpegBufSize == 0)
           THROW_TJG();
         if ((jpegBufs[i] = tj3Alloc(jpegBufSize)) == NULL)
@@ -767,7 +776,6 @@ static int decompTest(char *fileName)
       printf("%-5d  %-5d   ", CROPPED_WIDTH(tilew), CROPPED_HEIGHT(tileh));
     }
 
-    tsubsamp = subsamp;
     if (doTile || xformOp != TJXOP_NONE || xformOpt != 0 || customFilter) {
       if ((t = (tjtransform *)malloc(sizeof(tjtransform) * ntilesw *
                                      ntilesh)) == NULL)
@@ -782,26 +790,15 @@ static int decompTest(char *fileName)
           subsamp == TJSAMP_UNKNOWN)
         THROW("transforming",
               "Could not determine subsampling level of JPEG image");
-      if (xformOpt & TJXOPT_GRAY) tsubsamp = TJSAMP_GRAY;
+      if (xformOp == TJXOP_HFLIP || xformOp == TJXOP_TRANSVERSE ||
-      if (xformOp == TJXOP_HFLIP || xformOp == TJXOP_ROT180)
+          xformOp == TJXOP_ROT90 || xformOp == TJXOP_ROT180)
         tw = tw - (tw % tjMCUWidth[tsubsamp]);
-      if (xformOp == TJXOP_VFLIP || xformOp == TJXOP_ROT180)
+      if (xformOp == TJXOP_VFLIP || xformOp == TJXOP_TRANSVERSE ||
+          xformOp == TJXOP_ROT180 || xformOp == TJXOP_ROT270)
         th = th - (th % tjMCUHeight[tsubsamp]);
-      if (xformOp == TJXOP_TRANSVERSE || xformOp == TJXOP_ROT90)
-        tw = tw - (tw % tjMCUHeight[tsubsamp]);
-      if (xformOp == TJXOP_TRANSVERSE || xformOp == TJXOP_ROT270)
-        th = th - (th % tjMCUWidth[tsubsamp]);
       tntilesw = (tw + ttilew - 1) / ttilew;
       tntilesh = (th + ttileh - 1) / ttileh;
 
-      if (xformOp == TJXOP_TRANSPOSE || xformOp == TJXOP_TRANSVERSE ||
-          xformOp == TJXOP_ROT90 || xformOp == TJXOP_ROT270) {
-        if (tsubsamp == TJSAMP_422) tsubsamp = TJSAMP_440;
-        else if (tsubsamp == TJSAMP_440) tsubsamp = TJSAMP_422;
-        else if (tsubsamp == TJSAMP_411) tsubsamp = TJSAMP_441;
-        else if (tsubsamp == TJSAMP_441) tsubsamp = TJSAMP_411;
-      }
-
       for (row = 0, tile = 0; row < tntilesh; row++) {
         for (col = 0; col < tntilesw; col++, tile++) {
           t[tile].r.w = min(ttilew, tw - col * ttilew);

turbojpeg-jni.c

@@ -1218,6 +1218,10 @@ JNIEXPORT jintArray JNICALL Java_org_libjpegturbo_turbojpeg_TJTransformer_transf
     if (t[i].op == TJXOP_TRANSPOSE || t[i].op == TJXOP_TRANSVERSE ||
         t[i].op == TJXOP_ROT90 || t[i].op == TJXOP_ROT270) {
       w = jpegHeight;  h = jpegWidth;
+      if (dstSubsamp == TJSAMP_422) dstSubsamp = TJSAMP_440;
+      else if (dstSubsamp == TJSAMP_440) dstSubsamp = TJSAMP_422;
+      else if (dstSubsamp == TJSAMP_411) dstSubsamp = TJSAMP_441;
+      else if (dstSubsamp == TJSAMP_441) dstSubsamp = TJSAMP_411;
     }
     if (t[i].r.w != 0) w = t[i].r.w;
     if (t[i].r.h != 0) h = t[i].r.h;

turbojpeg.c

@@ -2740,26 +2740,25 @@ DLLEXPORT int tj3Transform(tjhandle handle, const unsigned char *jpegBuf,
   for (i = 0; i < n; i++) {
     int dstSubsamp = (t[i].options & TJXOPT_GRAY) ? TJSAMP_GRAY : srcSubsamp;
 
+    if (t[i].op == TJXOP_TRANSPOSE || t[i].op == TJXOP_TRANSVERSE ||
+        t[i].op == TJXOP_ROT90 || t[i].op == TJXOP_ROT270) {
+      if (dstSubsamp == TJSAMP_422) dstSubsamp = TJSAMP_440;
+      else if (dstSubsamp == TJSAMP_440) dstSubsamp = TJSAMP_422;
+      else if (dstSubsamp == TJSAMP_411) dstSubsamp = TJSAMP_441;
+      else if (dstSubsamp == TJSAMP_441) dstSubsamp = TJSAMP_411;
+    }
+
     if (!jtransform_request_workspace(dinfo, &xinfo[i]))
       THROW("Transform is not perfect");
 
     if (xinfo[i].crop) {
       if (dstSubsamp == TJSAMP_UNKNOWN)
         THROW("Could not determine subsampling level of JPEG image");
-      if (t[i].op == TJXOP_TRANSPOSE || t[i].op == TJXOP_TRANSVERSE ||
+      if ((t[i].r.x % tjMCUWidth[dstSubsamp]) != 0 ||
-          t[i].op == TJXOP_ROT90 || t[i].op == TJXOP_ROT270) {
+          (t[i].r.y % tjMCUHeight[dstSubsamp]) != 0)
-        if ((t[i].r.x % tjMCUHeight[dstSubsamp]) != 0 ||
+        THROWI("To crop this JPEG image, x must be a multiple of %d\n"
-            (t[i].r.y % tjMCUWidth[dstSubsamp]) != 0)
+               "and y must be a multiple of %d.", tjMCUWidth[dstSubsamp],
-          THROWI("To crop this JPEG image, x must be a multiple of %d\n"
+               tjMCUHeight[dstSubsamp]);
-                 "and y must be a multiple of %d.", tjMCUHeight[dstSubsamp],
-                 tjMCUWidth[dstSubsamp]);
-      } else {
-        if ((t[i].r.x % tjMCUWidth[dstSubsamp]) != 0 ||
-            (t[i].r.y % tjMCUHeight[dstSubsamp]) != 0)
-          THROWI("To crop this JPEG image, x must be a multiple of %d\n"
-                 "and y must be a multiple of %d.", tjMCUWidth[dstSubsamp],
-                 tjMCUHeight[dstSubsamp]);
-      }
     }
   }
 
@@ -2772,6 +2771,10 @@ DLLEXPORT int tj3Transform(tjhandle handle, const unsigned char *jpegBuf,
     if (t[i].op == TJXOP_TRANSPOSE || t[i].op == TJXOP_TRANSVERSE ||
         t[i].op == TJXOP_ROT90 || t[i].op == TJXOP_ROT270) {
       dstWidth = dinfo->image_height;  dstHeight = dinfo->image_width;
+      if (dstSubsamp == TJSAMP_422) dstSubsamp = TJSAMP_440;
+      else if (dstSubsamp == TJSAMP_440) dstSubsamp = TJSAMP_422;
+      else if (dstSubsamp == TJSAMP_411) dstSubsamp = TJSAMP_441;
+      else if (dstSubsamp == TJSAMP_441) dstSubsamp = TJSAMP_411;
     }
 
     if (xinfo[i].crop) {

turbojpeg.h

@@ -2065,13 +2065,15 @@ DLLEXPORT int tj3DecodeYUV8(tjhandle handle, const unsigned char *srcBuf,
  * you, or
  * -# pre-allocate the buffer to a "worst case" size determined by calling
  * #tj3JPEGBufSize() with the transformed or cropped width and height and the
- * level of subsampling used in the destination image.  Under normal
+ * level of subsampling used in the destination image (taking into account
- * circumstances, this should ensure that the buffer never has to be
+ * grayscale conversion and transposition of the width and height.)  Under
+ * normal circumstances, this should ensure that the buffer never has to be
  * re-allocated.  (Setting #TJPARAM_NOREALLOC guarantees that it won't be.)
  * Note, however, that there are some rare cases (such as transforming images
  * with a large amount of embedded Exif or ICC profile data) in which the
  * transformed JPEG image will be larger than the worst-case size, and
- * #TJPARAM_NOREALLOC cannot be used in those cases.
+ * #TJPARAM_NOREALLOC cannot be used in those cases unless the embedded data is
+ * discarded using #TJXOPT_COPYNONE.
  * .
  * If you choose option 1, then `dstSizes[i]` should be set to the size of your
  * pre-allocated buffer.  In any case, unless you have set #TJPARAM_NOREALLOC,