Lossless decomp: Range-limit 12-bit samples

12-bit is the only data precision for which the range of the sample data
type exceeds the valid sample range, so it is possible to craft a 12-bit
lossless JPEG image that contains out-of-range 12-bit samples.
Attempting to decompress such an image using color quantization or merged
upsampling (NOTE: libjpeg-turbo cannot generate YCbCr or subsampled
lossless JPEG images, but it can decompress them) caused segfaults or
buffer overruns when those algorithms attempted to use the out-of-range
sample values as array indices.  This commit modifies the lossless
decompressor so that it range-limits the output of the scaler when using
12-bit samples.

Fixes #670
Fixes #672
Fixes #673
Fixes #674
Fixes #675
Fixes #676
Fixes #677
Fixes #678
Fixes #679
Fixes #681
Fixes #683

ChangeLog.md

@@ -8,6 +8,13 @@ subsampling, which allows losslessly transposed or rotated 4:1:1 JPEG images to
 be losslessly cropped, partially decompressed, or decompressed to planar YUV
 images.
 
+2. Fixed various segfaults and buffer overruns that occurred when attempting to
+decompress various specially-crafted malformed 12-bit-per-component lossless
+JPEG images.  These issues were caused by out-of-range sample values that were
+not range-limited before being used as array indices.  The issues were specific
+to 12-bit data precision, since that is the only data precision for which the
+range of the sample data type exceeds the valid sample range.
+
 
 2.1.91 (3.0 beta2)
 ==================

jdlossls.c

@@ -6,7 +6,7 @@
  * Lossless JPEG Modifications:
  * Copyright (C) 1999, Ken Murchison.
  * libjpeg-turbo Modifications:
- * Copyright (C) 2022, D. R. Commander.
+ * Copyright (C) 2022-2023, D. R. Commander.
  * For conditions of distribution and use, see the accompanying README.ijg
  * file.
  *
@@ -217,7 +217,15 @@ simple_upscale(j_decompress_ptr cinfo,
                JDIFFROW diff_buf, _JSAMPROW output_buf, JDIMENSION width)
 {
   do {
+#if BITS_IN_JSAMPLE == 12
+    /* 12-bit is the only data precision for which the range of the sample data
+     * type exceeds the valid sample range.  Thus, we need to range-limit the
+     * samples, because other algorithms may try to use them as array indices.
+     */
+    *output_buf++ = (_JSAMPLE)((*diff_buf++ << cinfo->Al) & 0xFFF);
+#else
     *output_buf++ = (_JSAMPLE)(*diff_buf++ << cinfo->Al);
+#endif
   } while (--width);
 }
 
@@ -226,7 +234,11 @@ noscale(j_decompress_ptr cinfo,
         JDIFFROW diff_buf, _JSAMPROW output_buf, JDIMENSION width)
 {
   do {
+#if BITS_IN_JSAMPLE == 12
+    *output_buf++ = (_JSAMPLE)((*diff_buf++) & 0xFFF);
+#else
     *output_buf++ = (_JSAMPLE)(*diff_buf++);
+#endif
   } while (--width);
 }