Fix buffer overrun in 12-bit prog Huffman encoder

Regression introduced by 16bd984557 and
5b177b3cab

The pre-computed absolute values used in encode_mcu_AC_first() and
encode_mcu_AC_refine() were stored in a JCOEF (signed short) array.
When attempting to losslessly transform a specially-crafted malformed
12-bit JPEG image with a coefficient value of -32768 into a progressive
12-bit JPEG image, the progressive Huffman encoder attempted to store
the absolute value of -32768 in the JCOEF array, thus overflowing the
16-bit signed data type.  Therefore, at this point in the code:
8c5e78ce29/jcphuff.c (L889)
the absolute value was read as -32768, which caused the test at
8c5e78ce29/jcphuff.c (L896)
to fail, falling through to
8c5e78ce29/jcphuff.c (L908)
with an overly large value of r (46) that, when shifted left four
places, incremented, and passed to emit_symbol(), exceeded the maximum
index (255) for the derived code tables.  Fortunately, the buffer
overrun was fully contained within phuff_entropy_encoder, so the issue
did not generate a segfault or other user-visible errant behavior, but
it did cause a UBSan failure that was detected by OSS-Fuzz.

This commit introduces an unsigned JCOEF (UJCOEF) data type and uses it
to store the absolute values of DCT coefficients computed by the
AC_first_prepare() and AC_refine_prepare() methods.

Note that the changes to the Arm Neon progressive Huffman encoder
extensions cause signed 16-bit instructions to be replaced with
equivalent unsigned 16-bit instructions, so the changes should be
performance-neutral.

Based on:
bbf61c0382

Closes #628
This commit is contained in:
DRC
2022-11-15 17:01:17 -06:00
parent aa3dd0bd29
commit 78a36f6dc3
14 changed files with 166 additions and 145 deletions

View File

@@ -3,8 +3,8 @@
*
* This file was part of the Independent JPEG Group's software:
* Copyright (C) 1991-1997, Thomas G. Lane.
* It was modified by The libjpeg-turbo Project to include only code relevant
* to libjpeg-turbo.
* libjpeg-turbo Modifications:
* Copyright (C) 2022, D. R. Commander.
* For conditions of distribution and use, see the accompanying README.ijg
* file.
*
@@ -25,6 +25,14 @@
#define MAX_COEF_BITS 14
#endif
/* The progressive Huffman encoder uses an unsigned 16-bit data type to store
* absolute values of coefficients, because it is possible to inject a
* coefficient value of -32768 into the encoder by attempting to transform a
* malformed 12-bit JPEG image, and the absolute value of -32768 would overflow
* a signed 16-bit integer.
*/
typedef unsigned short UJCOEF;
/* Derived data constructed for each Huffman table */
typedef struct {