Disallow merged upsampling with lossless decomp

Colorspace conversion is explicitly not supported with lossless JPEG
images.  Merged upsampling implies YCbCr-to-RGB colorspace conversion,
so allowing it with lossless decompression was an oversight.
9f756bc67a eliminated interaction issues
between the lossless decompressor and the merged upsampler related to
out-of-range 12-bit samples, but referring to #690, other interaction
issues apparently still exist.  Such issues are likely, given the fact
that the merged upsampler was never designed with lossless decompression
in mind.

This commit also extends the decompress fuzzer so that it catches the
issue reported in #690.

Fixes #690
Redundantly fixes #670
Redundantly fixes #675
This commit is contained in:
DRC
2023-05-31 13:02:42 -04:00
parent 4e7ff7b922
commit 2192560d74
2 changed files with 6 additions and 2 deletions

View File

@@ -76,9 +76,10 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
int64_t sum = 0;
/* Test non-default decompression options on the first iteration. */
tj3Set(handle, TJPARAM_BOTTOMUP, pfi == 0);
tj3Set(handle, TJPARAM_FASTUPSAMPLE, pfi == 0);
if (!tj3Get(handle, TJPARAM_LOSSLESS)) {
tj3Set(handle, TJPARAM_BOTTOMUP, pfi == 0);
tj3Set(handle, TJPARAM_FASTUPSAMPLE, pfi == 0);
tj3Set(handle, TJPARAM_FASTDCT, pfi == 0);
/* Test IDCT scaling on the second iteration. */

View File

@@ -35,6 +35,9 @@ LOCAL(boolean)
use_merged_upsample(j_decompress_ptr cinfo)
{
#ifdef UPSAMPLE_MERGING_SUPPORTED
/* Colorspace conversion is not supported with lossless JPEG images */
if (cinfo->master->lossless)
return FALSE;
/* Merging is the equivalent of plain box-filter upsampling */
if (cinfo->do_fancy_upsampling || cinfo->CCIR601_sampling)
return FALSE;