ryan/mozjpeg
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tj3Compress*(): Free virt arrays if mem limit hit

Browse Source 
This is very subtle, but if a user specifies a libjpeg virtual array
memory limit via the JPEGMEM environment variable and one of the
tj3Compress*() functions hits that limit, the libjpeg error handler
will be invoked in jpeg_start_compress() (more specifically in
realize_virt_arrays() in jinit_compress_master()) before the libjpeg
global compression state can be incremented.  Thus,
jpeg_abort_compress() will not be called before the tj3Compress*()
function exits, the unrealized virtual arrays will not be freed, and if
the TurboJPEG compression instance is reused, those unrealized virtual
arrays will count against the specified memory limit.  This could cause
subsequent compression operations that require smaller virtual arrays
(or even no virtual arrays at all) to fail when they would otherwise
succeed.  In reality, the vast majority of calling programs would abort
and free the TurboJPEG compression instance if one of the tj3Compress*()
functions failed, but TJBench is a rare exception.  This issue does not
bear documenting because of its subtlety and rarity and because JPEGMEM
is not a documented feature of the TurboJPEG API.

Note that the issue does not exist in the tj3Encode*() and tj3Decode*()
functions, because realize_virt_arrays() is never called in the body of
those functions.  The issue also does not exist in the tj3Decompress*()
and tj3Transform() functions, because those functions ensure that the
JPEG header is read (and thus the libjpeg global decompression state is
incremented) prior to calling a function that calls
realize_virt_arrays() (i.e. jpeg_start_decompress() or
jpeg_read_coefficients().)  If realize_virt_arrays() failed in the body
of jpeg_write_coefficients(), then tj3Transform() would abort without
calling jpeg_abort_compress().  However, since jpeg_start_compress() is
never called in the body of tj3Transform(), no virtual arrays are ever
requested from the compression object, so failing to call
jpeg_abort_compress() would be innocuous.
This commit is contained in:
DRC
2023-11-11 15:14:07 -05:00
parent 220bd76188
commit bf248a5093
2 changed files with 6 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
turbojpeg-mp.c
View File

@@ -125,10 +125,10 @@ DLLEXPORT int GET_NAME(tj3Compress, BITS_IN_JSAMPLE)
jpeg_finish_compress(cinfo);
bailout:
if (cinfo->global_state > CSTATE_START) {
if (alloc) (*cinfo->dest->term_destination) (cinfo);
if (cinfo->global_state > CSTATE_START && alloc)
(*cinfo->dest->term_destination) (cinfo);
if (cinfo->global_state > CSTATE_START || retval == -1)
jpeg_abort_compress(cinfo);
}
free(row_pointer);
if (this->jerr.warning) retval = -1;
return retval;

6
turbojpeg.c
View File

@@ -1589,10 +1589,10 @@ DLLEXPORT int tj3CompressFromYUVPlanes8(tjhandle handle,
jpeg_finish_compress(cinfo);
bailout:
if (cinfo->global_state > CSTATE_START) {
if (alloc) (*cinfo->dest->term_destination) (cinfo);
if (cinfo->global_state > CSTATE_START && alloc)
(*cinfo->dest->term_destination) (cinfo);
if (cinfo->global_state > CSTATE_START || retval == -1)
jpeg_abort_compress(cinfo);
}
for (i = 0; i < MAX_COMPONENTS; i++) {
free(tmpbuf[i]);
free(inbuf[i]);