Catch libjpeg errors in tjDecompressToYUV2()

Even though tjDecompressToYUV2() is mostly just a wrapper for
tjDecompressToYUVPlanes(), tjDecompressToYUV2() still calls
jpeg_read_header(), so it needs to properly set up the libjpeg error
handler prior to making this call.  Otherwise, under very esoteric (and
arguably incorrect) use cases, a program could call tjDecompressToYUV2()
without first checking the JPEG header using tjDecompressHeader3(), and
if the header was corrupt, then the libjpeg API would invoke
my_error_exit().  my_error_exit() would in turn call longjmp() on the
previous value of myerr->setjmp_buffer, which was probably set in a
previous TurboJPEG function, such as tjInitDecompress().  Thus, when a
libjpeg error was triggered within the body of tjDecompressToYUV2(), the
PC would jump to the error handler of the previous TurboJPEG function,
and this usually caused stack corruption in the calling program (because
the signature and return type of the previous TurboJPEG function
probably wasn't the same as that of tjDecompressToYUV2().)

ChangeLog.txt

@@ -38,6 +38,14 @@ included a similar fix for ASCII PPM/PGM files.  Note that these issues were
 not security bugs, since they were confined to the cjpeg program and did not
 affect any of the libjpeg-turbo libraries.
 
+[7] Fixed an issue whereby attempting to decompress a JPEG file with a corrupt
+header using the tjDecompressToYUV2() function would cause the function to
+abort without returning an error and, under certain circumstances, corrupt the
+stack.  This only occurred if tjDecompressToYUV2() was called prior to calling
+calling tjDecompressHeader3(), or if the return value from
+tjDecompressHeader3() was ignored (both cases represent incorrect usage of the
+TurboJPEG API.)
+
 
 1.4.2
 =====

turbojpeg.c

@@ -1886,6 +1886,12 @@ DLLEXPORT int DLLCALL tjDecompressToYUV2(tjhandle handle,
 		|| !isPow2(pad) || height<0)
 		_throw("tjDecompressToYUV2(): Invalid argument");
 
+	if(setjmp(this->jerr.setjmp_buffer))
+	{
+		/* If we get here, the JPEG code has signaled an error. */
+		return -1;
+	}
+
 	jpeg_mem_src_tj(dinfo, jpegBuf, jpegSize);
 	jpeg_read_header(dinfo, TRUE);
 	jpegSubsamp=getSubsamp(dinfo);