TJPARAM_MAXPIXELS was previously hidden and used only for fuzz testing,
but it is potentially useful for calling applications as well,
particularly if they want to guard against excessive memory consumption
by the tj3LoadImage*() functions. The parameter has also been extended
to decompression and lossless transformation functions/methods, mainly
as a convenience. (It was already possible for calling applications to
impose their own JPEG image size limits by reading the JPEG header prior
to decompressing or transforming the image.)
Color quantization is a legacy feature that serves little or no purpose
with lossless JPEG images. 9f756bc67a
eliminated interaction issues between the lossless decompressor and the
color quantizers related to out-of-range 12-bit samples, but referring
to #701, other interaction issues apparently still exist. Such issues
are likely, given the fact that the color quantizers were not designed
with lossless decompression in mind.
This commit reverts 9f756bc67a, since the
issues it fixed are no longer relevant because of this commit and
2192560d74.
Fixed#672Fixes#673Fixes#674Fixes#676Fixes#677Fixes#678Fixes#679Fixes#681Fixes#683Fixes#701
Lossless: Accommodate LJT colorspace/SIMD exts
In libjpeg-turbo, grayscale_convert() and null_convert() aren't the only
lossless color conversion algorithms. We can also losslessly convert
RGB to and from any of the extended RGB colorspaces, and some platforms
have SIMD-accelerated null color conversion.
This commit also disallows RGB565 output in lossless mode, and it moves
the IsExtRGB() macro from cdjpeg.h to jpegint.h and repurposes it to
make jinit_color_converter() and jinit_color_deconverter() more
readable.
- Rename jpeg_simple_lossless() to jpeg_enable_lossless() and modify the
function so that it stores the lossless parameters directly in the Ss
and Al fields of jpeg_compress_struct rather than using a scan script.
- Move the cjpeg -lossless switch into "Switches for advanced users".
- Document the libjpeg API and run-time features that are unavailable in
lossless mode, and ensure that all parameters, functions, and switches
related to unavailable features are ignored or generate errors in
lossless mode.
- Defer any action that depends on whether lossless mode is enabled
until jpeg_start_compress()/jpeg_start_decompress() is called.
- Document the purpose of the point transform value.
- "Codec" stands for coder/decoder, so it is a bit awkward to say
"lossless compression codec" and "lossless decompression codec".
Use "lossless compressor" and "lossless decompressor" instead.
- Restore backward API/ABI compatibility with libjpeg v6b:
* Move the new 'lossless' field from the exposed jpeg_compress_struct
and jpeg_decompress_struct structures into the opaque
jpeg_comp_master and jpeg_decomp_master structures, and allocate the
master structures in the body of jpeg_create_compress() and
jpeg_create_decompress().
* Remove the new 'process' field from jpeg_compress_struct and
jpeg_decompress_struct and replace it with the old
'progressive_mode' field and the new 'lossless' field.
* Remove the new 'data_unit' field from jpeg_compress_struct and
jpeg_decompress_struct and replace it with a locally-computed
data unit variable.
* Restore the names of macros and fields that refer to DCT blocks, and
document that they have a different meaning in lossless mode. (Most
of them aren't very meaningful in lossless mode anyhow.)
* Remove the new alloc_darray() method from jpeg_memory_mgr and
replace it with an internal macro that wraps the alloc_sarray()
method.
* Move the JDIFF* data types from jpeglib.h and jmorecfg.h into
jpegint.h.
* Remove the new 'codec' field from jpeg_compress_struct and
jpeg_decompress_struct and instead reuse the existing internal
coefficient control, forward/inverse DCT, and entropy
encoding/decoding structures for lossless compression/decompression.
* Repurpose existing error codes rather than introducing new ones.
(The new JERR_BAD_RESTART and JWRN_MUST_DOWNSCALE codes remain,
although JWRN_MUST_DOWNSCALE will probably be removed in
libjpeg-turbo, since we have a different way of handling multiple
data precisions.)
- Automatically enable lossless mode when a scan script with parameters
that are only valid for lossless mode is detected, and document the
use of scan scripts to generate lossless JPEG images.
- Move the sequential and shared Huffman routines back into jchuff.c and
jdhuff.c, and document that those routines are shared with jclhuff.c
and jdlhuff.c as well as with jcphuff.c and jdphuff.c.
- Move MAX_DIFF_BITS from jchuff.h into jclhuff.c, the only place where
it is used.
- Move the predictor and scaler code into jclossls.c and jdlossls.c.
- Streamline register usage in the [un]differencers (inspired by similar
optimizations in the color [de]converters.)
- Restructure the logic in a few places to reduce duplicated code.
- Ensure that all lossless-specific code is guarded by
C_LOSSLESS_SUPPORTED or D_LOSSLESS_SUPPORTED and that the library can
be built successfully if either or both of those macros is undefined.
- Remove all short forms of external names introduced by the lossless
JPEG patch. (These will not be needed by libjpeg-turbo, so there is
no use cleaning them up.)
- Various wordsmithing, formatting, and punctuation tweaks
- Eliminate various compiler warnings.
In libjpeg-turbo 2.1.x and prior, the WITH_12BIT CMake variable was used
to enable 12-bit JPEG support at compile time, because the libjpeg API
library could not handle multiple JPEG data precisions at run time. The
initial approach to handling multiple JPEG data precisions at run time
(7fec5074f9) created a whole new API,
library, and applications for 12-bit data precision, so it made sense to
repurpose WITH_12BIT to allow 12-bit data precision to be disabled.
e8b40f3c2b made it so that the libjpeg API
library can handle multiple JPEG data precisions at run time via a
handful of straightforward API extensions. Referring to
6c2bc901e2, it hasn't been possible to
build libjpeg-turbo with both forward and backward libjpeg API/ABI
compatibility since libjpeg-turbo 1.4.x. Thus, whereas we retain full
backward API/ABI compatibility with libjpeg v6b-v8, forward libjpeg
API/ABI compatibility ceased being realistic years ago, so it no longer
makes sense to provide compile-time options that give a false sense of
forward API/ABI compatibility by allowing some (but not all) of our
libjpeg API extensions to be disabled. Such options are difficult to
maintain and clutter the code with #ifdefs.
The Gordian knot that 7fec5074f9 attempted
to unravel was caused by the fact that there are several
data-precision-dependent (JSAMPLE-dependent) fields and methods in the
exposed libjpeg API structures, and if you change the exposed libjpeg
API structures, then you have to change the whole API. If you change
the whole API, then you have to provide a whole new library to support
the new API, and that makes it difficult to support multiple data
precisions in the same application. (It is not impossible, as example.c
demonstrated, but using data-precision-dependent libjpeg API structures
would have made the cjpeg, djpeg, and jpegtran source code hard to read,
so it made more sense to build, install, and package 12-bit-specific
versions of those applications.)
Unfortunately, the result of that initial integration effort was an
unreadable and unmaintainable mess, which is a problem for a library
that is an ISO/ITU-T reference implementation. Also, as I dug into the
problem of lossless JPEG support, I realized that 16-bit lossless JPEG
images are a thing, and supporting yet another version of the libjpeg
API just for those images is untenable.
In fact, however, the touch points for JSAMPLE in the exposed libjpeg
API structures are minimal:
- The colormap and sample_range_limit fields in jpeg_decompress_struct
- The alloc_sarray() and access_virt_sarray() methods in
jpeg_memory_mgr
- jpeg_write_scanlines() and jpeg_write_raw_data()
- jpeg_read_scanlines() and jpeg_read_raw_data()
- jpeg_skip_scanlines() and jpeg_crop_scanline()
(This is subtle, but both of those functions use JSAMPLE-dependent
opaque structures behind the scenes.)
It is much more readable and maintainable to provide 12-bit-specific
versions of those six top-level API functions and to document that the
aforementioned methods and fields must be type-cast when using 12-bit
samples. Since that eliminates the need to provide a 12-bit-specific
version of the exposed libjpeg API structures, we can:
- Compile only the precision-dependent libjpeg modules (the
coefficient buffer controllers, the colorspace converters, the
DCT/IDCT managers, the main buffer controllers, the preprocessing
and postprocessing controller, the downsampler and upsamplers, the
quantizers, the integer DCT methods, and the IDCT methods) for
multiple data precisions.
- Introduce 12-bit-specific methods into the various internal
structures defined in jpegint.h.
- Create precision-independent data type, macro, method, field, and
function names that are prefixed by an underscore, and use an
internal header to convert those into precision-dependent data
type, macro, method, field, and function names, based on the value
of BITS_IN_JSAMPLE, when compiling the precision-dependent libjpeg
modules.
- Expose precision-dependent jinit*() functions for each of the
precision-dependent libjpeg modules.
- Abstract the precision-dependent libjpeg modules by calling the
appropriate precision-dependent jinit*() function, based on the
value of cinfo->data_precision, from top-level libjpeg API
functions.
The macros in jerror.h refer to j_common_ptr, so it is unfortunately
necessary to introduce a 12-bit-specific version of that header file
(j12error.h) with 12-bit specific ERREXIT*(), WARNMS*(), and
TRACEMS*() macros. (The message table is still shared between 8-bit and
12-bit implementations.)
Fixes#607
After the completion of the start_input() method, it's too late to check
the image size, because the image readers may have already tried to
allocate memory for the image. If the width and height are excessively
large, then attempting to allocate memory for the image could slow
performance or lead to out-of-memory errors prior to the fuzz target
checking the image size.
NOTE: Specifically, the aforementioned OOM errors and slow units were
observed with the compression fuzz targets when using MSan.
- Restore GIF read/compressed GIF write support from jpeg-6a and
jpeg-9d.
- Integrate jpegtran -wipe and -drop options from jpeg-9a and jpeg-9d.
- Integrate jpegtran -crop extension (for expanding the image size) from
jpeg-9a and jpeg-9d.
- Integrate other minor code tweaks from jpeg-9*
libjpeg-turbo never included that code, because it requires an external
library (the Utah Raster Toolkit.) The RLE image format was supplanted
by GIF in the late 1980s, so it is rarely seen these days. (It had a
lousy Weissman score, anyhow.)
- Enable progress reporting at run time using a new -report argument
(cjpeg now supports that argument as well)
- Limit the allowable number of scans using a new -maxscans argument
- Treat warnings as fatal using a new -strict argument
This mainly demonstrates how to work around the two issues with the
JPEG standard described here:
https://libjpeg-turbo.org/pmwiki/uploads/About/TwoIssueswiththeJPEGStandard.pdf
since those and similar issues continue to be erroneously reported as
libjpeg-turbo bugs.
With rare exceptions ...
- Always separate line continuation characters by one space from
preceding code.
- Always use two-space indentation. Never use tabs.
- Always use K&R-style conditional blocks.
- Always surround operators with spaces, except in raw assembly code.
- Always put a space after, but not before, a comma.
- Never put a space between type casts and variables/function calls.
- Never put a space between the function name and the argument list in
function declarations and prototypes.
- Always surround braces ('{' and '}') with spaces.
- Always surround statements (if, for, else, catch, while, do, switch)
with spaces.
- Always attach pointer symbols ('*' and '**') to the variable or
function name.
- Always precede pointer symbols ('*' and '**') by a space in type
casts.
- Use the MIN() macro from jpegint.h within the libjpeg and TurboJPEG
API libraries (using min() from tjutil.h is still necessary for
TJBench.)
- Where it makes sense (particularly in the TurboJPEG code), put a blank
line after variable declaration blocks.
- Always separate statements in one-liners by two spaces.
The purpose of this was to ease maintenance on my part and also to make
it easier for contributors to figure out how to format patch
submissions. This was admittedly confusing (even to me sometimes) when
we had 3 or 4 different style conventions in the same source tree. The
new convention is more consistent with the formatting of other OSS code
bases.
This commit corrects deviations from the chosen formatting style in the
libjpeg API code and reformats the TurboJPEG API code such that it
conforms to the same standard.
NOTES:
- Although it is no longer necessary for the function name in function
declarations to begin in Column 1 (this was historically necessary
because of the ansi2knr utility, which allowed libjpeg to be built
with non-ANSI compilers), we retain that formatting for the libjpeg
code because it improves readability when using libjpeg's function
attribute macros (GLOBAL(), etc.)
- This reformatting project was accomplished with the help of AStyle and
Uncrustify, although neither was completely up to the task, and thus
a great deal of manual tweaking was required. Note to developers of
code formatting utilities: the libjpeg-turbo code base is an
excellent test bed, because AFAICT, it breaks every single one of the
utilities that are currently available.
- The legacy (MMX, SSE, 3DNow!) assembly code for i386 has been
formatted to match the SSE2 code (refer to
ff5685d5344273df321eb63a005eaae19d2496e3.) I hadn't intended to
bother with this, but the Loongson MMI implementation demonstrated
that there is still academic value to the MMX implementation, as an
algorithmic model for other 64-bit vector implementations. Thus, it
is desirable to improve its readability in the same manner as that of
the SSE2 implementation.
Loading RGB image files into a grayscale buffer isn't a particularly
useful feature, given that libjpeg-turbo can perform this conversion
much more optimally (with SIMD acceleration on some platforms) during
the compression process. Also, the RGB2GRAY() macro was not producing
deterministic cross-platform results because of variations in the
round-off behavior of various floating point implementations, so
`tjunittest -bmp` was failing in i386 builds.
The main justification for this is to provide new libjpeg-turbo users
with a quick & easy way of developing a complete JPEG
compression/decompression program without requiring them to build
libjpeg-turbo from source (which was necessary in order to use the
project-private bmp API) or to use external libraries. These new
functions build upon significant enhancements to rdbmp.c, wrbmp.c,
rdppm.c, and wrppm.c which allow those engines to convert directly
between the native pixel format of the file and a pixel format
("colorspace" in libjpeg parlance) specified by the calling program.
rdbmp.c and wrbmp.c have also been modified such that the calling
program can choose to read or write image rows in the native (bottom-up)
order of the file format, thus eliminating the need to use an inversion
array. tjLoadImage() and tjSaveImage() leverage these new underlying
features in order to significantly improve upon the performance of the
old bmp API.
Because these new functions cannot work without the libjpeg-turbo
colorspace extensions, the libjpeg-compatible code in turbojpeg.c has
been removed. That code was only there to serve as an example of how
to use the TurboJPEG API on top of libjpeg, but more specific, buildable
examples now exist in the https://github.com/libjpeg-turbo/ijg
repository.
- Referring to 073b0e88a1 and #185, the
reason why BMP and RLE didn't (and won't) work with partial image
decompression is that the output engines for both formats maintain a
whole-image buffer, which is used to reverse the order of scanlines.
However, it was straightforward to add -crop support for GIF and
Targa, which is useful for testing partial image decompression along
with color quantization.
- Such testing reproduced a bug reported by Mozilla (refer to PR #182)
whereby jpeg_skip_scanlines() would segfault if color quantization was
enabled. To fix this issue, read_and_discard_scanlines() now sets up
a dummy quantize function in the same manner that it sets up a dummy
color conversion function.
Closes#182
The convention used by libjpeg:
type * variable;
is not very common anymore, because it looks too much like
multiplication. Some (particularly C++ programmers) prefer to tuck the
pointer symbol against the type:
type* variable;
to emphasize that a pointer to a type is effectively a new type.
However, this can also be confusing, since defining multiple variables
on the same line would not work properly:
type* variable1, variable2; /* Only variable1 is actually a
pointer. */
This commit reformats the entirety of the libjpeg-turbo code base so
that it uses the same code formatting convention for pointers that the
TurboJPEG API code uses:
type *variable1, *variable2;
This seems to be the most common convention among C programmers, and
it is the convention used by other codec libraries, such as libpng and
libtiff.
The IJG README file has been renamed to README.ijg, in order to avoid
confusion (many people were assuming that that was our project's README
file and weren't reading README-turbo.txt) and to lay the groundwork for
markdown versions of the libjpeg-turbo README and build instructions.
