Benchmark

The following tools were used to automatically compress a set of images on default settings. The test images are PNG test corpus by Cédric Louvrier and Guetzli photo dataset (converted to JPEG at highest quality) by Jyrki Alakuijala. All images are publicly available for download.

The results were compared with the originals by using a flip test on the calibrated Macbook Pro Retina screen under normal viewing conditions. If an image is visually indistinguishable from its original, it is scored 1 (one), otherwise 0 (zero). The final score is the average of the relative values for each format.

It is highly recommended to view the images full size at 100% zoom. Some images especially on mobile are scaled down by the browser, making it harder to spot the differences.

OptimageImageOptimGuetzliJPEGminiTinyPNGSquashGoogle Photos
Lossless compression
PNG274 KB296 KB304 KB
JPEG16.9 MB16.9 MB
Lossy compression
PNG616 KB578 KB415 KB
JPEG3.69 MB2.08 MB4.21 MB3.74 MB3.58 MB3.63 MB2.68 MB
Human-rated score*
PNG100%76%54%
JPEG100%13%81%58%42%29%13%
Total100%44%81%58%48%29%13%
Butteraugli score**
PNG1.2 ± 0.71.2 ± 1.91.6 ± 2.0
JPEG2.3 ± 0.83.7 ± 1.91.3 ± 0.02.5 ± 0.72.5 ± 1.42.7 ± 1.93.3 ± 1.3

*Score shows how many images are visually indistinguishable from originals under normal viewing conditions. Hover over results to see the actual number of images.
**Butteraugli score is in multiples of just noticeable difference (JND) with a confidence level of 95%.

Notes

  • The human-rated scores correlate with the scores estimated by Butteraugli, which is considered a state-of-the-art image quality metric.
  • The Guetzli score matches the results of the official study, where 75% of human ratings are in favor of Guetzli, which also implies Guetzli performed worse than the baseline JPEG encoder in 25% of cases.
  • Guetzli achieved best quality/size ratio in some cases but failed to provide consistent quality especially at lower quality settings.
  • Due to global error averaging and forced 4:2:0 chroma subsampling, the local error is virtually unlimited in tools like ImageOptim and TinyPNG, regardless of quality setting.
  • ImageOptim forces 4:2:0 chroma subsampling even at higher quality 92-95, which leads to noticeable image quality degradation at comparable file size.
  • TinyPNG and similar tools use stock PNGQuant at more aggressive settings, which greatly increases the likehood of a high local error in images mixing flat colored and photographic content, e.g. screenshots.
  • TinyPNG images could be losslessly compressed by 6.3% more using Optimage.
  • Guetzli is extremely slow requiring approximately 1 minute of CPU and 100 megabytes (MB) of memory per 1 megapixel (MPix).
  • JPEGmini adds a hidden metadata containing version information and image hash to all recompressed JPEG files.
  • ImageOptim will not strip metadata if recompression is suboptimal and increases the file size.

Test samples

Excessive ringing artifacts in JPEGmini, ImageOptim, TinyPNG, Squash and Google Photos
JPEGmini – 84 408 bytes, Optimage – 74 429 bytes
Excessive ringing artifacts in chroma in JPEGmini, ImageOptim, TinyPNG, Squash and Google Photos
JPEGmini – 54 943 bytes, Optimage – 71 102 bytes
Destructive chroma subsampling in ImageOptim, TinyPNG, Squash and Google Photos
ImageOptim – 76 680 bytes, Optimage – 146 324 bytes
Visible block boundary artifacts on soft gradients in Guetzli, JPEGmini and ImageOptim
Guetzli – 155 512 bytes, Optimage – 90 412 bytes
Blurred fine details in Guetzli, ImageOptim, JPEGmini, TinyPNG, Squash and Google Photos
Guetzli – 159 855 bytes, Optimage – 74 820 bytes
Blurred fine details in JPEGmini, ImageOptim, Guetzli, TinyPNG, Squash and Google Photos
JPEGmini – 26 759 bytes, Optimage – 27 970 bytes
Strong gradient dithering artifacts in ImageOptim and TinyPNG
ImageOptim – 9 082 bytes, Optimage – 24 421 bytes
High local error in ImageOptim and TinyPNG
ImageOptim – 8 169 bytes, Optimage – 16 017 bytes
Broken gradient in ImageOptim and TinyPNG
ImageOptim – 5 126 bytes, Optimage – 11 312 bytes

Additional tests

Noticeable color shift when stripping metadata in ImageOptim
ImageOptim – 72 313 bytes, Optimage – 176 935 bytes
Rotated photo when metadata containing Exif Orientation tag stripped in ImageOptim
ImageOptim – 69 856 bytes, Optimage – 69 839 bytes
High local error in JPEGmini, ImageOptim, TinyPNG, Squash and Google Photos
JPEGmini – 4 382 bytes, Optimage – 1 452 bytes
Significant quality degradation caused by overcompression in JPEGmini, ImageOptim, Guetzli, TinyPNG, Squash and Google Photos
JPEGmini – 18 122 bytes, Optimage – 26 216 bytes