Benchmark
The following tools were used to automatically compress a set of images on default settings. The test images are PNG test corpus by Cédric Louvrier and Guetzli photo dataset (converted to JPEG at highest quality) by Jyrki Alakuijala. All images are publicly available for download.
The results were compared with the originals by using a flip test on the calibrated Macbook Pro Retina screen under normal viewing conditions. If an image is visually indistinguishable from its original, it is scored 1 (one), otherwise 0 (zero). The final score is the average of the relative values for each format.
It is highly recommended to view the images full size at 100% zoom. Some images especially on mobile are scaled down by the browser, making it harder to spot the differences.
| Optimage | ImageOptim | Guetzli | JPEGmini | TinyPNG | Squash | Google Photos | |
|---|---|---|---|---|---|---|---|
| Lossless compression | |||||||
| PNG | 274 KB | 296 KB | – | – | – | 304 KB | – |
| JPEG | 16.9 MB | 16.9 MB | – | – | – | – | – |
| Lossy compression | |||||||
| PNG | 616 KB | 578 KB | – | – | 415 KB | – | – |
| JPEG | 3.68 MB | 2.08 MB | 4.21 MB | 3.74 MB | 3.58 MB | 3.63 MB | 2.68 MB |
| Human-rated score* | |||||||
| PNG | 100% | 76% | – | – | 54% | – | – |
| JPEG | 100% | 13% | 81% | 58% | 42% | 29% | 13% |
| Total | 100% | 44% | 81% | 58% | 48% | 29% | 13% |
| Butteraugli score** | |||||||
| PNG | 1.2 ± 0.7 | 1.2 ± 1.9 | – | – | 1.6 ± 2.0 | – | – |
| JPEG | 2.3 ± 0.8 | 3.7 ± 1.9 | 1.3 ± 0.0 | 2.5 ± 0.7 | 2.5 ± 1.4 | 2.7 ± 1.9 | 3.3 ± 1.3 |
*Score shows how many images are visually indistinguishable from originals under normal viewing conditions. Hover over results to see the actual number of images.
**Butteraugli score is in multiples of just noticeable difference (JND) with a confidence level of 95%.
Notes
- The human-rated scores correlate with the scores estimated by Butteraugli, which is considered a state-of-the-art image quality metric.
- The Guetzli score matches the results of the official study, where 75% of human ratings are in favor of Guetzli, which also implies Guetzli performed worse than the baseline JPEG encoder in 25% of cases.
- Guetzli achieved best quality/size ratio in some cases but failed to provide consistent quality especially at lower quality settings.
- Due to global error averaging and forced 4:2:0 chroma subsampling, the local error is virtually unlimited in tools like ImageOptim and TinyPNG, regardless of quality setting.
- ImageOptim forces 4:2:0 chroma subsampling even at higher quality 92-95, which leads to noticeable image quality degradation at comparable file size.
- TinyPNG and similar tools use stock PNGQuant at more aggressive settings, which greatly increases the likehood of a high local error in images mixing flat colored and photographic content, e.g. screenshots.
- TinyPNG images could be losslessly compressed by 6.3% more using Optimage.
- Guetzli is extremely slow requiring approximately 1 minute of CPU and 100 megabytes (MB) of memory per 1 megapixel (MPix).
- JPEGmini adds a hidden metadata containing version information and image hash to all recompressed JPEG files.
- ImageOptim will not strip metadata if recompression is suboptimal and increases the file size.
Test samples
JPEGmini – 84 408 bytes, Optimage – 74 301 bytes
JPEGmini – 54 943 bytes, Optimage – 71 090 bytes
ImageOptim – 76 680 bytes, Optimage – 146 051 bytes
Guetzli – 122 114 bytes, Optimage – 90 368 bytes
Guetzli – 121 662 bytes, Optimage – 74 772 bytes
JPEGmini – 26 759 bytes, Optimage – 27 950 bytes
ImageOptim – 9 082 bytes, Optimage – 24 421 bytes
ImageOptim – 8 169 bytes, Optimage – 16 017 bytes
ImageOptim – 5 126 bytes, Optimage – 11 312 bytes
Additional tests
ImageOptim – 72 313 bytes, Optimage – 176 935 bytes
ImageOptim – 69 856 bytes, Optimage – 69 839 bytes
JPEGmini – 4 382 bytes, Optimage – 1 452 bytes
JPEGmini – 18 122 bytes, Optimage – 26 216 bytes
