Lepton image compression: saving 22% losslessly from images at 15MB/s

https://dropbox.tech/infrastructure/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs

This open-source project is no longer maintained or supported by Dropbox. Please refer to Lepton’s GitHub page for more information.

~ ~ ~

We are pleased to announce the open source release of Lepton, our new streaming image compression format, under the Apache license.

Lepton achieves a 22% savings reduction for existing JPEG images, by predicting coefficients in JPEG blocks and feeding those predictions as context into an arithmetic coder. Lepton preserves the original file bit-for-bit perfectly. It compresses JPEG files at a rate of 5 megabytes per second and decodes them back to the original bits at 15 megabytes per second, securely, deterministically, and in under 24 megabytes of memory.

We have used Lepton to encode 16 billion images saved to Dropbox, and are rapidly recoding our older images. Lepton has already saved Dropbox multiple petabytes of space.

Community participation and improvement to this new compression algorithm is welcome and encouraged!

Lepton at scale

At Dropbox, the security and durability of your data are our highest priorities. As an added security layer, Lepton runs within seccomp to disable all system calls except read and write of already-open file descriptors. Lepton has gone through a rigorous automated testing process demonstrating determinism on over 4 billion photos and counting. This means that once we verify an image decodes back to its original bits the first time, we can always get back to the original file in future decodes.

All of our compression algorithms, including Lepton, decode every compressed file at least once and compare the result to the input, bit-for-bit, before persisting that file. Compressed files are placed into kernel-protected, read-only, memory before the bit-for-bit comparison to guarantee they are immutable during the full verification process.

https://github.com/dropbox/lepton

https://github.com/packjpg/packJPG

PackJPG v2.5k (01/22/2016)
~~~~~~~~~~~~~~~~~~~~~~~~~~

packJPG is a compression program specially designed for further
compression of JPEG images without causing any further loss. Typically
it reduces the file size of a JPEG file by 20%.