CinemaDNG compression modes in slimRAW

slimRAW implements various CinemaDNG compression modes. This article describes them in some detail and will hopefully help you get the most out of slimRAW.

The purpose of lossless compression is creating a smaller representation of the original data with no loss of information. DNG/CinemaDNG lossless compression is truly lossless as opposed to "visually lossless" (which is a euphemism for "lossy" outside the strict scientific community). In the raw processor the original data is fully reconstructed from the compressed data: the decompressed image is exactly the same as the original uncompressed image.

Lossless compression is variable bit rate by nature. Typical compression achieved is around 2:1 (or down to around 50% of the original size). This can go up or down depending on image complexity and tonal specifics. For example, Sony FS-series CinemaDNG will often get compressed to 2.5:1 or better, and Blackmagic lossless CinemaDNG will be around 1.6:1. Enabling the option "Maximum compression" (in File->Other options) can often squeeze a little bit more space savings in exchange for slower execution. But if your storage throughput (and not the CPU) is the bottleneck, you may leave this option enabled with no processing speed hit.

Lossless compression is included in the DNG specification and is widely supported by video and photo raw processing applications.

A little bit of background first. Sensors capture light in a linear fashion. Double the amount of light leads to double the photons captured. When these light levels are digitized, the highest stop of the captured dynamic range occupies half of the digital coding space, the next stop below the highest takes half of what is left, etc. For example, in 12-bit linear raw the highest stop uses 2048 values, the stop below uses 1024 values, the next stop down uses 512 values, and so on. This distribution of coding values is quite ineffective, because eye response to light intensity is non-linear (approximately logarithmic): we don't really benefit from the excess of tonal info in the brightest captured stops.

This is where the "Lossless 10-bit log" mode is grounded. It utilizes the same lossless entropy coding as the Lossless mode, but applies a non-linear (logarithmic) tone mapping curve before entropy coding. This is an entropy reducing step which allows for further space savings compared to Lossless. The curve is designed so that only the brightest few stops of the captured dynamic range are tonally compressed and use less coding values than the linear distribution; darks are left untouched. The log conversion is transparent for the user: the log-to-linear inverse transform is described in the DNG metadata and the curve is reversed by the raw processing software on import.

The curve maps the original linear signal into a 10-bit coding space. The same idea was used in the 10-bit Cineon log format for negative film scans (film density itself is a logarithmic function of captured light intensity) and, more recently, in 10-bit log raw from the Canon C500 cinema camera. Logarithmic tonal compression is also used in ARRIRAW (although Arri prefer to call their raw "uncompressed") and other raw formats. Note that the tonal compression step is technically lossy (albeit only in the bright stops), so it would be more precise to call the mode "Lossless" 10-bit log. Unlike lossy compression modes though, no frequency domain quantization takes place.

Lossless 10-bit log is intended for 12-bit, 14-bit and 16-bit linear raw sources. Examples include Canon Magic Lantern sourced CinemaDNG, DJI X5R, Digital Bolex D16, Ikonoskop, Sony FS-series, etc. It will fall back to "Lossless" for bit depths lower than 12 bits (for example, DJI X5R high frame rate 10-bit raw), as well as for non-linear raw (for example, CinemaDNG from Blackmagic cameras or slimRAW'ed lossy raw).

Savings relative to straight Lossless are up to 30% or more; but typically around 20-25% for 14-bit raw (Magic Lantern raw) and around 15% for 12-bit sources (Sony FS-series 12-bit raw may result in lower relative gains in this mode due to the entropy reducing nature of Sony processing). Higher relative savings are obtained for brighter images (including ETTR or cameras rated slower than nominal speed).

This mode uses standard lossless compression as per the DNG specification and the output is widely compatible with video production software.

Unlike lossless compression, lossy compression discards some information so that file sizes are reduced further. Lossy CinemaDNG compression is an extension of the CinemaDNG standard by Blackmagic Design. Lossy CinemaDNG is 12-bit.

In slimRAW lossy modes the entropy reduction step is done in a perceptually based manner (the result is often called "visually lossless" by marketing). For linear raw sources slimRAW will first do a non-linear (log) transformation of the data. Here it serves two purposes: blacks protection and an initial entropy reduction in the brights. This can be disabled in File->Other options, but you should only ever do it in some obscure cases where the input raw files contain non-linear raw data but don't include linearization metadata.

slimRAW implements both variable bit rate (VBR) constant quality lossy compression and constant bit rate (CBR) variable quality lossy compression. While VBR modes offer predictable and constant image quality, CBR modes provide predictable output file sizes.

There are two lossy VBR modes. VBR HQ is suitable for all resolutions. VBR LT is more aggressive and will lead to more savings: it is usually best suited for oversampled delivery (for example, when shooting 4K for 2K delivery). VBR HQ will typically result in average compression ratios in the range of 3:1 to 4:1 (but may get as high as 5:1). VBR LT will typically achieve 4:1 to 6:1 (but may get as high as 8:1). These ratios are based on uncompressed input sizes.

slimRAW also offers constant bit rate variable quality lossy modes (3:1, 4:1, 5:1 and 7:1). The target compression ratio is based on uncompressed input. For example, feeding slimRAW with uncompressed files of size 1 MB and selecting "3:1 lossy" will target an output file size of approximately 0.33 MB (33% of the original size); selecting "5:1 lossy" will target an output file size of approximately 0.2 MB (20% of the original size). But if you are doing recompression (for example, when the source is Blackmagic lossless CinemaDNG) the resulting ratio will depend on the level of compression in the source files. For example, if the source is 1.5:1 losslessly compressed, and 3:1 is selected as target, then the result will be approximately 2:1 (or down 50% of the original size, for a total ratio of 1.5*2 = 3:1).

The "Prioritize quality over rate" option can be used to have slimRAW take some extra care of image quality in CBR modes even if this could mean that the target ratio cannot be achieved.

Since lossy compression is an extension of the CinemaDNG specification support by video production software is more limited. See the front page for a current list of compatible software.

• slimRAW can do various cross conversions: uncompressed CinemaDNG to lossless or lossy CinemaDNG, lossless to lossy CinemaDNG, and lossy to lossless CinemaDNG. Lossy to lossless can be handy when you need to import lossy CinemaDNG in an application which only supports lossless CinemaDNG.
• Use Lossless compression for 10-bit and 8-bit raw sources.
• For linear raw sources Lossless 10-bit log offers further space gains compared to Lossless and still results in a widely compatible CinemaDNG output.
• Lossy VBR HQ and 3:1 are suitable for all resolutions.
• Lossy VBR LT and 5:1 are best suited for high resolutions and oversampled deliveries (4K for 2K, etc.). Experimentation is encouraged though.
• Lossy 7:1 is best suited for CinemaDNG proxies and oversampled delivery from 4K+ cameras (Ursa Mini 4.6K, DJI X5S, etc).
• When shooting a Blackmagic camera which allows for both lossless and lossy raw with the intent of passing the raw footage through any of the slimRAW lossy compression modes it is best to record lossless CinemaDNG in camera.