Why is it so hard to replace JPEG?

Replay: JPEG is a technologically ancient format. More modern alternatives have appeared, but none have replaced it in popularity. With Apple's HEIC format, will we finally be able to consign JPEG to the dustbin?

Has anyone recently sent you a picture from an iPhone that a non-Apple widget couldn't handle? If so, the reason that might have happened is that Apple have, for the last year (at the time of original publication) or so, been moving towards a still image format that – gasp – isn't JPEG.

It makes sense: the JPEG still image format is based on work that started in 1986, and was finally standardised in 1992. Much as with video, it's possible to do much more effective compression with more modern computers. Compare the fact that the principal video codec of the time was Cinepak, which was designed around the requirement for it to be decoded on the computing equivalent of a turnip rather than the need to achieve broadcast quality images. Cinepak has been obsolete for decades, but we still shoot (literally) billions of JPEGs a year.

So Apple's move makes sense. The format in question is increasingly called HEIC, based on the filenames created by iPhones, and was first widely discussed a little over a year ago. The picture encoding is based on some of the mathematics used by the HEVC video encoder. In its simplest incarnation, HEIC images use HEVC intra prediction, which means that to some degree an HEIC image is an HEVC I-frame. That makes sense, because HEVC already supports high bit depth, lossless compression and other things that JPEG doesn't, but which a high-end stills camera file format should (JPEG itself theoretically supports 12-bit images, but it's rarely done.)

ISO Base Media File

That's the codec. The file format itself is an ISO Base Media File, based on the QuickTime movie file format in the same way as MP4 and 3GP. So, does that mean that a HEIC file is simply a single-frame QuickTime movie, compressed with HEVC/H.265?

That would have been way, way too easy. The HEIC files produced by iPhones implement some of the things that the High Efficiency Image File format, HEIF, is designed to do; a HEIC is a type of HEIF, and HEIF does a lot of things.

Some HEIF features make a lot of sense, such as the ability to store “derivations,” which essentially means the file can store information such as the cropping or orientation. That's important in smartphones, so that the image can be displayed appropriately without actually reencoding it and losing quality. Other formats offer this, but it's often rather unevenly supported, hence the occasional appearance of cellphone video that appears upside-down or sideways. A format with these things built in at ground level is a good idea. There are other, perhaps slightly frivolous features: image sequences (sometimes called “cinemagraphs”) are supported, presumably pursuant to phone features such as live photos, as well as ways to add text or sound notations to images.

Because of all this, the format is widely considered a little unfriendly to implementers, or, to put it another way, a pain to code for, because a normal QuickTime movie decoder won't read them.

Anyway. If the idea is to replace JPEG with a more effective compressor, then fine. HEIC images should achieve similar quality at half the size of a JPEG. The thing is, if you just wanted to replace JPEG, then the Joint Photographic Experts Group has already done that. JPEG itself is not a particularly advanced thing; it's broadly similar to a single frame of DV or ProRes video.

JPEG 2000

JPEG-2000, published in its final form by the same Group in 2000, offers a number of improvements, particularly wavelet rather than cosine transforms (it formed the core of Red's original camera codec.) JPEG XR retains the cosine transform of JPEG but achieves better results with some additional, more advanced techniques.

It's hard to say why the successors to JPEG didn't take off; it may simply be that JPEG is hugely entrenched and represents a workable compromise of complexity and filesize. That factor will work against HEIF/HEIC in the same way. The new format also seems to have been created with a lot of future-proofing in mind in terms of both features (who shoots cellphone snaps then adds narration?) and technical capability. An in-depth discussion of the file format itself is beyond a single article like this, but the low-level stuff recalls MXF in that it's designed for such a degree of extensibility that it might end up being quite confusing. Extensibility sounds good, but there are penalties of complexity that can mean support is slower to emerge because it takes longer and costs more to build the software.

Apple have regularly been bolder than this in the past – remember the first iMacs that had no floppy drive? Certainly HEIC is a single file format that could replace those used for cellphone snapshots, high-end photography and even animated GIFs. Whether there's any actual need for that remains to be seen, but support is already popping up in other operating systems.