Codecs

Welcome back. In the introduction we looked at the basic mechanics of a video conference and made note that:

Data cost = FPS x Horizontal Pixels x Vertical Pixels

To put this in perspective: 1 Byte is 8 bits, and a bit can have 2 values: 0 or 1. This means that 8 bits give us 2^8 = 256 possibilities. As a result you can store 256 colours in one Byte. In 2 bytes you can store 2^8 * 2^8 = 65,536 colours, and in 3 bytes you can store 16,777,216 colours. If you've worked with Windows for a while you will recall it used to offer you the options to show 256, 65 thousand or "Millions of colours". You will almost certainly be looking at a screen that is displaying millions of colours and using 3 bytes to store each and every pixel.

Armed with this knowledge we can now do the calculation for the data cost of one second of VGA video

Data cost = 25 * 640 * 480 * 3 = 23,040,000 Bytes

So the data cost for 1 second of uncompressed video is 23 Megabytes and we all know how long it takes to download multi-megabyte sized files..... Houston I think we have a problem! Enter the codec.....

Codecs

A video codec is an extremely cunning piece of software that can take that 23 Megabytes and compress it down to a more manageable size. Small enough that we can push 1 second of video across the Internet in 1 second of real time. The 3 tricks are increasingly ingenious:

• Remove some of the colour detail. Moving from 3 to 2 bytes per pixel saves us 33% in size but results in the loss of some colour detail. To hide this from the viewer the codec looks at what the commonest colours actually in use are and tries to throw out colours that are not used. To do this it makes a table that maps from the 65 thousand spaces it has available to the 65 thousand most common colours in the image. This way a video of snowboarding, where most of the colours are shades of white, will still have lots of detail as the codec will allocate lots of spaces for shades of white in its colour table. Similarly a night scene would have lots of shades of black.
• If you look at most video you will see that, as a general rule, only a small part of the image is changing. Using this observation we come to the idea of keyframes. A keyframe is one complete 640*480 picture. Each keyframe is followed by 1 or more inbetweens. The inbetweens (that's their real name) document what has changed in the image since the last keyframe. You can see that if you had 1 keyframe per second, followed by 24 inbetweens that were only 10% of the size we have reduced the data cost by 90%. You can see how this is a vital technique, that also explains why compressed video takes a while to "put itself back together" if you fast forward (you're waiting for the next keyframe).
• The final trick leverages the observation that many areas of a picture are the same colour. Imagine we have a picture of a person on a beach with the sky in the background, and sand at their feet. Consider the top of the picture. 640 blue pixels in a row, all the same colour. So instead of storing each one individually we store something that says "the next 640 pixels are all this colour blue". We can do this with a mere 4 bytes - 2 for the colour and 2 for the number of pixels, so we have converted 1280 bytes into 4 - a massive saving. Of course it's not always that easy but you get the idea.

H.261, H.262, H.263, H.264 Codec Family

H.261 appeared in 1990, was improved to H.262 in 1995, H.263 in 1996 and H.264 in 2003. These codecs are also known by the names MPEG-1, MPEG-2, MPEG-3, and MPEG-4 (H.264) where MPEG stands for "Motion Picture Experts Group".

H.264 seems to strike a near perfect balance between the various compression techniques so has become the codec of choice for Video Conferencing, DVD Video, Blu-ray, Digital Video Broadcasting, and iPod Video to name a few.

VP8 Codec

VP8 is an open source video compression codec released by Google and used by Skype. Google for "VP8 vs H.264" if you want to read discussions similar to the LP record vs CD discussion of the 1980s. Both H.264 and VP8 do a good job and most viewers can't tell the difference when comparing sided by side. I know I can't.

So there you have it, the secrets of the codec.

Next - Protocols