The Insidious Creep of

● 0.1 milliseconds - Seek time latency for typical SSD drive
● 2ms - Max lag for singing with ear monitor without annoyance
● 5ms - Average hard drive access time
● 12ms - Max lag for no noticeable effect in virtual reality apps.
● 0-17ms - Theoretical best-case input lag for 60fps refresh.
● 0-33ms - Theoretical best-case input lag for 30fps refresh.
● 50ms - Lag from Guitar Hero III (Xbox 360 & CRT)
● 60ms - Infamous Samsung 244T monitor lag time.
● 100ms - Tab switching in Ubuntu 10.10 GUI
● 166ms - Average lag from Grand Theft Auto IV
● 200ms - Lag from Killzone or Kinect VR
● 225ms - Our own reaction time
● 300ms - Navigating folders in Ubuntu
● 750ms - Boot Mozilla Firefox 3.6 (Ubuntu or Windows)
● 2000ms - Time taken to switch user: Ubuntu
● 5000ms - Time taken to switch user: Windows 7

The title of this article, dramatic though it may appear, isn't actually too far off in characterizing a problem we all have to suffer on some level. Slowly, we've all been giving way to an issue which is hard to see, hard to measure, and in some cases, goes unnoticed until it's too late.
A degree of latency (delay) exists in everything from mobiles, microphones, TVs, cameras, CFL lights, and your phone reception, to the computers we use: soundcards, LCD monitors, games, networks, hard drives, and of course the GUI: buttons, open/save requestors, tab changing, browsing, you name it. In this special feature, Skytopia charts some of the devices which are most prone to this near-invisible monster.

♦ Introduction
♦ The Many Faces of Latency
♦ LCD Monitors and Lag in Games
♦ Slowing of the Intertubes
♦ Latency on the Desktop
♦ Conclusions

t shouldn't have to be this hard. I come home and sit down to watch a bit of TV after a long day's work. Nothing decent is on the first 4 channels, so I repeatedly flick through the thirty or so Freeview channels available. As I do, a noticeable delay rears its ugly head between each switch. We're talking nearly 2 seconds just to change a channel. That doesn't sound like a lot, but do it enough and believe me, you'll get sick of it. Bear in mind that (like all and terrestrial and satellite TV signals), all channels arrive simultaneously, so in theory, it should be able to immediately decode the signal on the spot.

I can imagine the conversation horror story at the design stage:

Engineer 1: "Hey we need to buffer this, otherwise we're going to get some stutter. How many frames of buffer shall we use for our video decoder?"
Engineer 2: "Well, 30 sounds like a nice round number - why don't we use that?"
Engineer 1: "Hey, that's a good idea."
Engineer 3: "Guys, just done some checks, we can reduce CPU power by 2% if we use 100 frames of buffering instead of 30."
Engineer 2: "That's what... almost 2 seconds isn't it?.... yeah sure, whatever."
Engineer 1: "Let's go with that then."

"If someone stole a large amount from your bank account, you'd notice. But if they were more sneaky, and stole smaller amounts on a regular basis, you might 'feel' something was up, but never know quite what. That's a bit what latency or input lag is like - silent, but deadly." - DW

Okay, the truth is probably a bit more subtle, with a few weak links in the chain (e.g. perhaps a lack of key frames in the video encoding), but it highlights the sloppy attitude that tends to sweep latency under the carpet.

To aggravate the problem, say hello to my new 6-in-1 TV remote. Everyone was giving it 5 stars from Amazon, so I recently bought it for 10 quid after my old one semi-died last month. To adjust the volume or other buttons before was instant, or very near instant. This new remote takes not less than 300 milliseconds to do what my old Poundland TV remote did in less than 50. Again, 300 milliseconds sounds small, but it gives a certain 'detached' feeling, almost as if someone else is controlling the remote. It has the irritation of a dripping tap or constant noisy traffic - you don't notice it at first, but it grows on you, in a bad way.

But in any case, the issues with my TV are merely the tip of this particular iceberg, a symptom of something much bigger and widespread...

The Many Faces of Latency

The issue isn't just confined to TVs; switch on a CFL bulb, and notice how long it takes to power up. Or recall those mobile conversations where there's an awkward gap between both sides of the conversation, sometimes seconds long. Or how about that painful half second wait while you wait for the Launcher to reappear in the latest Ubuntu 11 operating system. Latency (or seek time) crops up in our hard disk drives, due to the mechanical nature of spinning disks. It's only about 5 milliseconds, but when Windows is reading hundreds or even thousands of tiny files every minute, it begins to add up, sometimes to painful levels if you've experienced constant disk thrashing for 10 minutes (the new SSD drives will partly solve this issue at least).

"...when we suddenly remove the delay [on the game], it feels as though the effect on the screen happened just before they commanded it." - David M. Eagleman (neuro.bcm.edu).

As will be discussed later, some monitors, computer software, and games are afflicted with latency and lag issues. Lagged controls feel disconnected, sloppy, or sticky, and particular types of hardware lag really are nasty and affect us all. Even the internet itself can have painful degrees of lag or latency. For sure, the speed of light prevents us from sending a signal to the other side of the world instantly, but you can bet that's a tiny problem in comparison to the latency created by human stupidity (discussed later).

Composers, musicians, DJs and specifically singers also find latency especially irritating, as hearing yourself on the amp 100 or 200 milliseconds after your real voice is a disconcerting experience to put it kindly (many PC soundcards are a bit crappy in this respect, and if you happen to use Windows, you'll need ASIO to get past Windows' numerous layers of bloatedness too). After digital processing of input has taken place (e.g. reverb or more demanding effects such as time stretch or pitch shifting), the lag can approach half a second in total, which would put any band out of sync. Perhaps this is why many audio professionals use dedicated gear, rather than a 'mere' all-purpose digital laptop which would make things simpler but introduce lag much more easily.

On the extreme side of audio latency, even something as low as 2+ milliseconds can unsettle singers who are wearing headphones/ear monitors due to interference with their own voice.

Now, let's take a look at another BIG latency demon...

LCD Monitors and Lag in Games

Some cases of latency mentioned above are at least more forgivable since the problem is an inherent part of the technology used. What's not so forgivable is the lag we've sometimes seen from various games and LCD monitors. Whether you wish to avoid a soupy mouse cursor and unresponsive GUI, or just need to beat your opponent to the trigger in Halo 3, "input lag" - as it's commonly known - is a subtle, silent enemy. As we shall see, it comes in many shapes and guises...

"Players, and sometimes even designers, cannot always put into words what they feel is wrong with a particular game�s controls. Often they will try to do something that requires some synchronization, but will fail, and they won�t be able to tell you "the event happened 0.10 seconds after my input", instead they will just tell you that the game felt 'slow' or 'not tight', or 'difficult'. Or they might not be able to tell you anything, and simply say the game sucked, without really understanding why it sucked." - Mick West (Gamasutra.com)

Rarely has an issue generated so much confusion, argument and frustration as "input lag" - another form of latency. First off, you have the lag of the monitor, which can be caused by any number of factors - image scaling, colour processing, deinterlacing, and the infamous overdrive. And secondly, you have a lag internal to the game itself where the logic and rendering pipeline is often to 'blame' (lots of stuff happening in parallel which helps the frame rate, but doesn't help lag). Games like Grand Theft Auto IV and Killzone were infamous for the problem, with some people experiencing maybe up to 200 (!) milliseconds of lag ).

The lag issues become apparent particularly in quick action games such as first-person shooters where you need every last millisecond to beat your enemy to that gun trigger or to dodge that laser. Fighting games also suffer as combos often need split second timing to pull off. I personally found myself cursing my emulated pinball game (Pinball Dreams - Amiga) which had around 50ms of lag - not exactly appreciated when you try to flip the ball at the tip edge of the flipper for obvious reasons.

Interestingly, games which run at 30fps tend to have twice the lag as games which run at 60fps (due to an overly bloated frame-per-frame pipeline that game producers often use). Other than streamlining the rendering/logic engine of games and reducing lag in monitors, one of the best solutions is to increase the frame rate from 60 to 120 or even 240 frames per second. This way, we'll have super smooth action, and avoid all the issues of monitor blurring and input lag.

Virtual reality style action suffers even more. Research shows that latencies as low as even 16 milliseconds are perceptible by some due to the way our body coordinates with vision. That's particularly frustrating for mass-market VR systems such as Kinect which suffer perhaps 200 milliseconds of lag - over 10x slower than ideal, and an experience slightly akin to wading through thick treacle.

"In another line of experiments, we have subjects play various forms of simple video games. Unbeknownst to the subjects, we inject variable delays between their movement of the computer mouse, and the effect that has on the game. For example, a subject might move the mouse to the right, and 150 msec later, the movement registers on screen. Our findings show two striking results: first, participants playing the game quickly come to feel as though there is less delay between their mouse movement and the sensory feedback, and second, when we suddenly remove the delay, it feels as though the effect on the screen happened just before they commanded it." - David M. Eagleman (neuro.bcm.edu).

Initially, many people denied the problem altogether, often mistaking "input lag" for a monitor's "response time" or our own "reaction time" which is around 150-300ms but essentially has no bearing on the input lag problem whatsoever. If they passed those hurdles, they would otherwise state that latency was zero, or at least not humanly noticeable. Over time though, as more and more people complained (around the time of Dell 2405FPW?), and careful measurements took place, the naysayers and industry slowly began to take notice, but not without claiming the issue was still exaggerated by those affected. Indeed, most normal people probably don't notice latencies above say, 50 - 100 milliseconds in games (at least not consciously!). In addition, unless you have the right gear and know what you're doing, it can be tricky to accurately measure lag due to the myriad variables, so that adds a further layer of confusion.

And so we have the many colourful flame wars, of which there are indeed quite a few. Without taking anyone's side in those threads, we can all see no one would be wasting their time and money discussing it if LCD manufacturers and game companies took a bit more care in the first place.

But as bad as monitor/game latency is, even this pales into comparison to a type of latency so insidious, that it's probably costing the world billions of pounds and preventing technologies such as high-bandwidth video conferencing or VoIP from taking off more easily...

Slowing of the Intertubes

Has your internet connection been crawling lately? If so, you may just be another victim of 'Bufferbloat', a plague of poor design which has spread over the entire web, but has only recently started making its way into the headlines. It hasn't been widely noticed until now because the problem has been slowly getting worse as a result of networks increasing their 'buffer' sizes over the years (we're talking everything from the ISP backbone to your home router - yes, the problem has tentacles).

"Years ago David Cheriton at Stanford taught me something that seemed very obvious at the time -- that if you have a network link with low bandwidth then it's an easy matter of putting several in parallel to make a combined link with higher bandwidth, but if you have a network link with bad latency then no amount of money can turn any number of them into a link with good latency." - Stuart Cheshire (rescomp.stanford.edu), 1996

Essentially, it boils down to the industry's recent tunnel vision in achieving the highest raw bandwidth, without worrying about too much else. In theory this sounds like we get videos and big downloads faster, at the cost of small transfers (such as visiting a website or sending an email), but in reality it's even worse than that:- video often crawls to a stutter because these big downloads rely on frequent tiny messages between the upload site and downloader, which can't be delivered quickly because the network is heavily congested. In other words, shunning latency often has a terrible impact for raw bandwidth and big downloads anyway.

Bufferbloat video and explanation
Download this AVI animation created by Richard Scheffenegger.
The bad Bufferbloat setup is on the left (yellow dots), and the 'good' setup (i.e. how things used to be configured about 10-20 years ago when RAM was more expensive!) is on the right (cyan/blue dots).

Both sides start off okay, but notice how the left side 'queues' (tall yellow dot columns) keep on growing over time, while the right side blue columns stop short because of the small buffer size. As they stop short, some data 'packets' must be dropped, and this gets reported back to the upload site that it's shoving data to the user too fast. As a result, the upload site temporarily slows the sending of data, and thus the system self-corrects.

Meanwhile, on the left side, these packets of data never get dropped, so the giant bloated yellow buffers get filled more and more, but the computer at the upload site doesn't realise the carnage of these giant queues further down the line, and instead thinks "All is okay, let's keep sending data fast!".

Finally, when a smaller piece of data needs to be sent to the user (see 2:30+ signified by red dots on the left and dark blue dots on the right), the left side shows the red dots (which could be say, a small email) wading through giant queues to reach their destination, really slowly. Furthermore these tiny bits of data often need special 'emergency' treatment as they hold up other larger data associated with it. On the good right side, the dark blue dots have no such giant queues.

Even if you know nothing about networks, it's pretty easy to see what's happening with this AVI animation created by Richard Scheffenegger. See the boxout on the left for a full description, but to simplify (deep breath!): The bad way it currently is, large globs of data (e.g. video) will sometimes congest the network pipe and prevent communication of crucial small data, by filling up overly large buffers (along 'mediator' network points), which of course take a while to drain.

10-20 years ago, these buffers used to be small, and this is good because it meant that bits of data were dropped sooner when the buffer overspilled (and therefore those bits had to be resent), which sounds like a bad thing, but in fact is usually a good thing, because it's a signal (often the only one) to tell the upload computer to send data more slowly, just temporarily while the buffer queues dissipate, and then it can go fast again.

The sad thing is, we can have our cake and eat it if we sort our networks out. We can have high bandwidth (megabytes per second) as well as immediate access to a site or piece of data (short/quick latency). For more info, see Wikipedia, or this very short history behind the story confirming the problem. Also see the short definition and this interesting Bufferbloat FAQ. For those who want to read the full story, try Jim's blog post here, and don't forget to read his interesting replies to the comments he received. Many top network professionals and even Comcast have confirmed his findings.

Latency on the Desktop

Compared to the previous section, this will seem fairly trivial, but in fact, it too is a silent demon, capable of untold wasted hours, or a less pleasant experience at the least. Furthermore, it is rarely spoken about, at least not directly. You will however see plenty of indirect evidence as people have moaned in countless forums and blogs ever since the scourge of latency found its way into our GUIs, programs and operating systems (except maybe BeOS or Haiku). Users don't call it 'latency' or 'lag' here, but instead use terms such as 'snappy', 'bloated', 'lightning', 'tight', 'unresponsive', or 'sluggish'. Whatever, it all comes under the same roof...

There are many programs which exhibit a degree of lag. Whether it's Adobe Reader, iTunes, Word, IE or Visual Studio (especially 2010) - each are guilty to varying degrees. But for this section, I'd like to focus upon something we all have to use - the operating system. Whether it's clicking a button, changing tab, or switching user - almost nothing is completely latency-free.

(Measurements in apx. milliseconds)	Ubuntu laptop	Windows Laptop	Ubuntu desktop	Windows desktop
Switch user:	2000	6000	8000	5000
Boot Google Chrome 10 (non-fresh):	400	400	?	200
Boot Mozilla Firefox 3.6 (non-fresh):	750	1750	650	750
Tab switching in typical GUI:	100	<30	100	<30
Change tabs in Firefox 3.6:	100	50	150	50
Opening a file open requestor:	150-200	100-200	200	150-200
Navigating folders (list view - 0 items):	200	100	200	75
Navigating folders (list view - 20 items):	400	250	400	150
Opening a new disk folder/window:	500	450	500	200
Closing tabs in Firefox 3.6:	0-50	0-50	0-50	0-50
Clicking window pulldown menus:	100	<30	100	<30
Opening a disk drive folder for 1st time:	apx 1000-2000	400-500	apx 3000	apx. 400
Open location from Places/Libraries:	350-500	400-500	350-450	100
Opening up 2k file in gedit/notepad:	200	50	100-150	<30
Clicking LMB page down in slider area:	100	50	100	<30
Clicking LMB page down in slider area (files):	150-200	50	100	50
Switch to desktop with 10 windows open:	<30	300	<30	50
Closing windows:	50^	50	50	50
Clicking dropdown box gadgets:	50	<30	50-100	<30
Maximizing windows:	50-100	100	50	100

For those who want to experience the strange slippery feeling you get from mouse lag, try this excellent web applet (flash required, and doesn't quite work In Opera browser):
www.themolehole.eclipse.co.uk/mouselaggn6.swf If that fails, then try this local version.

From around 20ms, I find there's a subtle subconscious irritation, though it's hard to pick up or be too sure. At around 30ms, it becomes clearer that's something's not quite right, and becomes a source of slight irritation.

At 40ms, it hits my consciousness directly - a definite slippery feeling under my cursor, something I'm not sure I could live with.

80ms on the other hand feels like someone's smothered my desktop with melted butter. I feel sorry for anyone who has a monitor with that kind of lag (the infamous Samsung 244T is almost that, clocking in at about 60ms). One can probably add 10-20ms to those numbers as my laptop surely isn't perfect.

Despite my best efforts, and repeated measurements, these figures are only rough (I would estimate a margin-of-error of around 50ms). And yes, it wouldn't surprise me if others get faster or slower times for at least some of these benchmarks. However, in terms of at least their relative delay, this chart gives a fair indication of what you can expect.

The laptop is a 2-year old Lenevo G550 with Pentium(R) dual core running Windows 7 32bit Home and Ubuntu 10.10, and the desktop is around 3 years old with an Intel Quad-core Q6600 and dedicated GPU (Nvidia 9500GT) running Windows 7 64bit Professional. I made sure to boot from fresh and quit all unnecessary programs (including hidden processes).

Apart from switching user, Windows appears to trump Ubuntu on most benchmarks, but in this game, no one's innocent. With the arguable exception of Windows 7 on the desktop, navigating folders in either OS is completely unforgivable, especially when you need to do it several times to find the file you're looking for (we should all be using a metadata filesystem anyway, but that's another story for another day).

Ubuntu surpasses Windows when switching to the desktop or switching user, but unfortunately falls down on some other more basic operations. If you want users to think your OS is lightning-fast, then clicking window pulldown menus and switching tabs have to got to be fast. Very fast. Remember of course that the monitor adds its own lag, so if you settle for worse than 50ms, then in combination, the experience is simply going to be less engaging and snappy. Apparently, various Linux kernel patches have been submitted to reduce UI latency (particularly by Con Kolivas), but unfortunately many were never accepted due to an (over?) focus on server speed benchmarks in Linux generally. I think more of a balance is needed here.

Ubuntu 11 has reportedly made things worse in latency terms since switching between open apps takes half a second as you wait for the Launcher to reappear (it apparently automatically hides itself).

Windows on the other hand makes switching user a painful experience. If you're anything like us, you'll be switching several times a day. Each time, there's a painful 5 second gap, and that's ignoring mouse movements and clicking. Perhaps try for something closer to 30 milliseconds next time?

In conclusion, and as Mick West pointed out, remember that users won't necessarily be able to say why or how they dislike something. Unless you're ultra-fine tuned to the issue of input lag, something will only feel 'wrong' - you won't be able to pinpoint that it's actually input lag. At the most, some may say it feels sluggish, but if operations are in between sluggish and instant, most won't even say that.

Conclusions

What can we do reverse the onslaught of latency in all its forms? There's no one-stop solution, but it would first help if we recognized the problem as something more than just a minor inconvenience.

First off, we can push review sites to incorporate a latency benchmark in their reviews of hardware such as LCD monitors. This would obviously put pressure on manufacturers to streamline their technology. Maybe we can make LCD (or future OLED/QLED) monitors run at 120 or even 240 frames per second. And no, I don't mean interpolating every other frame (like some LCD TVs do), but actually accepting full independent frames at these speeds. With a decent GPU, many games (particularly in the future) are able to reach these speeds, and due to the nature of the rendering pipeline as mentioned previously, input lag will be cut in half or better. At such high frame rates, we also get the massive benefit of silky smooth, blur-free action, something not possible even at 60fps unless a black subframe is inserted between frames (imitating the CRT's 'vertical scan'), but of course, the drawback of that is the noticeable flicker for many people.

We can have our cake and eat it - let's have monitors run at 240fps or better, and forever not have to worry again about updating standards again.

Solving the 'bufferbloat' problem that plagues our networks will be slightly tricky, but not insurmountable. If even one link in the chain has a problem with bufferbloat, that can affect the whole situation. But we can start by producing new hardware which doesn't exhibit bufferbloat. Other than that, we can (hopefully) alter the settings of many devices manually. I'm sure since the news broke out, many ISPs have been doing just that.

People in charge of interface and connector standards also need to know (and hopefully do know) about latency. USB for example is sometimes associated with relatively poor latency times (particularly with regard to soundcards). But how bad is it exactly? I tried to look up the exact numbers without success. Somebody now should enter the appropriate figures in the Wikipedia USB entry. It's not a non-issue. In addition to raw throughput, it's surely one of the most important statistics you can get for a standard such as USB.

Finally, many software and games companies are starting to tackle the problem. We can do our part by recognizing the symptoms and effects of latency, and making our voice heard to any who care to listen.

Further reading:

Amusing animation of hypothetical latency in real life

Gamasutra.com - Programming Responsiveness

Gamasutra.com - Measuring Responsiveness in Video Games

Human Reaction Times as a Response to Delays in Control Systems

Eurogamer.net - Console Gaming: The Lag Factor

Bufferbloat.net

Wikipedia - Comparison of latency and bandwidth

Slashdot.org - Input Lag, Or Why Faster Isn't Always Better

Slashdot.org - Does Your LCD Play Catch Up To Your Mouse

Slashdot.org - OnLive Latency Tested

Wikipedia - Input lag