Today Google announced a public DNS service they are hosting. They claim that their DNS infrastructure is faster and more secure, because their servers do some clever things. I wanted to test their performance claims, so I wrote a little script to measure a lookup times for different domains across a number of DNS servers.

Methodology

Firstly I found a list of the 1 million most popular sites. I then picked a number of DNS servers to test against. I chose Google‘s, OpenDNS‘s, my ISP (Sky/Easynet) and my old ISP’s (Plus.net).

I decided I would query an A record for each of the domains in the list, one at a time, starting with the most popular. I would query each server three times for the same domain name. The ordering was like this:

foreach (domain)
   for (i =0; i<3;i++)
      foreach (server)
         query(server, domain)

I recorded the time it took for each query to be answered, and I also recorded the replies. I was curious to see if some servers replied with different answers, or if some returned more information, i.e. additional records.

I wrote a script in PHP 5.3, and ran it from the command line on my Windows Server 2008 machine. The script used PEAR’s Net_DNS to craft and send the DNS questions. I was not using my operating system’s resolver, and I was not using any form of client side caching. I ran the experiments from my home ADSL connection and as far as I know my ISP was not interfering with my DNS requests in any way. I live in the North West of the UK.

Results

After letting this run for a few hours, and querying roughly the top 10,000 domains, I have some preliminary results.

From this table of results, we can see that Google’s median response time is 41-42ms, however, OpenDNS performs much better with a result of ~31ms. Both my current ISP and my old ISP don’t perform as well and each achieved a result between 42ms and 61ms. To get a better feel for the data I plotted an empirical CDF of the lookup times for each server.

The first impression I can make from this CDF is that OpenDNS serves far more of the queries faster than anyone else. Secondly the secondary DNS servers all seem to be faster than their primaries. I suspect this is because most hosts query the primary, and rarely query the secondary. I even read that BIND (a popular DNS server) has/had a bug in it which favoured the primary DNS.

The minimum lookup time for each pair of DNS servers seems to be the same, most likely caused by the network latency between me and the servers. Even so, if we normalise all the data by taking the servers’ minimum value away from each sample, we still find that OpenDNS performs better than Google, and Google performs slightly better than Plus.net and quite a bit better than my current ISP, Easynet.

Rather worryingly is that the latency to OpenDNS is smaller than the latency to my own ISP’s DNS servers. This makes me wonder where the hell my ISP hosts their DNS servers. Also, the ~38ms minimum time with Google indicates that at least some of their DNS servers are hosted in Europe, and possibly the UK.

SInce I ran each lookup three times, I wanted to compare the lookup times for each request. This time I plotted the empirical CDF of each iteration of request.

This CDF seems to show that the 2nd and 3rd requests always get served quicker than the first. In most cases the 2nd and 3rd request have equal ranking, but the first is always slow. This could easily be attributed to the fact that the caching DNS server does not have the record in its cache, and thus must be fetched. The second time I request the domain name (only moments later), the server already has the query, and most likely has it stored in RAM or L1/L2 CPU cache.

OpenDNS_B seems to respond equally quick for the first, second and third request. This could be because I would always query OpenDNS_A first, then move on to B. If A and B were actually the same machine, it would be like sending 6 requests to the machine instead of 3. Therefore, B’s 1st request is actually its second. Even if A and B were different machines, there could be some clever replication, or shared caching going on to cause this behaviour.

Finally, I’m surprised that the 2nd and 3rd requests are slower, especially since I’m requesting the most popular domain names. Surely others would have already requested the domain name, and thus the DNS server has no need to fetch it. Looking through the list of domain names I see that none of them have the www prefix. I personally never type the www and just hope the site works, but perhaps many users do. Maybe I should re-run the experiment with the www prefix.

Conclusion

For now I would stick to using OpenDNS, as this is clearly the winner. However, the Google DNS service is very new, so perhaps the servers haven’t had enough time to fill their caches, and their admins haven’t had enough time to tweak them. I will perhaps rerun this experiment in a few weeks and see what happens.

TODO

I still have some analysis to do, for example, looking at packet loss, the type of records returned, and anything else I can think of.

I have mentioned many times that I would like to know how much current something is drawing, so today Nic brought me a mains power meter. It is a very simple meter which plugs between the mains and the device and can measure volts, amps, watts, frequency, and a few other metrics. I decided to run around my house measuring everything I could.

Methodology

A couple notes about my methodology, firstly, I was very lazy in my measurement taking. I repeated some of the measurements, but not all. However, when I did repeat the measurements they always seem to have the same value, so this discouraged me from repeating.

To test a device I would plug the meter between it and the mains electricity and give it a some time to settle. Once settled I would take the reading. Sometimes the reading would flicker between two values, in these cases I took the higher of the two. Finally, I noticed that after I turned the devices off the meter would always read 8W. Either the meter was unable to read less than 8W, or zero was not calibrated correctly on the meter.

All these measurements were taken in my house in the UK, which according to my meter has mains electricity at 248V and at a AC frequency of 59.9Hz

Results

Some observations I made about these results:

• My speakers use the same amount of power playing music as when they aren’t. I assume this is because the amps in them are always trying to amplify the signal.
• My Sky+ box is pretty poor at conserving power. I found this interesting because Sky made a big deal about the Sky box automatically turning off at night. It even advertises this fact in big letters on the side of the packaging.
• My PC which has a 800W power supply, with a quad core Intel, 4GB of RAM, and 6 harddrives, obviously doesn’t need 800W of power. The peak value, which I found during POST, was only 230W, with it idling (in Windows) at 128W.
• Linux uses more power than Windows! This is using the latest Debian testing (2.6.30-8), and Windows Server 2008. Clearly, Linux has less energy efficient drivers, or is not powering something down that Windows is.