Accurately timing data transfers with Curl
One of the most important tasks when managing a web server is to ensure that content is being delivered to your visitors at a reasonable speed. This is also important when one of your applications uses an internet-based data source. One of our projects relies on frequent calls to a particular web API to gather data. We were experiencing significant delays during testing, and went to work to find the bottleneck.
Profiling the query internally within Excel, we were counting about two seconds to connect to the site, send the query, gather the response, and disconnect. When doing the same procedure with wget from the Linux command line, wget reported closer to 0.6 seconds.
eric@eric-pc ~ $ wget www.pinoytux.com/index.php
--2014-08-12 15:16:24-- http://www.pinoytux.com/index.php
Resolving www.pinoytux.com (www.pinoytux.com)... 74.208.242.143, 2607:f1c0:1000:40e5:e2cc:7734:f2c1:a813
Connecting to www.pinoytux.com (www.pinoytux.com)|74.208.242.143|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.php’
[ <=> ] 44,856 78.4KB/s in 0.6s
2014-08-12 15:16:25 (78.4 KB/s) - ‘index.php’ saved [44856]
That’s not the whole story though; that command above, although it reported .6 seconds, actually took about two seconds – just like Excel! What’s going on? The answer is simple, and involves simple connection overheads. Every time you connect to a website, you incur a certain amount of overhead while your browser agent negotiates with the remote server. This is a non-issue most of the time, because it’s a very short duration – but if you’re making a lot of brief connections, it can add up.
Time for some slightly better diagnostics. Thanks to Rai at PinoyTux, we’re able to use a tool called Curl to measure the connection overhead: the duration between asking your system to get some data, and the first byte actually being delivered to your system.
Curl is installed by default by most versions of Linux and MacOS, but it doesn’t come as part of Windows. The Curl website stores links to different versions of binaries/installers here.
As provided by Rai, here is a sample command using Curl:
curl -o /dev/null -s -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} n" http://inserturl.here
Let’s break this command down into it’s individual components:
curl
Can’t get more self explanatory than that. Run Curl.
-s
Silent mode! Strips out the progress bar and some other stats that we don’t need right now.
-o /dev/null
Output the result of the command to the destination specified. In this case, the command outputs to a special character device file called /dev/null (the unix name for a null device), which notifies the user that the output was received but discards the contents of the output.
-w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} n"
If the operation is successful, ‘write-out’ the following output, substituting any of the desired and available variables with their values. In this case, return the time between command start and connecting to a remote host or proxy (time_connect), the time between command start and the first byte being ready to transfer (time_starttransfer), and the total time for the whole operation to run (time_total). All of these variables are measured in seconds.
http://inserturl.here
The URL that you’d like to run the test on.
Putting it all together, you’ll get an output like this:
eric@eric-pc ~ $ curl -o /dev/null -s -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} n" http://www.pinoytux.com
Connect: 0.259 TTFB: 0.838 Total time: 1.752
As you can see, Curl connects to the site in a mere 333ms… but then it takes an additional 600ms before it receives the first byte of data. Some of our experiments have revealed a delay of a massive 1200ms – enough to really slow down an application!
Times will vary depending on factors like your connection speed, but the command will give you a pretty good impression of where your site is at. To learn more about Curl, the about.com page provides a pretty good breakdown of various options.