From owner-freebsd-questions@FreeBSD.ORG Wed Feb 10 18:03:31 2010 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A96511065676 for ; Wed, 10 Feb 2010 18:03:31 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email1.allantgroup.com (email1.emsphone.com [199.67.51.115]) by mx1.freebsd.org (Postfix) with ESMTP id 6D2158FC13 for ; Wed, 10 Feb 2010 18:03:30 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by email1.allantgroup.com (8.14.0/8.14.0) with ESMTP id o1AI3TZZ066480 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 10 Feb 2010 12:03:30 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.4/8.14.3) with ESMTP id o1AI3TUp003413 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 10 Feb 2010 12:03:29 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.4/8.14.3/Submit) id o1AI3Tci003412; Wed, 10 Feb 2010 12:03:29 -0600 (CST) (envelope-from dan) Date: Wed, 10 Feb 2010 12:03:29 -0600 From: Dan Nelson To: Kurt Buff Message-ID: <20100210180329.GA9318@dan.emsphone.com> References: <20100210050518.GA64193@dan.emsphone.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-OS: FreeBSD 7.2-STABLE User-Agent: Mutt/1.5.20 (2009-06-14) X-Virus-Scanned: clamav-milter 0.95.3 at email1.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0.2 (email1.allantgroup.com [199.67.51.78]); Wed, 10 Feb 2010 12:03:30 -0600 (CST) X-Scanned-By: MIMEDefang 2.45 Cc: freebsd-questions@freebsd.org Subject: Re: curl question - not exactly on-topic X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Feb 2010 18:03:31 -0000 In the last episode (Feb 10), Kurt Buff said: > On Tue, Feb 9, 2010 at 21:05, Dan Nelson wrote: > > In the last episode (Feb 09), Kurt Buff said: > >> Actually, it's not merely a curl question, it's a "curl and squid" > >> question. > >> > >> I'm trying to determine the cause of a major slowdown in web browsing on > >> our network, so I've put curl on the squid box, and am using the following > >> incantations to see if I can determine the cause of the slowdown: > >> > >>   curl -s -w "%{time_total}\n" "%{time_namelookup}\n" -o /dev/null http://www.example.com > >> > >> and > >> > >>   curl -s -w "%{time_total}\n" "%{time_namelookup}\n" -o /dev/null -x 192.168.1.72 http://www.example.com > >> > >> The problem arises with the second version, which uses the proxy. The > >> first incantation just returns the times, which is exactly what I want. > >> > >> However, when I use the -x parameter, to use the proxy, I get html > >> returned as well as the times, which is a pain to separate out. > > > > Your problem is what's after -w.  You want one argument: > > "%{time_total}\n%{time_namelookup}\n", not two.  With your original > > command, "%{time_namelookup}\n" is treated as another URL to fetch. > > With no proxy option, curl realizes it's not an url immediately and > > skips to the next argument on the commandline - http://www.example.com. > > With a proxy, curl has to send each url to the proxy for processing. > > The proxy probably returns a "400 Bad Request" error on the first > > (invalid) url, which is redirected to /dev/null.  The next url doesn't > > have another -o so it falls back to printing to stdout. > > > > Adding -v to the curl commandline will help you diagnose problems like > > this. > > Thanks for that, though it's unfortunate. > > I would really like a better understanding of the times, to help further > diagnose the problem, and 'man curl' says that multiple invocations of > '-w' will result in the last one winning, which I've verified. > > Do you have any suggestions for a way to get the timing of these > operations without resorting to tcpdump? Does -w "%{time_total}\n%{time_namelookup}\n" not do what you want? There are a bunch of other time_* variables you could add, too. Also, there's nothing wrong with tcpdump (or wireshark). If your traffic passes through multiple proxies or content-analyzing firewalls, you can run multiple simultaneous tcpdumps, one on each interface. Then you can run your curl command, and compare the traces side-by-side and see if any servers are taking longer than expected to forward the data. If you have a managed switch, you might even be able to configure a "monitor" port that will forward all traffic it sees to that port, and you can run just one tcpdump and see the same packet multiple times as it passes from server to server. -- Dan Nelson dnelson@allantgroup.com