Date: Wed, 10 Feb 2010 10:36:42 -0800 From: Kurt Buff <kurt.buff@gmail.com> To: freebsd-questions@freebsd.org Subject: Re: curl question - not exactly on-topic Message-ID: <a9f4a3861002101036x1cd9d138q7e42ee62d288d6c2@mail.gmail.com> In-Reply-To: <20100210180329.GA9318@dan.emsphone.com> References: <a9f4a3861002091721h6b38e3beu5e55f0bbf4bff9e5@mail.gmail.com> <20100210050518.GA64193@dan.emsphone.com> <a9f4a3861002100935w32b21832v6cf26e517a64a885@mail.gmail.com> <20100210180329.GA9318@dan.emsphone.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 10, 2010 at 10:03, Dan Nelson <dnelson@allantgroup.com> wrote:
> In the last episode (Feb 10), Kurt Buff said:
>> On Tue, Feb 9, 2010 at 21:05, Dan Nelson <dnelson@allantgroup.com> wrote=
:
>> > In the last episode (Feb 09), Kurt Buff said:
>> >> Actually, it's not merely a curl question, it's a "curl and squid"
>> >> question.
>> >>
>> >> I'm trying to determine the cause of a major slowdown in web browsing=
on
>> >> our network, so I've put curl on the squid box, and am using the foll=
owing
>> >> incantations to see if I can determine the cause of the slowdown:
>> >>
>> >> =C2=A0 curl -s -w "%{time_total}\n" "%{time_namelookup}\n" -o /dev/nu=
ll http://www.example.com
>> >>
>> >> and
>> >>
>> >> =C2=A0 curl -s -w "%{time_total}\n" "%{time_namelookup}\n" -o /dev/nu=
ll -x 192.168.1.72 http://www.example.com
>> >>
>> >> The problem arises with the second version, which uses the proxy. The
>> >> first incantation just returns the times, which is exactly what I wan=
t.
>> >>
>> >> However, when I use the -x parameter, to use the proxy, I get html
>> >> returned as well as the times, which is a pain to separate out.
>> >
>> > Your problem is what's after -w. =C2=A0You want one argument:
>> > "%{time_total}\n%{time_namelookup}\n", not two. =C2=A0=C2=A0With your =
original
>> > command, "%{time_namelookup}\n" is treated as another URL to fetch.
>> > With no proxy option, curl realizes it's not an url immediately and
>> > skips to the next argument on the commandline - http://www.example.com=
.
>> > With a proxy, curl has to send each url to the proxy for processing.
>> > The proxy probably returns a "400 Bad Request" error on the first
>> > (invalid) url, which is redirected to /dev/null. =C2=A0=C2=A0The next =
url doesn't
>> > have another -o so it falls back to printing to stdout.
>> >
>> > Adding -v to the curl commandline will help you diagnose problems like
>> > this.
>>
>> Thanks for that, though it's unfortunate.
>>
>> I would really like a better understanding of the times, to help further
>> diagnose the problem, and 'man curl' says that multiple invocations of
>> '-w' will result in the last one winning, which I've verified.
>>
>> Do you have any suggestions for a way to get the timing of these
>> operations without resorting to tcpdump?
>
> Does -w "%{time_total}\n%{time_namelookup}\n" not do what you want? =C2=
=A0There
> are a bunch of other time_* variables you could add, too.
>
> Also, there's nothing wrong with tcpdump (or wireshark). =C2=A0If your tr=
affic
> passes through multiple proxies or content-analyzing firewalls, you can r=
un
> multiple simultaneous tcpdumps, one on each interface. =C2=A0Then you can=
run
> your curl command, and compare the traces side-by-side and see if any
> servers are taking longer than expected to forward the data. =C2=A0If you=
have a
> managed switch, you might even be able to configure a "monitor" port that
> will forward all traffic it sees to that port, and you can run just one
> tcpdump and see the same packet multiple times as it passes from server t=
o
> server.
Sigh. A failure of imagination on my part. Putting multiple parameters
inside a single set of quote marks works exactly as I need.
Nothing wrong with wireshark/tcpdump, but I'm not nearly as competent
with them as I'd like to be, and curl offers a pretty easy way to
break out the timing of various parts of the conversation - in
particular the name resolution vs. the rest of the conversation for
any given transaction.
I do indeed have monitor ports set up on my switches, and use them to
feed ntop, with an occasional tcpdump capture to figure out problems.
If it comes to that, I'll get those traces and work on the comparisons
as you suggest. However, given what I've seen so far, I'm looking at
my firewall as being the culprit, and further testing with curl should
give me the [dis]confirmation I need.
Thanks ever so much for your help. Greatly appreciated.
Kurt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a9f4a3861002101036x1cd9d138q7e42ee62d288d6c2>