Date: Sat, 5 Jun 2010 20:54:20 -0700 From: Garrett Cooper <yanefbsd@gmail.com> To: Jeremy Chadwick <freebsd@jdc.parodius.com> Cc: Nick Rogers <ncrogers@gmail.com>, FreeBSD STABLE <freebsd-stable@freebsd.org> Subject: Re: arp -na performance w/ many permanent entries Message-ID: <AANLkTikHCs2x2qiinD9LRnbV0YhXsBKrUFuT_utstzUu@mail.gmail.com> In-Reply-To: <20100606031627.GA4742@icarus.home.lan> References: <AANLkTilKiJDLqqwgdKIAUJY-O7jXb4A4FGyPlZZSnjyj@mail.gmail.com> <AANLkTimQFH5biSlhy0Bm1Cc61DF5lutMacAn_ONiSB2W@mail.gmail.com> <20100606031627.GA4742@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Jun 5, 2010 at 8:16 PM, Jeremy Chadwick
<freebsd@jdc.parodius.com> wrote:
> On Sat, Jun 05, 2010 at 09:48:01PM -0400, Nick Rogers wrote:
>> On Mon, May 31, 2010 at 10:54 PM, Nick Rogers <ncrogers@gmail.com> wrote=
:
>>
>> >
>> > [root@ ~]# time arp -na > /dev/null
>> >
>> > real 0m12.761s
>> > user 0m2.959s
>> > sys 0m9.753s
>> > [root@ ~]#
>> >
>> >
>> > Notice that "arp -na" takes about 13s to execute even though there is =
no
>> > other load. This can get a lot worse by a few orders of magnitude on a
>> > loaded machine in a production environment, and seems to scale up line=
arly
>> > when more aliases are added to the interface (permanent ARP entries
>> > created).
>> >
>> > Is this a reasonable problem that can be fixed/improved, or am I stuck=
with
>> > the slow arp -na output? Any help or comments is greatly appreciated.
>> >
>>
>> I tried the same scenario on 8.1-BETA1 and it still takes a very long ti=
me
>> for arp(8) to complete.
>>
>> I was able to isolate the performance bottleneck to a small piece of the
>> arp(8) code. It seems that looking up the interface for an ARP entry is =
a
>> very heavy operation when that entry corresponds to an alias assigned to=
the
>> interface. Permanent ARP entries that do not correspond with an interfac=
e
>> alias do not seem to cause arp(8) to puke on the interface lookup.
>>
>> The following commands and code diff illustrates how arp(8) can be modif=
ied
>> to run a lot faster in this scenario, but obviously the associated inter=
face
>> is no longer printed for each entry.
>>
>> [root@ /usr/src/usr.sbin/arp]# uname -a
>> FreeBSD .localdomain 8.1-BETA1 FreeBSD 8.1-BETA1 #0: Thu May 27 15:03:30=
UTC
>> 2010 =A0 =A0 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC =A0=
amd64
>> [root@ /usr/src/usr.sbin/arp]# time /usr/sbin/arp -na | wc -l
>> =A0 =A0 4100
>>
>> real 0m14.903s
>> user 0m3.133s
>> sys 0m11.519s
>> [root@ /usr/src/usr.sbin/arp]# pwd
>> /usr/src/usr.sbin/arp
>> [root@ /usr/src/usr.sbin/arp]# !diff
>> diff -ruN arp.c.orig arp.c
>> --- arp.c.orig 2010-06-05 18:25:24.000000000 +0000
>> +++ arp.c 2010-06-05 18:28:19.000000000 +0000
>> @@ -562,7 +562,7 @@
>> =A0 const char *host;
>> =A0 struct hostent *hp;
>> =A0 struct iso88025_sockaddr_dl_data *trld;
>> - char ifname[IF_NAMESIZE];
>> + //char ifname[IF_NAMESIZE];
>> =A0 int seg;
>>
>> =A0 if (nflag =3D=3D 0)
>> @@ -591,8 +591,8 @@
>> =A0 }
>> =A0 } else
>> =A0 printf("(incomplete)");
>> - if (if_indextoname(sdl->sdl_index, ifname) !=3D NULL)
>> - printf(" on %s", ifname);
>> + //if (if_indextoname(sdl->sdl_index, ifname) !=3D NULL)
>> + //printf(" on %s", ifname);
>> =A0 if (rtm->rtm_rmx.rmx_expire =3D=3D 0)
>> =A0 printf(" permanent");
>> =A0 else {
>> [root@ /usr/src/usr.sbin/arp]# make clean && make
>> rm -f arp arp.o arp.4.gz arp.8.gz arp.4.cat.gz arp.8.cat.gz
>> Warning: Object directory not changed from original /usr/src/usr.sbin/ar=
p
>> cc -O2 -pipe =A0-std=3Dgnu99 -fstack-protector -Wsystem-headers -Werror =
-Wall
>> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
>> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sig=
n -c
>> arp.c
>> cc -O2 -pipe =A0-std=3Dgnu99 -fstack-protector -Wsystem-headers -Werror =
-Wall
>> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
>> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sig=
n
>> =A0-o arp arp.o
>> gzip -cn arp.4 > arp.4.gz
>> gzip -cn arp.8 > arp.8.gz
>> [root@ /usr/src/usr.sbin/arp]# time ./arp -na | wc -l
>> =A0 =A0 4099
>>
>> real 0m0.036s
>> user 0m0.015s
>> sys 0m0.021s
>> [root@ /usr/src/usr.sbin/arp]#
>>
>> Notice that 0.036s without the interface lookup is a heck of a lot faste=
r
>> than 14.903s when doing the interface lookup.
>>
>> Is there something that can be done to speedup the call to if_indextonam=
e(),
>> or would it be worthwhile for me to submit a patch that adds the ability=
to
>> skip the interface lookup as an arp(8) option?
>
> This might be a better question for either freebsd-net or
> freebsd-hackers. =A0I should warn you in advance that you might receive a
> bit of flack given that you have over 4000 IP aliases assigned to an
> interface. =A0Explaining your setup may also help people understand why i=
t
> is you need what you do.
I agree with Jeremy. I think that the problem that you've
discovered is the fact that it's using stdio-based buffered output
instead of buffering more of the contents in a string and punting it
out in larger chunks.
HTH,
-Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTikHCs2x2qiinD9LRnbV0YhXsBKrUFuT_utstzUu>