From owner-freebsd-stable@FreeBSD.ORG Sun Jun 6 03:54:23 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27783106566B for ; Sun, 6 Jun 2010 03:54:23 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id C952D8FC18 for ; Sun, 6 Jun 2010 03:54:22 +0000 (UTC) Received: by vws18 with SMTP id 18so1157162vws.13 for ; Sat, 05 Jun 2010 20:54:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=44+uVNFCYaVPHYGJp1ITtc36XFp3HVx6BD+/Ez2ASeM=; b=fUAKLe3ajgRJv/BG6HXUKEPw2aYZ5I7SU+vF2YbN9om6EGlpzc3CN5ckItJhgusdwl yrp1G331SHBZv9C00z2bBpjWSYsWbR+AvbsiCawxbO4ZP/cPgMVDDrqJDdL9PsAjSfXj QHW6wz7yUrVAdwVLJN0RSUO1GsYrGzeBpoIPw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=vf4NuGhS8qgxDHGyqoa+KfU2zjDDbhFsA5zfvNWJiMTELJfYCQZrr/AzOIhCTadQ7+ VLpWsvQaJlMNVIyhr1h7YGQvybTMHmQ/tkZyK7kyvQjcbTl/y2Rt6JL3yhTeD1sMMwyK BnVfVfUczIq4NaP+NgruQh/Idw+Ps+PIDvd5g= MIME-Version: 1.0 Received: by 10.224.53.153 with SMTP id m25mr1442074qag.133.1275796460818; Sat, 05 Jun 2010 20:54:20 -0700 (PDT) Received: by 10.229.233.74 with HTTP; Sat, 5 Jun 2010 20:54:20 -0700 (PDT) In-Reply-To: <20100606031627.GA4742@icarus.home.lan> References: <20100606031627.GA4742@icarus.home.lan> Date: Sat, 5 Jun 2010 20:54:20 -0700 Message-ID: From: Garrett Cooper To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Nick Rogers , FreeBSD STABLE Subject: Re: arp -na performance w/ many permanent entries X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jun 2010 03:54:23 -0000 On Sat, Jun 5, 2010 at 8:16 PM, Jeremy Chadwick wrote: > On Sat, Jun 05, 2010 at 09:48:01PM -0400, Nick Rogers wrote: >> On Mon, May 31, 2010 at 10:54 PM, Nick Rogers wrote= : >> >> > >> > [root@ ~]# time arp -na > /dev/null >> > >> > real 0m12.761s >> > user 0m2.959s >> > sys 0m9.753s >> > [root@ ~]# >> > >> > >> > Notice that "arp -na" takes about 13s to execute even though there is = no >> > other load. This can get a lot worse by a few orders of magnitude on a >> > loaded machine in a production environment, and seems to scale up line= arly >> > when more aliases are added to the interface (permanent ARP entries >> > created). >> > >> > Is this a reasonable problem that can be fixed/improved, or am I stuck= with >> > the slow arp -na output? Any help or comments is greatly appreciated. >> > >> >> I tried the same scenario on 8.1-BETA1 and it still takes a very long ti= me >> for arp(8) to complete. >> >> I was able to isolate the performance bottleneck to a small piece of the >> arp(8) code. It seems that looking up the interface for an ARP entry is = a >> very heavy operation when that entry corresponds to an alias assigned to= the >> interface. Permanent ARP entries that do not correspond with an interfac= e >> alias do not seem to cause arp(8) to puke on the interface lookup. >> >> The following commands and code diff illustrates how arp(8) can be modif= ied >> to run a lot faster in this scenario, but obviously the associated inter= face >> is no longer printed for each entry. >> >> [root@ /usr/src/usr.sbin/arp]# uname -a >> FreeBSD .localdomain 8.1-BETA1 FreeBSD 8.1-BETA1 #0: Thu May 27 15:03:30= UTC >> 2010 =A0 =A0 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC =A0= amd64 >> [root@ /usr/src/usr.sbin/arp]# time /usr/sbin/arp -na | wc -l >> =A0 =A0 4100 >> >> real 0m14.903s >> user 0m3.133s >> sys 0m11.519s >> [root@ /usr/src/usr.sbin/arp]# pwd >> /usr/src/usr.sbin/arp >> [root@ /usr/src/usr.sbin/arp]# !diff >> diff -ruN arp.c.orig arp.c >> --- arp.c.orig 2010-06-05 18:25:24.000000000 +0000 >> +++ arp.c 2010-06-05 18:28:19.000000000 +0000 >> @@ -562,7 +562,7 @@ >> =A0 const char *host; >> =A0 struct hostent *hp; >> =A0 struct iso88025_sockaddr_dl_data *trld; >> - char ifname[IF_NAMESIZE]; >> + //char ifname[IF_NAMESIZE]; >> =A0 int seg; >> >> =A0 if (nflag =3D=3D 0) >> @@ -591,8 +591,8 @@ >> =A0 } >> =A0 } else >> =A0 printf("(incomplete)"); >> - if (if_indextoname(sdl->sdl_index, ifname) !=3D NULL) >> - printf(" on %s", ifname); >> + //if (if_indextoname(sdl->sdl_index, ifname) !=3D NULL) >> + //printf(" on %s", ifname); >> =A0 if (rtm->rtm_rmx.rmx_expire =3D=3D 0) >> =A0 printf(" permanent"); >> =A0 else { >> [root@ /usr/src/usr.sbin/arp]# make clean && make >> rm -f arp arp.o arp.4.gz arp.8.gz arp.4.cat.gz arp.8.cat.gz >> Warning: Object directory not changed from original /usr/src/usr.sbin/ar= p >> cc -O2 -pipe =A0-std=3Dgnu99 -fstack-protector -Wsystem-headers -Werror = -Wall >> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes >> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sig= n -c >> arp.c >> cc -O2 -pipe =A0-std=3Dgnu99 -fstack-protector -Wsystem-headers -Werror = -Wall >> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes >> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sig= n >> =A0-o arp arp.o >> gzip -cn arp.4 > arp.4.gz >> gzip -cn arp.8 > arp.8.gz >> [root@ /usr/src/usr.sbin/arp]# time ./arp -na | wc -l >> =A0 =A0 4099 >> >> real 0m0.036s >> user 0m0.015s >> sys 0m0.021s >> [root@ /usr/src/usr.sbin/arp]# >> >> Notice that 0.036s without the interface lookup is a heck of a lot faste= r >> than 14.903s when doing the interface lookup. >> >> Is there something that can be done to speedup the call to if_indextonam= e(), >> or would it be worthwhile for me to submit a patch that adds the ability= to >> skip the interface lookup as an arp(8) option? > > This might be a better question for either freebsd-net or > freebsd-hackers. =A0I should warn you in advance that you might receive a > bit of flack given that you have over 4000 IP aliases assigned to an > interface. =A0Explaining your setup may also help people understand why i= t > is you need what you do. I agree with Jeremy. I think that the problem that you've discovered is the fact that it's using stdio-based buffered output instead of buffering more of the contents in a string and punting it out in larger chunks. HTH, -Garrett