From owner-freebsd-stable@FreeBSD.ORG Tue Jun 1 02:54:27 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45070106566C for ; Tue, 1 Jun 2010 02:54:27 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0B2FD8FC0A for ; Tue, 1 Jun 2010 02:54:26 +0000 (UTC) Received: by iwn5 with SMTP id 5so705962iwn.13 for ; Mon, 31 May 2010 19:54:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=HLBxYSD9zVohYHUQNxrTPCke46Tx5iEk5OcBryVjhJI=; b=r1IcjQMuj5qac1h35Nap77bisCuyuCnDzu7AYfqG90m+4fxa99M5ILJBr3txu9HHPH DpD2Rx2AWZqMaVJrAk2D5Fv7uVqOy9czFlT6saPN3xBb+B12rrBAnExywItRReTv1v9k 2llyoQ7fecxhJpB1/DkGLNFQfXVwhmIofhlaA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=eIOQiiKd5vxDyyYfxlT9vBUoD5h32qvTJgLhkMiPcrOngarFM+fm727eVemAv4BBmt /Ck7s2VWWcSyq5aksAdKgPBx8MsZPAPUrJbxD0cID+HrQdzL0iN0+pT3WHSG1R1O/sFw LqStTS383xjRo78WyWHmqckHD00Pm10K5uBUg= MIME-Version: 1.0 Received: by 10.231.187.6 with SMTP id cu6mr6889735ibb.76.1275360866233; Mon, 31 May 2010 19:54:26 -0700 (PDT) Received: by 10.231.192.144 with HTTP; Mon, 31 May 2010 19:54:26 -0700 (PDT) Date: Mon, 31 May 2010 22:54:26 -0400 Message-ID: From: Nick Rogers To: FreeBSD STABLE Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: arp -na performance w/ many permanent entries X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jun 2010 02:54:27 -0000 I have an 8.0-RELEASE system with 4000 "permanent" ARP entries due to having a network interface (em(4)) configured with 4000 aliases. The "arp -na" command takes what I consider to be an extremely long time to finish (up to 30s on an otherwise unloaded system). I am able to replicate this in a test environment by installing 8.0-RELEASE-amd64 on a VMWare VM w/ 1GB of RAM and a 2GHz CPU. The 4000 aliases/entries is arbitrary, but nicely illustrates the performance problem. The performance is much worse on a real/loaded system. I realize the 4k aliases on an interface is unusual but I have been effectively using this configuration in my network to try and keep my end-users's each on his/her own broadcast domain. The box is a router and I allocate addresses to each user and put each on his/her own subnet with a netmask of /30. If you would like more info on this I can provide it, but it has worked effectively in FreeBSD 6.0-7.2. The slow performance of "arp -na" is an issue for me because I have a web/CGI tool that runs various reports, many of them relying on acquiring the current "ARP table", and the performance of arp(8) makes the web interface extremely slow. I believe the problem was introduced between 7.2 and 8.0, when, as far as I understand, parts of the ARP subsystem were improved. In 7.2, the aliases configured on an interface were not considered ARP entries (at least according to arp(8)), but as of 8.0 they are marked as "permanent" ARP entries and displayed by arp(8), which seems to attribute to the performance problem. I ran the following perl script to setup my test system. This script was run after installing 8.0-RELEASE and adding the bash, perl, and p5-NetAddr-IP packages via pkg_add -r. #!/usr/bin/perl use strict; use diagnostics; use NetAddr::IP; my $interface = 'em1'; my $cidr = '10.0.0.1/18'; # configure the interface with 4000 or so aliases foreach my $na (@{NetAddr::IP->new($cidr)->splitref(30)}) { my $ip = $na->addr(); my $mask = $na->mask(); my $bcast = $na->broadcast()->addr(); my $cmd = "ifconfig $interface inet alias $ip netmask $mask broadcast $bcast"; print STDERR "$cmd\n"; system($cmd); } The results are as follows: [root@ ~]# uname -a FreeBSD .localdomain 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:02:08 UTC 2009 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 [root@ ~]# ifconfig -a | wc -l 4113 [root@ ~]# ifconfig -a | head -15 em0: flags=8843 metric 0 mtu 1500 options=9b ether 00:0c:29:65:4d:3e inet 172.16.16.244 netmask 0xffffff00 broadcast 172.16.16.255 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8843 metric 0 mtu 1500 options=9b ether 00:0c:29:65:4d:48 inet 10.0.0.0 netmask 0xfffffffc broadcast 10.0.0.3 inet 10.0.0.4 netmask 0xfffffffc broadcast 10.0.0.7 inet 10.0.0.8 netmask 0xfffffffc broadcast 10.0.0.11 inet 10.0.0.12 netmask 0xfffffffc broadcast 10.0.0.15 inet 10.0.0.16 netmask 0xfffffffc broadcast 10.0.0.19 inet 10.0.0.20 netmask 0xfffffffc broadcast 10.0.0.23 [root@ ~]# time ifconfig -a > /dev/null real 0m0.032s user 0m0.023s sys 0m0.008s [root@ ~]# arp -na | wc -l 4100 [root@ ~]# arp -na | tail -15 ? (10.0.5.80) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.5.48) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.5.16) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.1.244) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.1.212) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.1.180) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.1.148) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.1.116) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.1.84) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.1.52) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (10.0.1.20) at 00:0c:29:65:4d:48 on em1 permanent [ethernet] ? (172.16.16.1) at 00:50:56:c0:00:08 on em0 [ethernet] ? (172.16.16.2) at 00:50:56:ea:ea:1a on em0 [ethernet] ? (172.16.16.254) at 00:50:56:f2:75:00 on em0 [ethernet] ? (172.16.16.244) at 00:0c:29:65:4d:3e on em0 permanent [ethernet] [root@ ~]# uptime 7:28PM up 42 mins, 2 users, load averages: 0.00, 0.00, 0.00 [root@ ~]# time arp -na > /dev/null real 0m12.761s user 0m2.959s sys 0m9.753s [root@ ~]# Notice that "arp -na" takes about 13s to execute even though there is no other load. This can get a lot worse by a few orders of magnitude on a loaded machine in a production environment, and seems to scale up linearly when more aliases are added to the interface (permanent ARP entries created). Is this a reasonable problem that can be fixed/improved, or am I stuck with the slow arp -na output? Any help or comments is greatly appreciated.