From owner-freebsd-small Thu Apr 25 9:49:53 2002 Delivered-To: freebsd-small@freebsd.org Received: from iguana.icir.org (iguana.icir.org [192.150.187.36]) by hub.freebsd.org (Postfix) with ESMTP id B10C737B416 for ; Thu, 25 Apr 2002 09:49:46 -0700 (PDT) Received: (from rizzo@localhost) by iguana.icir.org (8.11.6/8.11.3) id g3PGnif34773; Thu, 25 Apr 2002 09:49:44 -0700 (PDT) (envelope-from rizzo) Date: Thu, 25 Apr 2002 09:49:44 -0700 From: Luigi Rizzo To: Brian Candler Cc: freebsd-small@FreeBSD.ORG Subject: Re: NFS root and unnecessary NFS operations Message-ID: <20020425094944.A34575@iguana.icir.org> References: <20020425174139.A871@linnet.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020425174139.A871@linnet.org> User-Agent: Mutt/1.3.23i Sender: owner-freebsd-small@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Off the top of my head, i think you might see accesses to /etc/malloc.conf (on stock 'diskless' FreeBSD installations, at least recently, /etc goes onto a RAMdisk so you don't see these accesses). using tcpdump -v -s 256 might give you a better idea on what accesses are being attempted. cheers luigi On Thu, Apr 25, 2002 at 05:41:39PM +0100, Brian Candler wrote: > This is a diskless/NFS question - I hope this is the most appropriate place > to post it. > > When setting up some diskless servers in the past, just using the standard > rc.diskless1/2 way (NFS root, ramdisk on /tmp and /var), I found that the > load generated on the NFS server was much higher than I expected. > > I built some mailservers with: > - read-only NFS root > - local disk for /var > - a read-write NFS directory for maildrops > > I found that the number of NFS operations/second generated was very high, > but reduced greatly if the root was on local disk. As a result, I was forced > to put system disks back into the machines. The measurement of ops/second > was crude: the front-panel LCD display on a Network Appliance fileserver, > and nfsstat on the machines themselves. > > Anyway, I've now decided to look into this a bit further. Here are some > tests on a diskless machine running FreeBSD 4.1.1 (I know it's ancient :-) > which has booted from a separate DHCP/TFTP server. It has its root partition > on the Netapp, using NFSv3 because that's the only way I could get the mode > bits to be seen properly, mounted read-only. > > (1) > > The first test invokes some read-only accesses to the Netapp (/usr/bin/touch > and /bin/rm, I would have thought they were cached) and some read-write > accesses to a ramdisk. > > On console 1: > > # perl -e 'for ($i=0;$i<1000;$i++) { system("touch /var/tmp/xxx"); system("rm /var/tmp/xxx");}' > > On console 2: > > # nfsstat -c -W 1 > GtAttr Lookup Rdlink Read Write Rename Access Rddir Attr Lkup BioR BioW Accs BioD > 0 0 0 0 0 0 0 0 - - - - - - > 7363 5323 2 187 0 0 6065 0 90% 86% 100% - 87% - > 11496 8336 0 287 0 0 9487 0 89% 86% 100% - 87% - > 11471 8318 0 287 0 0 9462 0 90% 86% 100% - 87% - > 9756 7069 0 244 0 0 8048 0 89% 86% 100% - 87% - > 0 0 0 0 0 0 0 0 - - - - - - > > Ouch. That's a lot of NFS operations. Now, you would hope there is some > caching going on, but if I look at 'tcpdump udp port 2049' on the interface > which connects to the Netapp I can see a lot of physical traffic is being > generated: > > # tcpdump -n -i fxp0 udp port 2049 >/dev/null > tcpdump: listening on fxp0 > ^C > 16014 packets received by filter > 0 packets dropped by kernel > > (that's for the 1000 iterations of that Perl script). > > (2) Now, if I rewrite the Perl script to remove the forks and execs and > accesses to /bin and /usr/bin, the problem goes away: > > # perl -e 'for ($i=0;$i<1000;$i++) { open F,">/var/tmp/xxx"; print F "x"; close F; unlink "/var/tmp/xxx";}' > > # tcpdump -n -i fxp0 udp port 2049 >/dev/null > tcpdump: listening on fxp0 > ^C > 14 packets received by filter > 0 packets dropped by kernel > > That's great, but it's not a realistic simulation of a server machine, > where (for example) sendmail or exim is constantly forking and execing. > > (3) Another test, just open /bin/rm for read: > > # perl -e 'for ($i=0;$i<1000;$i++) { open F,"; close F;}' > > # tcpdump -n -i fxp0 udp port 2049 >/dev/null > tcpdump: listening on fxp0 > ^C > 2014 packets received by filter > 0 packets dropped by kernel > > That's part-way between. It looks like I am getting one packet exchange each > time the file is opened: > > 17:25:53.015554 192.168.0.91.649103617 > 192.168.0.1.2049: 108 access [|nfs] > 17:25:53.015698 192.168.0.1.2049 > 192.168.0.91.649103617: reply ok 120 access c xxxxxxxx > > I tried "mount -u -o ro,noatime /" but that didn't make any difference. > > So, I'm interested to know if anyone can explain why so many NFS operations > are being generated, and whether there's a workaround. I can see two > possible solutions: > > - Run with root as a ramdisk, and mount /usr on the Netapp. This should > at least ensure that an access to /var/xxx (where /var is a local disk) > cannot generate any NFS traffic. > I think that {bin,etc,modules,sbin,stand} will fit into about 24MB. > But I guess it will still generate NFS traffic when I access > /usr/lib/sendmail or whatever, so I would have to try and put all > my applications in ramdisk too. > > - Turn on some optimisation that I am missing, or disable some checking > that the kernel is doing, when repeatedly opening the same file > (such as /bin/rm) > > Any observations gratefully received. I guess I'll have to set up a > FreeBSD-4.5 image to boot from too... > > Cheers, > > Brian Candler. > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-small" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-small" in the body of the message