From owner-freebsd-stable Wed Jul 24 13:53:22 2002 Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 79C8937B400 for ; Wed, 24 Jul 2002 13:53:14 -0700 (PDT) Received: from mail.thinkburst.com (juno.geocomm.com [204.214.64.110]) by mx1.FreeBSD.org (Postfix) with ESMTP id ACB2443E5E for ; Wed, 24 Jul 2002 13:53:13 -0700 (PDT) (envelope-from jbozza@thinkburst.com) Received: from mailgate.thinkburstmedia.com (gateway.thinkburstmedia.com [204.214.64.100]) by mail.thinkburst.com (Postfix) with ESMTP id C4E31AEFB for ; Wed, 24 Jul 2002 15:53:13 -0500 (CDT) Received: (qmail 26576 invoked from network); 24 Jul 2002 20:53:12 -0000 From: "Jaime Bozza" To: "'Matthew Dillon'" Cc: , "'Olaf R'" Subject: RE: RE: Abominable NFSv3 read performance / FreeBSD server / Solaris client Date: Wed, 24 Jul 2002 15:52:04 -0500 Message-ID: <025c01c23353$f75cc730$6401010a@bozza> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.3416 In-Reply-To: <200207241715.g6OHF9Ne028675@apollo.backplane.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4910.0300 Importance: Normal Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Ok, I fiddled with the tcp_recv_hiwat parameter on Solaris 8, along with the tcp.sendspace parameter on FreeBSD. No matter what, anything above 32767 would continually get the delays. Just another FYI, by default, Solaris 8 sets tcp_recv_hiwat to 24576. And another interesting note. Keeping all defaults (sendspace=32768, recv_hiwat=24576) but setting rsize to 28672 or 30720, I get almost a 30% increase in speed (for this particular test) from rsize=20480. (13 seconds or so instead of 18 or so) In typing this message and testing at the same time, I came across these two parameters: nfs:nfs3_bsize nfs:nfs3_max_transfer_size Unfortunately, I don't think I can change them with a running system. (Have to set them in /etc/system and reboot to make the changes) But regardless, both are set to 32768. I wouldn't be surprised if rsize=32768 would work fine if I increased both of these values. Here are the beginnings of a couple of tcpdumps. FreeBSD to FreeBSD: 14:52:36.435205 freebsd-client.1700858811 > freebsd-server.nfs: 132 lookup [|nfs] 14:52:36.435334 freebsd-server.nfs > freebsd-client.1700858811: reply ok 236 lookup [|nfs] 14:52:36.435580 freebsd-client.1700858812 > freebsd-server.nfs: 132 lookup [|nfs] 14:52:36.435625 freebsd-server.nfs > freebsd-client.1700858812: reply ok 236 lookup [|nfs] 14:52:36.435844 freebsd-client.1700858813 > freebsd-server.nfs: 144 lookup [|nfs] 14:52:36.435884 freebsd-server.nfs > freebsd-client.1700858813: reply ok 236 lookup [|nfs] 14:52:36.436104 freebsd-client.1700858814 > freebsd-server.nfs: 124 access [|nfs] 14:52:36.436140 freebsd-server.nfs > freebsd-client.1700858814: reply ok 120 access c 65610bbf 14:52:36.436326 freebsd-client.1700858815 > freebsd-server.nfs: 124 access [|nfs] 14:52:36.436356 freebsd-server.nfs > freebsd-client.1700858815: reply ok 120 access c 65610bc0 14:52:36.441686 freebsd-client.1700858816 > freebsd-server.nfs: 132 read [|nfs] 14:52:36.441727 freebsd-client.1700858817 > freebsd-server.nfs: 132 read [|nfs] 14:52:36.441845 freebsd-server.nfs > freebsd-client.1700858816: reply ok 1472 read (frag 42077:1480@0+) 14:52:36.441851 freebsd-server > freebsd-client: (frag 42077:1480@1480+) 14:52:36.441858 freebsd-server > freebsd-client: (frag 42077:1480@2960+) 14:52:36.441863 freebsd-server > freebsd-client: (frag 42077:1480@4440+) 14:52:36.441870 freebsd-server > freebsd-client: (frag 42077:1480@5920+) 14:52:36.441876 freebsd-server > freebsd-client: (frag 42077:928@7400) 14:52:36.441965 freebsd-server.nfs > freebsd-client.1700858817: reply ok 1472 read (frag 42078:1480@0+) 14:52:36.441971 freebsd-server > freebsd-client: (frag 42078:1480@1480+) 14:52:36.441976 freebsd-server > freebsd-client: (frag 42078:1480@2960+) 14:52:36.441981 freebsd-server > freebsd-client: (frag 42078:1480@4440+) 14:52:36.441986 freebsd-server > freebsd-client: (frag 42078:1480@5920+) 14:52:36.441991 freebsd-server > freebsd-client: (frag 42078:928@7400) 14:52:36.443058 freebsd-client.1700858818 > freebsd-server.nfs: 132 read [|nfs] FreeBSD to Solaris: 14:53:22.433685 solaris-client.1823430800 > freebsd-server.nfs: 172 lookup [|nfs] (DF) 14:53:22.433820 freebsd-server.nfs > solaris-client.1823430800: reply ok 240 lookup [|nfs] (DF) 14:53:22.434593 solaris-client.1823430801 > freebsd-server.nfs: 172 lookup [|nfs] (DF) 14:53:22.434647 freebsd-server.nfs > solaris-client.1823430801: reply ok 240 lookup [|nfs] (DF) 14:53:22.435247 solaris-client.1823430802 > freebsd-server.nfs: 184 lookup [|nfs] (DF) 14:53:22.435292 freebsd-server.nfs > solaris-client.1823430802: reply ok 240 lookup [|nfs] (DF) 14:53:22.436768 solaris-client.1823430803 > freebsd-server.nfs: 164 access [|nfs] (DF) 14:53:22.436805 freebsd-server.nfs > solaris-client.1823430803: reply ok 124 access c 6caf5894 (DF) 14:53:22.440026 solaris-client.1823430804 > freebsd-server.nfs: 172 read [|nfs] (DF) 14:53:22.440120 solaris-client.1823430805 > freebsd-server.nfs: 172 read [|nfs] (DF) 14:53:22.440142 freebsd-server.nfsd > solaris-client.962: . ack 3598640967 win 33176 (DF) 14:53:22.440267 freebsd-server.nfs > solaris-client.1823430804: reply ok 984 read (DF) 14:53:22.440276 freebsd-server.nfs > solaris-client.2661457985: reply ERR 984 (DF) 14:53:22.440284 freebsd-server.nfs > solaris-client.808722359: reply ERR 984 (DF) 14:53:22.440293 freebsd-server.nfs > solaris-client.377809557: reply ERR 984 (DF) 14:53:22.440303 freebsd-server.nfs > solaris-client.3567562942: reply ERR 984 (DF) 14:53:22.440313 freebsd-server.nfs > solaris-client.2021609926: reply ERR 984 (DF) 14:53:22.440323 freebsd-server.nfs > solaris-client.1352417867: reply ERR 984 (DF) 14:53:22.440331 freebsd-server.nfs > solaris-client.2579146601: reply ERR 984 (DF) 14:53:22.440341 freebsd-server.nfs > solaris-client.1630463784: reply ERR 984 (DF) 14:53:22.440351 freebsd-server.nfs > solaris-client.1567037614: reply ERR 984 (DF) 14:53:22.440360 freebsd-server.nfs > solaris-client.1074370493: reply ERR 984 (DF) 14:53:22.440368 freebsd-server.nfs > solaris-client.3968313526: reply ERR 984 (DF) 14:53:22.440377 freebsd-server.nfs > solaris-client.1204350401: reply ERR 984 (DF) 14:53:22.440386 freebsd-server.nfs > solaris-client.1850210752: reply ERR 984 (DF) 14:53:22.440396 freebsd-server.nfs > solaris-client.787132493: reply ERR 984 (DF) 14:53:22.440405 freebsd-server.nfs > solaris-client.2182931510: reply ERR 984 (DF) 14:53:22.440413 freebsd-server.nfs > solaris-client.2865285158: reply ERR 984 (DF) 14:53:22.440422 freebsd-server.nfs > solaris-client.717978252: reply ERR 984 (DF) 14:53:22.440431 freebsd-server.nfs > solaris-client.2724501050: reply ERR 984 (DF) 14:53:22.440441 freebsd-server.nfs > solaris-client.581303660: reply ERR 984 (DF) 14:53:22.440452 freebsd-server.nfs > solaris-client.659913908: reply ERR 984 (DF) 14:53:22.440459 freebsd-server.nfs > solaris-client.50598912: reply ERR 984 (DF) 14:53:22.440468 freebsd-server.nfs > solaris-client.3323815642: reply ERR 984 (DF) 14:53:22.440475 freebsd-server.nfs > solaris-client.1853246635: reply ERR 984 (DF) 14:53:22.440484 freebsd-server.nfs > solaris-client.2530310829: reply ERR 984 (DF) I'm not really good at interpreting what I'm seeing here, but I know that the first couple of lines are looking up the file. After that, I'm at a loss with the Solaris dump, other than the fact that I'm pretty sure the "reply ERR" are causing delays somewhere. (The FreeBSD one is so clean!) Does FreeBSD attempt to send packets out using a size of NFS_MAXPACKET (33172?) all the time? Could the "win 33176" be centered around that? From what I'm seeing (correct me if I'm wrong), FreeBSD is sending 8328 bytes in each TCP packet when Sending to a FreeBSD client. I don't know enough about the internals of tcp on both systems to really figure out what's really going on here. Any pointers on where to research next would be helpful. I'm pretty sure that it would have to be something on Solaris that's causing the problem, but I'm at a loss as to what. Jaime -----Original Message----- From: Matthew Dillon [mailto:dillon@apollo.backplane.com] Sent: Wednesday, July 24, 2002 12:15 PM To: Jaime Bozza Cc: stable@FreeBSD.ORG; 'Olaf R' Subject: Re: RE: Abominable NFSv3 read performance / FreeBSD server / Solaris client Increase sendspace on the server and recvspace on the client, and make sure they've actually increased. A tcpdump should show much larger window advertisements from the solaris client, and netstat -tn on the FreeBSD box during the transfer (over and over again) should show that the FreeBSD box is able to queue up more then 32K. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message