From owner-freebsd-performance@FreeBSD.ORG Wed Feb 23 18:44:57 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DFDDE16A4CE; Wed, 23 Feb 2005 18:44:56 +0000 (GMT) Received: from corp.globat.com (corp.globat.com [216.193.201.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id 27AEE43D41; Wed, 23 Feb 2005 18:44:56 +0000 (GMT) (envelope-from drice@globat.com) Received: from globat.com (globat [66.159.202.156]) by corp.globat.com (8.12.11/8.12.9) with ESMTP id j1NIiqHZ087019; Wed, 23 Feb 2005 10:44:52 -0800 (PST) (envelope-from drice@globat.com) From: David Rice Organization: Globat To: Robert Watson Date: Wed, 23 Feb 2005 10:44:54 -0800 User-Agent: KMail/1.5.4 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200502231044.54801.drice@globat.com> cc: freebsd-performance@freebsd.org Subject: Re: High traffic NFS performance and availability problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Feb 2005 18:44:57 -0000 Where can I find the MPSAFE version of the amr PERC driver. I checked the release notes for 5.3-STABLE and it makes no refrence to the amr driver being MPSAFE. On Monday 21 February 2005 01:26 pm, Robert Watson wrote: > On Mon, 21 Feb 2005, David Rice wrote: > > Here are the snapshots of the output you requested. These are from the > > NFS server. We have just upgraded them to 5.3-RELEASE as so many have > > recomended. Hope that makes them more stable. The performance still > > needs some attention. > > In the top output below, it looks like there's a lot of contention on > Giant. In 5.3-RELEASE and before, the amr driver is not MPSAFE, but my > understanding is that in 5-STABLE, it has been made MPSAFE, which may make > quite a difference in performance. I pinged Scott Long, who did the work > on the driver, and he indicated that backporting the patch to run on > -RELEASE would be quite difficult, so an upgrade to 5-STABLE is the best > way to get the changes. I believe that you can build a 5-STABLE kernel > and run with a 5.3-RELEASE user space to avoid having to commit to a full > upgrade to see if that helps or not. > > Two other observations: > > - It looks like the amr storage array is pretty busy, which may be part of > the issue. > > - It looks like you have four processors, suggesting a two-processor Xeon > with hyper-threading turned on. For many workloads, hyper-threading does > not improve performance, so you may want to try turning that off in the > BIOS to see if that helps. > > Robert N M Watson > > > Thank You > > > > ------------------------------------------------------------------------- > >------------------------- D USERNAME PRI NICE SIZE RES STATE C > > TIME WCPU CPU COMMAND 4 users Load 5.28 19.37 28.00 > > Feb 21 12:18 > > > > Mem:KB REAL VIRTUAL VN PAGER SWAP > > PAGER Tot Share Tot Share Free in out in out > > Act 19404 2056 90696 3344 45216 count > > All 1020204 4280 4015204 7424 pages > > zfod > > Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt cow > > 7226 total 5128 5 60861 3 14021584 9 152732 wire > > 4: sio0 23228 act 6: fdc0 30.2%Sys 11.8%Intr 0.0%User 0.0%Nice > > 58.0%Idl 803616 inact 128 8: rtc > > > > | | | | | | | | | | 43556 cache 13: > > | | | | | | | | | | npx > > > > ===============++++++ 1660 free 15: > > ata daefr 6358 16: bge Namei Name-cache Dir-cache > > prcfr 1 17: bge Calls hits % hits % > > react 18: mpt 1704 971 57 11 1 > > pdwak 19: mpt 5342 pdpgs 639 24: amr Disks amrd0 da0 > > pass0 pass1 pass2 intrn 100 0: clk KB/t 22.41 > > 0.00 0.00 0.00 0.00 114288 buf > > tps 602 0 0 0 0 510 dirtybuf > > MB/s 13.16 0.00 0.00 0.00 0.00 70235 desiredvnodes > > % busy 100 0 0 0 0 20543 numvnodes > > 7883 freevnodes > > ------------------------------------------------------------------------- > >---------------- last pid: 10330; load averages: 14.69, 11.81, 18.62 > > up 0+09:01:13 12:32:57 > > 226 processes: 5 running, 153 sleeping, 57 waiting, 11 lock > > CPU states: 0.1% user, 0.0% nice, 66.0% system, 24.3% interrupt, 9.6% > > idle Mem: 23M Active, 774M Inact, 150M Wired, 52M Cache, 112M Buf, 1660K > > Free Swap: 1024M Total, 124K Used, 1024M Free > > > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU > > COMMAND 63 root -44 -163 0K 12K WAIT 0 147:05 45.07% 45.07% > > swi1: net 30 root -68 -187 0K 12K WAIT 0 101:39 32.32% > > 32.32% irq16: bge0 > > 12 root 117 0 0K 12K CPU2 2 329:09 19.58% 19.58% idle: > > cpu2 11 root 116 0 0K 12K CPU3 3 327:29 19.24% 19.24% > > idle: cpu3 13 root 114 0 0K 12K RUN 1 263:39 16.89% > > 16.89% idle: cpu1 14 root 109 0 0K 12K CPU0 0 228:50 > > 12.06% 12.06% idle: cpu0 368 root 4 0 1220K 740K *Giant 3 > > 45:27 7.52% 7.52% nfsd 366 root 4 0 1220K 740K *Giant 0 > > 48:52 7.28% 7.28% nfsd 364 root 4 0 1220K 740K *Giant 3 > > 53:01 7.13% 7.13% nfsd 367 root -8 0 1220K 740K biord 3 > > 41:22 7.08% 7.08% nfsd 372 root 4 0 1220K 740K *Giant 0 > > 28:54 7.08% 7.08% nfsd 365 root -1 0 1220K 740K *Giant 3 > > 51:53 6.93% 6.93% nfsd 370 root -1 0 1220K 740K nfsslp 0 > > 32:49 6.84% 6.84% nfsd 369 root -8 0 1220K 740K biord 1 > > 36:40 6.49% 6.49% nfsd 371 root 4 0 1220K 740K *Giant 0 > > 25:14 6.45% 6.45% nfsd 374 root -1 0 1220K 740K nfsslp 2 > > 22:31 6.45% 6.45% nfsd 377 root 4 0 1220K 740K *Giant 2 > > 17:21 5.52% 5.52% nfsd 376 root -4 0 1220K 740K *Giant 2 > > 15:45 5.37% 5.37% nfsd 373 root -4 0 1220K 740K ufs 3 > > 19:38 5.18% 5.18% nfsd 378 root 4 0 1220K 740K *Giant 2 > > 13:55 4.54% 4.54% nfsd 379 root -8 0 1220K 740K biord 3 > > 12:41 4.49% 4.49% nfsd 380 root 4 0 1220K 740K - 2 > > 11:26 4.20% 4.20% nfsd 3 root -8 0 0K 12K - 1 > > 21:21 4.05% 4.05% g_up 4 root -8 0 0K 12K - 0 > > 20:05 3.96% 3.96% g_down 381 root 4 0 1220K 740K - 3 > > 9:28 3.66% 3.66% nfsd 382 root 4 0 1220K 740K - 1 > > 10:13 3.47% 3.47% nfsd 385 root -1 0 1220K 740K nfsslp 3 > > 7:21 3.17% 3.17% nfsd 38 root -64 -183 0K 12K *Giant 0 > > 14:45 3.12% 3.12% irq24: amr0 > > 384 root 4 0 1220K 740K - 3 8:40 3.12% 3.12% nfsd > > 72 root -24 -143 0K 12K WAIT 2 16:50 2.98% 2.98% > > swi6:+ 383 root -8 0 1220K 740K biord 2 7:57 2.93% 2.93% > > nfsd 389 root 4 0 1220K 740K - 2 5:31 2.64% 2.64% > > nfsd 390 root -8 0 1220K 740K biord 3 5:54 2.59% 2.59% > > nfsd 387 root -8 0 1220K 740K biord 0 6:40 2.54% 2.54% > > nfsd 386 root -8 0 1220K 740K biord 1 6:22 2.44% 2.44% > > nfsd 392 root 4 0 1220K 740K - 3 4:27 2.10% 2.10% > > nfsd 388 root -4 0 1220K 740K *Giant 2 4:45 2.05% 2.05% > > nfsd 395 root 4 0 1220K 740K - 0 3:59 2.05% 2.05% > > nfsd 391 root 4 0 1220K 740K - 2 5:10 1.95% 1.95% > > nfsd 393 root 4 0 1220K 740K sbwait 1 4:13 1.56% 1.56% > > nfsd 398 root 4 0 1220K 740K - 2 3:31 1.56% 1.56% > > nfsd 399 root 4 0 1220K 740K - 3 3:12 1.56% 1.56% > > nfsd 401 root 4 0 1220K 740K - 1 2:57 1.51% 1.51% > > nfsd 403 root 4 0 1220K 740K - 0 3:04 1.42% 1.42% > > nfsd 406 root 4 0 1220K 740K - 1 2:27 1.37% 1.37% > > nfsd 397 root 4 0 1220K 740K - 3 3:16 1.27% 1.27% > > nfsd 396 root 4 0 1220K 740K - 2 3:42 1.22% 1.22% > > nfsd > > > > On Saturday 19 February 2005 04:23 am, Robert Watson wrote: > > > On Thu, 17 Feb 2005, David Rice wrote: > > > > Typicly we have 7 client boxes mounting storage from a single file > > > > server. Each client box servers 1000 web sites and associate email. > > > > We have done the basic NFS tuning (ie: Read write size optimization > > > > and kernel tuning) > > > > > > How many nfsd's are you running with? > > > > > > If you run systat -vmstat 1 on your server under high load, could you > > > send us the output? In particular, I'm interested in knowing how the > > > system is spending its time, the paging level, I/O throughput on > > > devices, and the systat -vmstat summary screen provides a good summary > > > of this and more. A few snapshots of "gstat" output would also be very > > > helpful. As would a snapshot or two of "top -S" output. This will > > > give us a picture of how the system is spending its time. > > > > > > > 2. Client boxes have high load averages and sometimes crashes due to > > > > slow NFS performance. > > > > > > Could you be more specific about the crash failure mode? > > > > > > > 3. File servers that randomly crash with "Fatal trap 12: page fault > > > > while in kernel mode" > > > > > > Could you make sure you're running with at least the latest 5.3 patch > > > level on the server, which includes some NFS server stability fixes, > > > and also look at sliding to the head of 5-STABLE? There are a number > > > of performance and stability improvements that may be relevant there. > > > > > > Could you provide serial console output of the full panic message, trap > > > details, compile the kernel with KDB+DDB, and include a full stack > > > trace? I'm happy to try to help debug these problems. > > > > > > > 4. With soft updates enabled during FSCK the fileserver will freeze > > > > with all NFS processs in the "snaplck" state. We disabled soft > > > > updates because of this. > > > > > > If it's possible to do get some more information, it would be quite > > > helpful. In particular, could you compile the server box with > > > DDB+KDB+BREAK_TO_DEBUGGER, breka into the serial debugger when it > > > appears wedged, and put the contents of "show lockedvnods", "ps", and > > > "trace " of any processes listed in "show lockedvnods" output, > > > that would be great. A crash dump would also be very helpful. For > > > some hints on the information that is necessary here, take a look at > > > the handbook chapter on kernel debugging and reporting kernel bugs, and > > > my recent post to current@ diagnosing a similar bug. > > > > > > If you e-enable soft updates but leave bgfsck disabled, does that > > > correct this stability problem? > > > > > > In any case, I'm happy to help try to figure out what's going on -- > > > some of the above information for stability and performance problems > > > would be quite helpful in tracking it down. > > > > > > Robert N M Watson > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org"