From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 16:31:38 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 968DA16A421 for ; Wed, 17 Oct 2007 16:31:38 +0000 (UTC) (envelope-from dclements@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.244]) by mx1.freebsd.org (Postfix) with ESMTP id 4B07F13C48D for ; Wed, 17 Oct 2007 16:31:38 +0000 (UTC) (envelope-from dclements@gmail.com) Received: by an-out-0708.google.com with SMTP id c24so225735ana for ; Wed, 17 Oct 2007 09:31:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=dSuwU9oPDJmUcmbzyl4NjqaduYgzMDu0M3xrzpjWfWk=; b=bW/VD0UHi5OVl2FuG/6M4oSRV7cpTUFk0BxN6khUt2VJvB6BCkNNGeY3a9ls0cv3ijU7+gTLGqHlKjels4ejGlB1ZjsJOrf52cTPGP6FkhRAaO3uuBUEeIIQRsEAC56+INkwHMBvFzkx7v/CRN6SEWrpEjy+1F4UqETrrhd9cR0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=hxZd+bEGjPb6R25nQOo4snc5eXMpEbelLUTr2yu4UVAYkvp/kViDQqpxeTTbXA4Efj4bPgN3/XVOBfOpruBzvCwoiOdzIb29Gxk6ScBvfkieqRClmlPYcx42GdDmmHHHixFGK15wx7NSfxu9OMCRHpRQFP5i6oOPaadL/YbInew= Received: by 10.142.148.7 with SMTP id v7mr2615790wfd.1192638695927; Wed, 17 Oct 2007 09:31:35 -0700 (PDT) Received: by 10.142.81.19 with HTTP; Wed, 17 Oct 2007 09:31:35 -0700 (PDT) Message-ID: <54da514e0710170931r1b558bedi69e1bba50807f465@mail.gmail.com> Date: Wed, 17 Oct 2007 09:31:35 -0700 From: "Doug Clements" To: freebsd-hackers@freebsd.org In-Reply-To: <4715F1A5.50605@freebsd.org> MIME-Version: 1.0 References: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> <4715F1A5.50605@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Interrupt/speed problems with 6.2 NFS server X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 16:31:38 -0000 > > Can you also send the output of ps -auxl? > > Also - do you notice this performance drop when running something like > one of the network performance tools? I'd like to isolate the disk > activity from the network activity for a clean test.. > I tested this with iperf, and while I did see some nfs performance degradation, it did not bog the machine or delay the terminal in the same way. NFS requests were still processed in an acceptable fashion. It was still responsive to commands and ran processes just fine. I would have to say it performed as to be expected during iperf tests (on which I got about 85mbit/sec, which is also to be expected). Interrupts went down to about 2000/sec on em, but the machine did not hang. Something I've noticed is that running 'systat -vm' seems to be part of the problem. If I run the file copy by itself with rdist, it's fast and runs ok. If I run it with systat -vm going, this is when the interrupts jump way up and the machine starts to delay badly. Noticing this, I tried running 'sysctl -a' during the file copy, thinking there was some problem with polling the kernel for certain statistics. Sure enough, sysctl -a delays at 2 spots. Once right after "kern.random.sys.harvest.swi: 0" and once again after "debug.hashstat.nchash: 131072 61777 6 4713". While it is delayed here (for a couple seconds for each one) the machine is totally hung. Maybe this is a statistics polling issue? Maybe the machine is delayed just long enough in systat -vm to make the nfs clients retry, causing a storm of interrupts? Other systat modes do not seem to cause the same problem (pigs, icmp, ifstat). I do not think the ps or systat output is very accurate, since I can't get them to run when the machine is hung up. I type in the command, but it does not run until the machine springs back to life. I'm not sure how this will affect measurements. http://toric.loungenet.org/~doug/sysctl-a http://toric.loungenet.org/~doug/psauxl http://toric.loungenet.org/~doug/systat-vm My real confusion lies in why there are still em interrupts at all, with polling on. Thanks! --Doug