From owner-freebsd-questions@FreeBSD.ORG Thu Dec 14 00:39:50 2006 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AFA9216A4C2 for ; Thu, 14 Dec 2006 00:39:50 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from schitzo.solgatos.com (pool-71-245-104-192.ptldor.fios.verizon.net [71.245.104.192]) by mx1.FreeBSD.org (Postfix) with ESMTP id 523F143D4C for ; Thu, 14 Dec 2006 00:37:52 +0000 (GMT) (envelope-from freebsd@sopwith.solgatos.com) Received: from schitzo.solgatos.com (localhost.home.localnet [127.0.0.1]) by schitzo.solgatos.com (8.13.8/8.13.6) with ESMTP id kBE0dI2q030634 for ; Wed, 13 Dec 2006 16:39:18 -0800 Received: from sopwith.solgatos.com (uucp@localhost) by schitzo.solgatos.com (8.13.8/8.13.4/Submit) with UUCP id kBE0dIN3030631 for freebsd-questions@freebsd.org; Wed, 13 Dec 2006 16:39:18 -0800 Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id AAA14550; Thu, 14 Dec 2006 00:37:42 GMT Message-Id: <200612140037.AAA14550@sopwith.solgatos.com> To: freebsd-questions@freebsd.org In-reply-to: Your message of "Wed, 13 Dec 2006 18:27:26 EST." <20061213232726.GA61149@xor.obsecurity.org> Date: Wed, 13 Dec 2006 16:37:42 +0000 From: Dieter Subject: Re: processes not getting fair share of available disk I/O X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Dec 2006 00:39:50 -0000 > > Is Giant the only mutex/lock that could be a bottleneck across disks? > > The only one I can think of that is generic. One would have to do > more extensive profiling and diagnosis to try and figure out what is > wrong with your system. Suggestions of what to look at would be welcome. > The only explanation that seems to fit is that it's something to do > with your particular hardware (i.e. driver issue), since it's > certainly not a problem on general configurations. > > I know that many people have bad things to say about nforce chipsets, > although I dont know if your particular problem has been reported > before. Could APIC have anything to do with this? It is currently turned off in firmware. Today I experimented with vfs.hirunningspace. If I crank it up, I get better total write speed with multiple drives doing dd from /dev/zero to files on disks. But it doesn't help my real applications, and in fact appears to hurt them. I have also discovered that my program that reads from Ethernet and does non-blocking writes to stdout which get redirected to a disk file isn't benefiting as much as it should from the large circular buffer. The buffer should be enough to hold 4.5-5 minutes worth of data, and is mlock(2)'d into memory. I can read from four disks at once without hurting the Ethernet transfer. But a dd from /dev/zero writing to a different disk that takes less than 3 seconds interferes with the Ethernet transfer. The fcntl(1, O_NONBLOCK); isn't failing, and the write() isn't returning -1/EAGAIN.