From owner-freebsd-arch@FreeBSD.ORG Tue Apr 20 21:39:53 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from green.homeunix.org (freefall.freebsd.org [216.136.204.21]) by hub.freebsd.org (Postfix) with ESMTP id 9058A16A4CE for ; Tue, 20 Apr 2004 21:39:53 -0700 (PDT) Received: from localhost (green@localhost [127.0.0.1]) by green.homeunix.org (8.12.11/8.12.11) with ESMTP id i3L4dqJg023358 for ; Wed, 21 Apr 2004 00:39:53 -0400 (EDT) (envelope-from green@green.homeunix.org) Message-Id: <200404210439.i3L4dqJg023358@green.homeunix.org> X-Mailer: exmh version 2.6.3 04/04/2003 with nmh-1.0.4 To: arch@FreeBSD.org From: Brian Fundakowski Feldman Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 21 Apr 2004 00:39:52 -0400 Sender: green@green.homeunix.org Subject: stable kqueue locking up and running on SMP X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Apr 2004 04:39:54 -0000 Please test out and provide feedback for the latest iteration: Things are now working in exactly the way I described I was going to implement them -- there is one kqueue lock, and if, as I suspect, it will _not_ be a point of contention, there is no reason at all to complicate things further. To do finer-grained locking, klists will have to be overhauled to become an actual, protected, object instead of hanging along with the object they represent. More complication in the form of read-locking (or holds) will also be necessary for that. For now, though, they belong to _kqueue_, and not struct selinfo or the object that contains them normally. A more complicated scheme for kqueues, knotes, klist(-alike)s would benefit greatly from divorce of kqueue from struct filedesc, if anyone is ever to tackle that (if there is ever a return on investment; MUTEX_PROFILING should help us figure that out.) The only known issue on my SMP box is this warning at boot time; I am unable to track down why witness(4) believes there to be a lock order reversal, but I welcome someone else to help identify this one: lock order reversal 1st 0xc0661500 global kqueue lock (global kqueue lock) @ kern/kern_event.c:413 2nd 0xc45bc438 filedesc structure (filedesc structure) @ kern/kern_event.c:422 Stack backtrace: backtrace(c0616b80,c45bc438,c060fd6e,c060fd6e,c0610941) at backtrace+0x17 witness_checkorder(c45bc438,9,c0610941,1a6,c455d594) at witness_checkorder+0x6f6 _mtx_lock_flags(c45bc438,0,c0610938,1a6,c455d594) at _mtx_lock_flags+0x9a kqueue(c45b73f0,db138d14,c19970ec,8133000,0) at kqueue+0x120 syscall(2f,2f,2f,8133000,2d) at syscall+0x272 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (362), eip = 0x282a7b8f, esp = 0xbfbfbcac, ebp = 0xbfbfc368 --- The major stress testers I know about are using make -j with -DUSE_KQUEUE compilation flags on make(1) and src/tools/regression/gaithstress. I notice no problems, but I haven't also tested to make sure nested kqueues are doing what is expected. The only missing feature is the ill-conceived NOTE_TRACK. I do not think that returning for that one note type EINVAL is a deal-breaker when trying to make kqueue not suck for 5.3; it is far more useful as a novelty than utility, especially considering I've never seen mention of it in actual software that uses kqueue. -- Brian Fundakowski Feldman \'[ FreeBSD ]''''''''''\ <> green@FreeBSD.org \ The Power to Serve! \ Opinions expressed are my own. \,,,,,,,,,,,,,,,,,,,,,,\