From owner-freebsd-current@FreeBSD.ORG Thu Nov 24 23:29:41 2005 Return-Path: X-Original-To: current@freebsd.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0D05316A41F; Thu, 24 Nov 2005 23:29:41 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id A99B443D55; Thu, 24 Nov 2005 23:29:40 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.13.4/8.13.4) with ESMTP id jAONTEEe055791; Thu, 24 Nov 2005 15:29:14 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.13.4/8.13.4/Submit) id jAONTEkr055790; Thu, 24 Nov 2005 15:29:14 -0800 (PST) Date: Thu, 24 Nov 2005 15:29:14 -0800 (PST) From: Matthew Dillon Message-Id: <200511242329.jAONTEkr055790@apollo.backplane.com> To: John Baldwin References: <200511231406.06282.jhb@freebsd.org> Cc: current@freebsd.org Subject: Re: nve locking fixes round 2 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2005 23:29:41 -0000 :Ok, now that the first set of locking overhaul is in the tree, can folks with :working nve(4) adapters test the patch referenced below and make sure there :are no regressions. Having the IFF_UP fiddling turned off may or may not :help folks getting the TX timeouts as well, btw, so if people are feeling :brave they can try this patch as well. Note it is only applicable to recent :current. : :http://www.FreeBSD.org/~jhb/patches/nve_locking.patch : :-- :John Baldwin <>< http://www.FreeBSD.org/~jhb/ :"Power Users Use the Power to Serve" = http://www.FreeBSD.org The reason I set sc->pending_txs to 0 in DFly after the reinit is because when a watchdog timeout occurs and you reset the device, *ALL* mbufs still sitting in the transmit ring are lost. They will never be acknowledged, ever. So pending_txs will never drop back to 0 on its own. This is what led to continuous watchdog timeout reports when, in fact, only one timeout actually occured. The FreeBSD code does set pending_txs to 0 in nve_stop(). I'm not sure this is correct, however, unless the pfnStop() ABI call cleans out pending mbufs in the transmit ring (which seems unlikely). The count would wind up going negative. Another problem that neither of us has dealt with yet is recovery of dead transmit mbufs. Right now that only occurs in nve_ospackettx(), but nve_ospackettx() is only called by the Nvidia code during normal operation. ABI calls to e.g. reset the Nvidia device will *NOT* clean out the transmit ring and call nve_ospackettx(), so we lose track of all the mbufs that were sitting in there at the time of a reinit. But, of course, the biggest problem is simply the fact that the NVidia ABI library seems to be rather broken. On my nForce4-based boxes the DFly driver can recover from numerous watchdog timeouts (and they occur quite often, even when the network load is virtually nil), but after an hour or two of testing at GiGE speeds the hardware itself stops working entirely, to the point where I have to physically unplug and replug the power cord for the machine for the hardware to start working again. -Matt Matthew Dillon