From owner-freebsd-smp@FreeBSD.ORG Wed Mar 12 20:01:33 2008 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C63F1065689 for ; Wed, 12 Mar 2008 20:01:33 +0000 (UTC) (envelope-from daniel@dgnetwork.com.br) Received: from mail.mastercabo.com.br (mail.mastercabo.com.br [200.179.179.14]) by mx1.freebsd.org (Postfix) with SMTP id AC1278FC22 for ; Wed, 12 Mar 2008 20:01:32 +0000 (UTC) (envelope-from daniel@dgnetwork.com.br) Received: (qmail 38372 invoked by uid 1008); 12 Mar 2008 20:01:30 -0000 X-Spam-Checker-Version: SpamAssassin 3.1.6-unknown (2006-10-03) on srvmail2 X-Spam-Level: X-Spam-Status: No, score=-1.7 required=4.7 tests=AWL,BAYES_00 autolearn=ham version=3.1.6-unknown Received: from unknown (HELO ?10.0.0.10?) (daniel@dgnetwork.com.br@200.243.216.36) by mail.mastercabo.com.br with SMTP; 12 Mar 2008 20:01:25 -0000 Message-ID: <47D834AE.8080301@dgnetwork.com.br> Date: Wed, 12 Mar 2008 16:53:18 -0300 From: =?ISO-8859-1?Q?Daniel_Dias_Gon=E7alves?= Organization: DGNET Network Solutions User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) MIME-Version: 1.0 To: freebsd-bugs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, freebsd-smp@freebsd.org Subject: FreeBSD 6.3 fxp0 MBUF and PAE X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: daniel@dgnetwork.com.br List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Mar 2008 20:01:33 -0000 Hi, When using the interface fxp0 with PAE enable in kernel, occurs the following error: fxp0: can't map mbuf (error 12) ... it repeats, repeats and lost communication. Information: 6.3-RELEASE fxp0@pci14:4:0: class=0x020000 card=0x00708086 chip=0x12298086 rev=0x10 hdr=0x00 vendor = 'Intel Corporation' device = '82550/1/7/8/9 EtherExpress PRO/100(B) Ethernet Adapter' class = network subclass = ethernet I wait reply. Thanks. Daniel From owner-freebsd-smp@FreeBSD.ORG Thu Mar 13 01:55:04 2008 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B44241065671 for ; Thu, 13 Mar 2008 01:55:04 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.238]) by mx1.freebsd.org (Postfix) with ESMTP id 725ED8FC13 for ; Thu, 13 Mar 2008 01:55:04 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by wr-out-0506.google.com with SMTP id c49so2612082wra.19 for ; Wed, 12 Mar 2008 18:55:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received:date:from:to:cc:subject:message-id:reply-to:references:mime-version:content-type:content-disposition:in-reply-to:user-agent; bh=cPnyEOp1Rzzd/jS2ttQoEijwEb0ocGvnwZEdexBB+H4=; b=nFq/31rRrJTOF1VZGtbnOzAxEgy/9rvAaeiWQ81zFJv9VuE6NdokoVZ9e4ujTNZjXHg9t0aBqJcoAlUGweWi/0llbTv7liJJUd9HyugX9kMPzJz+p/oxtmNEufC4JabmVW4zvxqXoSjkSsRe7twzskXhORJTUBJnByowFd4+1WU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:reply-to:references:mime-version:content-type:content-disposition:in-reply-to:user-agent; b=bAqgzQw+djkzZDBafAihEMA9oIMITJnaSzYA7qpgHPsUn70z0OFhv+BneC3i8v9X5Ie/Ds86i31Xwkzw6UwFPz6CRtjwEgdNQRsTgvK6po704ylxiwtyaQjFY9kg5K0OXcCqw0+WGaFsOXQKpbU1YpL6USakgvsZJtI8lDaI+/A= Received: by 10.150.203.8 with SMTP id a8mr4982783ybg.56.1205371672225; Wed, 12 Mar 2008 18:27:52 -0700 (PDT) Received: from michelle.cdnetworks.co.kr ( [211.53.35.84]) by mx.google.com with ESMTPS id 35sm17258713wra.37.2008.03.12.18.27.48 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 12 Mar 2008 18:27:50 -0700 (PDT) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id m2D1RjIm017355 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 13 Mar 2008 10:27:45 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id m2D1Rflt017354; Thu, 13 Mar 2008 10:27:41 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Thu, 13 Mar 2008 10:27:41 +0900 From: Pyun YongHyeon To: Daniel Dias Gon?alves Message-ID: <20080313012741.GC16972@cdnetworks.co.kr> References: <47D834AE.8080301@dgnetwork.com.br> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47D834AE.8080301@dgnetwork.com.br> User-Agent: Mutt/1.4.2.1i Cc: freebsd-net@freebsd.org, freebsd-bugs@freebsd.org, freebsd-smp@freebsd.org Subject: Re: FreeBSD 6.3 fxp0 MBUF and PAE X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Mar 2008 01:55:04 -0000 On Wed, Mar 12, 2008 at 04:53:18PM -0300, Daniel Dias Gon?alves wrote: > Hi, > > When using the interface fxp0 with PAE enable in kernel, occurs the > following error: > > fxp0: can't map mbuf (error 12) > ... > > it repeats, repeats and lost communication. > error 12 means ENOMEM. bus_dmamap_load_mbuf_sg(9) failed due to insuffcient resources. I guess there is no way to overcome this situation in driver. The only remaining way I can think of would be reclaiming of transmitted frames but how well it works would depends on circumstances. Personally I don't see a reason to print these ENOMEM errors for production box without late limiting. > Information: > 6.3-RELEASE > > fxp0@pci14:4:0: class=0x020000 card=0x00708086 chip=0x12298086 rev=0x10 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82550/1/7/8/9 EtherExpress PRO/100(B) Ethernet Adapter' > class = network > subclass = ethernet > > I wait reply. > > Thanks. > > Daniel -- Regards, Pyun YongHyeon From owner-freebsd-smp@FreeBSD.ORG Sat Mar 15 03:01:41 2008 Return-Path: Delivered-To: smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B366F106567A; Sat, 15 Mar 2008 03:01:41 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 98B1B8FC16; Sat, 15 Mar 2008 03:01:41 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id ECF0C1A4D82; Fri, 14 Mar 2008 19:41:14 -0700 (PDT) Date: Fri, 14 Mar 2008 19:41:14 -0700 From: Alfred Perlstein To: stable@freebsd.org Message-ID: <20080315024114.GD67856@elvis.mu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="F8dlzb82+Fcn6AgP" Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: smp@freebsd.org Subject: timeout/untimeout race conditions/crash [patch] X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Mar 2008 03:01:41 -0000 --F8dlzb82+Fcn6AgP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline We think we tracked down a defect in timeout/untimeout in FreeBSD. We have reduced the problem to the following scenario: 2+ cpu system, one cpu is running softclock at the same time another thread is running on another cpu which makes use of timeout/untimeout. CPU 0 is running "softclock" CPU 1 is running "driver" with Giant held. softclock: mtx_lock_spin(&callout_lock) softclock: CACHES the callout structure's fields. softclock: sees that it's a CALLOUT_LOCAL_ALLOC softclock: executes this code: if (c->c_flags & CALLOUT_LOCAL_ALLOC) { c->c_func = NULL; c->c_flags = CALLOUT_LOCAL_ALLOC; SLIST_INSERT_HEAD(&callfree, c, c_links.sle); curr_callout = NULL; } else { NOTE: that c->c_func has been set to NULL and curr_callout is also NULL. softclock: mtx_unlock_spin(&callout_lock) driver: calls untimeout(), the following sequence happens: mtx_lock_spin(&callout_lock); if (handle.callout->c_func == ftn && handle.callout->c_arg == arg) callout_stop(handle.callout); mtx_unlock_spin(&callout_lock); NOTE: untimeout() sees that handle.callout->c_func is not set to the function so it does NOT call callout_stop(9)! driver: free's backing structure for c->c_arg. softclock: executes callout. softclock: likely crashes at this point due to access after free. I have a patch I'm trying out here, but I need feedback on it. The way the patch works is to treat CALLOUT_LOCAL_ALLOC (timeout/untimeout) callouts the same as ~CALLOUT_LOCAL_ALLOC allocs, and moves the freelist manipulation to the end of the callout dispatch. Some light testing seems to have the system work. We are doing some testing in-house to also make sure this works. Please provide feedback. See attached delta. -- - Alfred Perlstein --F8dlzb82+Fcn6AgP Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="kern_timeout.diff" Index: kern_timeout.c =================================================================== RCS file: /cvs/ncvs/src/sys/kern/kern_timeout.c,v retrieving revision 1.97.2.2 diff -u -r1.97.2.2 kern_timeout.c --- kern_timeout.c 26 Sep 2005 19:49:12 -0000 1.97.2.2 +++ kern_timeout.c 15 Mar 2008 02:28:48 -0000 @@ -241,17 +241,8 @@ c_arg = c->c_arg; c_mtx = c->c_mtx; c_flags = c->c_flags; - if (c->c_flags & CALLOUT_LOCAL_ALLOC) { - c->c_func = NULL; - c->c_flags = CALLOUT_LOCAL_ALLOC; - SLIST_INSERT_HEAD(&callfree, c, - c_links.sle); - curr_callout = NULL; - } else { - c->c_flags = - (c->c_flags & ~CALLOUT_PENDING); - curr_callout = c; - } + c->c_flags &= ~CALLOUT_PENDING; + curr_callout = c; curr_cancelled = 0; mtx_unlock_spin(&callout_lock); if (c_mtx != NULL) { @@ -310,6 +301,12 @@ mtx_unlock(c_mtx); mtx_lock_spin(&callout_lock); done_locked: + if (c->c_flags & CALLOUT_LOCAL_ALLOC) { + c->c_func = NULL; + c->c_flags = CALLOUT_LOCAL_ALLOC; + SLIST_INSERT_HEAD(&callfree, c, + c_links.sle); + } curr_callout = NULL; if (wakeup_needed) { /* --F8dlzb82+Fcn6AgP--