From owner-freebsd-current@FreeBSD.ORG Thu Mar 23 14:53:28 2006 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A84A516A420 for ; Thu, 23 Mar 2006 14:53:28 +0000 (UTC) (envelope-from dave@randomparity.com) Received: from fed1rmmtao04.cox.net (fed1rmmtao04.cox.net [68.230.241.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8514B43D55 for ; Thu, 23 Mar 2006 14:53:26 +0000 (GMT) (envelope-from dave@randomparity.com) Received: from gateway.randomparity.com ([68.4.169.57]) by fed1rmmtao04.cox.net (InterMail vM.6.01.05.02 201-2131-123-102-20050715) with ESMTP id <20060323145324.YRFS17690.fed1rmmtao04.cox.net@gateway.randomparity.com>; Thu, 23 Mar 2006 09:53:24 -0500 Received: by gateway.randomparity.com (Postfix, from userid 429) id 1ADCF31BC57; Thu, 23 Mar 2006 00:28:16 -0800 (PST) Received: from snoopy.randomparity.com (snoopy.randomparity.com [10.1.1.4]) by gateway.randomparity.com (Postfix) with ESMTP id 7C1B631BC3A; Thu, 23 Mar 2006 00:28:14 -0800 (PST) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Thu, 23 Mar 2006 06:53:12 -0800 X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Message-ID: <22C21BFBEB52B340A1F422CB0D88F5872E0B@snoopy.randomparity.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Witness finds "malloc(M_WAITOK) with non-sleepable lock held" in FreeBSD 7.0-CURRENT (amd64) Thread-Index: AcZOSqFXjdYzhNTFRlSucI6WfjQZyQAAjJCA From: "David Christensen" To: "Scott Long" , "John-Mark Gurney" X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on gateway.randomparity.com X-Spam-Level: X-Spam-Status: No, score=-1.7 required=4.0 tests=AWL,BAYES_00, DATE_IN_FUTURE_06_12 autolearn=no version=3.0.4 Cc: freebsd-current@freebsd.org Subject: RE: Witness finds "malloc(M_WAITOK) with non-sleepable lock held" in FreeBSD 7.0-CURRENT (amd64) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2006 14:53:28 -0000 Actually I was following the example in sys/dev/if_em.c. The call chain is: bus_dma_tag_create() is called from em_allocate_receive_structures() is called from em_setup_receive_structures() is called from em_init_locked() The em driver doesn't release its lock before calling bus_dma_tag_create() and it definitely does it outside of the attach routine. Is the em driver also FUBAR or is there something else going on? David Christensen -----Original Message----- From: Scott Long [mailto:scottl@samsco.org]=20 Sent: Wednesday, March 22, 2006 11:23 PM To: John-Mark Gurney Cc: David Christensen; freebsd-current@freebsd.org Subject: Re: Witness finds "malloc(M_WAITOK) with non-sleepable lock held" in FreeBSD 7.0-CURRENT (amd64) John-Mark Gurney wrote: > David Christensen wrote this message on Wed, Mar 22, 2006 at 21:55 -0800: >=20 >>I'm developing an Ethernet driver with FreeBSD 7.0-CURRENT (amd64) and >>I'm=20 >>receiving many of the following witness errors: >> >>malloc(M_WAITOK) of "128", forcing M_NOWAIT with the following >>non-sleepable locks held: >>exclusive sleep mutex bce0 (network driver) r =3D 0 = (0xffffffff8111e068) >>locked @ if_bce.c:4607 >>KDB: stack backtrace: >>kdb_backtrace() at kdb_backtrace+0x37 >>witness_warn() at witness_warn+0x2c1 >>uma_zalloc_arg() at uma_zalloc_arg+0x69 >>malloc() at malloc+0xf5 >>sysctl_add_oid() at sysctl_add_oid+0xa9 >>alloc_bounce_zone() at alloc_bounce_zone+0x16b >>bus_dma_tag_create() at bus_dma_tag_create+0x1ea >>bce_init_rx_chain() at bce_init_rx_chain+0x8e >>bce_init_locked() at bce_init_locked+0x1e2 >>bce_init() at bce_init+0x39 >>ether_ioctl() at ether_ioctl+0x87 >>bce_ioctl() at bce_ioctl+0x48e >>in6_ifinit() at in6_ifinit+0xbd >>in6_update_ifa() at in6_update_ifa+0x563 >>in6_ifattach_linklocal() at in6_ifattach_linklocal+0x126 >>in6_ifattach() at in6_ifattach+0xdf >>in6_if_up() at in6_if_up+0x59 >>if_route() at if_route+0x8a >>if_up() at if_up+0x13 >>ifhwioctl() at ifhwioctl+0x2f4 >>ifioctl() at ifioctl+0x10b >>soo_ioctl() at soo_ioctl+0x38c >>ioctl() at ioctl+0x436 >>syscall() at syscall+0x350 >>Xfast_syscall() at Xfast_syscall+0xa8 >>--- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x8008219ac, rsp =3D >>0x7fffffffe6b8, rbp =3D 0x1 --- >> >>The bus_dma_tag_create looks like this: >> >>bus_dma_tag_create( >> sc->parent_tag, /* parent */ >> 4096, /* alignment */ >> 0, /* boundary */ >> BUS_SPACE_MAXADDR, /* lowaddr */ >> BUS_SPACE_MAX_ADDR, /* lowaddr */ >> NULL, /* filter */ >> NULL, /* filterarg */ >> 4096, /* maxsize */ >> 1, /* nsegments */ >> 4096, /* maxsegsize */ >> BUS_DMA_ALLOCNOW, /* flags */ >> NULL, /* lockfunc */ >> NULL, /* lockarg */ >> &sc->rx_bd_chain_tag)); >> >>Am I doing something wrong? The function bce_init_rx_chain is called >>from with >>a lock but isn't that normal? >=20 >=20 > Yeh, you have to unlock your driver lock before calling > bus_dma_tag_create.. If you look at the other ethernet drivers, some > call _tag_create as part of attach, not in _init... at this point, > it's safe to release your lock and allocate memory... >=20 In fact, it's really bad to be initializing the rx data structures like=20 this in if_init. It should be done in dev_attach. The reason is that if_init can be called at any time and will almost certainly be called multiple times. Also, do not use the BUS_DMA_ALLOCNOW flag here, as I assume that you are trying to use the busdma tag to allocate a static piece of memory for the rx chain/ring. The flag should only be used for flags that deal with dynamic buffers like mbufs and bio_data objects, or memory that has been allocated in the kernel with normal malloc. Scott