From owner-freebsd-current@FreeBSD.ORG Thu Dec 1 18:36:50 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 50FEF1065672; Thu, 1 Dec 2011 18:36:50 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id EDAAF8FC0A; Thu, 1 Dec 2011 18:36:49 +0000 (UTC) Received: by ggnk5 with SMTP id k5so3257903ggn.13 for ; Thu, 01 Dec 2011 10:36:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=D/XKkmjhtv3z1kxTU+FV3Kf0oGQUGEMQx4tKZwNxNJk=; b=D5pADUlac3NRPsufKFUSwqAlK2xY5Alvk0waTOEmIM7gXs94EBu3SfiGpmH77yG1Wn DECvvVQRaG+T2hWdNDlpLHgkZb3OsckIiBaS6zTM8A3PTb11gn8bgHUPKQ7KU82vvXKI Zy6cd/ATcF3pxP+RzVp5+GO2gC878eQ0Lk5IQ= Received: by 10.50.6.202 with SMTP id d10mr9389697iga.31.1322762777488; Thu, 01 Dec 2011 10:06:17 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id l28sm23994451ibc.3.2011.12.01.10.06.11 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 01 Dec 2011 10:06:13 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Thu, 01 Dec 2011 10:04:45 -0800 From: YongHyeon PYUN Date: Thu, 1 Dec 2011 10:04:45 -0800 To: Luigi Rizzo Message-ID: <20111201180445.GA12942@michelle.cdnetworks.com> References: <4ED7A0D8.90801@freebsd.org> <20111201165444.GA97698@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111201165444.GA97698@onelab2.iet.unipi.it> User-Agent: Mutt/1.4.2.3i Cc: Rene Ladan , current@freebsd.org Subject: Re: lock order reversals with netmap X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 18:36:50 -0000 On Thu, Dec 01, 2011 at 05:54:44PM +0100, Luigi Rizzo wrote: > On Thu, Dec 01, 2011 at 04:44:24PM +0100, Rene Ladan wrote: > > Hi, > > > > on FreeBSD 10.0-CURRENT #7 r228176M: Thu Dec 1 13:56:02 CET 2011 > > (GENERIC + CAPABILITIES + netmap with head.diff and bge patches applied) > > I get these lock order reversals when running a netmap-enabled program > > (details in the attachment) with syscall (54, FreeBSD ELF64, sys_ioctl): > > Rene, > thanks for the report. > > As i mentioned earlier to Rene, the 'bge' driver > is neither complete nor tested so i am even surprised > that it does not crash right away. I'll keep his report > in mind when we will complete the support for bge. > > BTW is someone is familiar with the architecture of the 'bge' NICs > please can she/he contact me. I am unclear on why there are two > lists of rx buffers (std and jumbo) and one ring -- perhaps the > NIC first receives the frame in its fifo and then decides which > type of buffer to use to store it ? > Actually there are three rings but the additional mini ring is only available for BCM5700. Controller determines which ring(mini, standard and jumbo) would be used to receive the frame based on the frame size. For example, if jumbo frame is enabled and controller receives a pure TCP ACK, controller will use standard RX ring, mini RX ring on BCM5700, which in turn can save system resources. Controller maintains pool of TX/RX buffers in NIC's internal memory space(2 MIPS processors in NIC) and all these decision is made by firmware of the NIC with the help of driver. Broadcom provides publicly available data sheet for open source developers. See the following URL. http://www.broadcom.com/support/ethernet_nic/open_source.php Having two RX buffers are common for controllers that support header splitting. igb(4) and ti(4) have the feature but I think that feature was disabled in igb(4) due to bugs or incomplete implementation in driver. > cheers > luigi > > > Dec 1 16:23:09 acer kernel: exclusive sleep mutex netmap memory > > allocator lock (netmap memory allocator lock) r = 0 (0xfffffe00027d1880) > > locked @ /usr/src/sys/dev/netmap/netmap.c:1484 > > > > Dec 1 16:23:09 acer kernel: exclusive sleep mutex bge0 (network driver) > > r = 0 (0xffffff8000768010) locked @ > > /usr/src/sys/dev/netmap/if_bge_netmap.h:60 > > > > The application does not invoke the offending function (netmap_malloc()) > > itself. > > > > Regards, > > Ren? > > -- > > http://www.rene-ladan.nl:8080/ > > > > GPG fingerprint = ADBC ECCD EB5F A6B4 549F 600D 8C9E 647A E564 2BFC > > (subkeys.pgp.net) > > > Dec 1 15:41:20 acer kernel: FreeBSD 10.0-CURRENT #7 r228176M: Thu Dec 1 13:56:02 CET 2011 > > Dec 1 15:41:20 acer kernel: real memory = 4294967296 (4096 MB) > > Dec 1 15:41:20 acer kernel: avail memory = 4080091136 (3891 MB) > > Dec 1 15:41:20 acer kernel: 001.000005 netmap_memory_init [1627] netmap_buffer_base 0xffffff8117eaa000 (offset 679936) > > Dec 1 15:41:20 acer kernel: 001.000006 netmap_memory_init [1636] Have 129 MB, use 661KB for rings, 65862 buffers at 0xffffff8117eaa000 > > Dec 1 15:41:20 acer kernel: netmap: loaded module with 129 Mbytes > > Dec 1 15:41:20 acer kernel: bge0: mem 0xf5100000-0xf510ffff irq 16 at device 0.0 on pci2 > > Dec 1 15:41:20 acer kernel: bge0: CHIP ID 0x05784100; ASIC REV 0x5784; CHIP REV 0x57841; PCI-E > > Dec 1 15:41:20 acer kernel: miibus0: on bge0 > > Dec 1 15:41:20 acer kernel: brgphy0: PHY 1 on miibus0 > > Dec 1 15:41:20 acer kernel: brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > > Dec 1 15:41:20 acer kernel: bge0: Ethernet address: 00:26:2d:5e:d8:ee > > Dec 1 15:41:20 acer kernel: 001.000009 netmap_attach [1243] ok for bge0 > > Dec 1 16:23:09 acer kernel: 989.882634 netmap_set_ringid [779] ringid bge0 set to SW RING > > Dec 1 16:23:09 acer kernel: uma_zalloc_arg: zone "64" with the following non-sleepable locks held: > > Dec 1 16:23:09 acer kernel: exclusive sleep mutex netmap memory allocator lock (netmap memory allocator lock) r = 0 (0xfffffe00027d1880) locked @ /usr/src/sys/dev/netmap/netmap.c:1484 > > Dec 1 16:23:09 acer kernel: exclusive sleep mutex bge0 (network driver) r = 0 (0xffffff8000768010) locked @ /usr/src/sys/dev/netmap/if_bge_netmap.h:60 > > Dec 1 16:23:09 acer kernel: KDB: stack backtrace: > > Dec 1 16:23:09 acer kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > Dec 1 16:23:10 acer kernel: kdb_backtrace() at kdb_backtrace+0x37 > > Dec 1 16:23:10 acer kernel: _witness_debugger() at _witness_debugger+0x2c > > Dec 1 16:23:10 acer kernel: witness_warn() at witness_warn+0x2c2 > > Dec 1 16:23:10 acer kernel: uma_zalloc_arg() at uma_zalloc_arg+0x335 > > Dec 1 16:23:10 acer kernel: malloc() at malloc+0xbe > > Dec 1 16:23:10 acer kernel: netmap_malloc() at netmap_malloc+0x86 > > Dec 1 16:23:10 acer kernel: netmap_ioctl() at netmap_ioctl+0x5bd > > Dec 1 16:23:10 acer kernel: devfs_ioctl_f() at devfs_ioctl_f+0x7a > > Dec 1 16:23:10 acer kernel: kern_ioctl() at kern_ioctl+0xcd > > Dec 1 16:23:10 acer kernel: sys_ioctl() at sys_ioctl+0xfd > > Dec 1 16:23:10 acer kernel: amd64_syscall() at amd64_syscall+0x3ac > > Dec 1 16:23:10 acer kernel: Xfast_syscall() at Xfast_syscall+0xf7 > > Dec 1 16:23:10 acer kernel: --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8022aef0c, rsp = 0x7fffffffd4b8, rbp = 0x802bfb100 --- > > Dec 1 16:23:10 acer kernel: uma_zalloc_arg: zone "64" with the following non-sleepable locks held: > > Dec 1 16:23:10 acer kernel: exclusive sleep mutex netmap memory allocator lock (netmap memory allocator lock) r = 0 (0xfffffe00027d1880) locked @ /usr/src/sys/dev/netmap/netmap.c:1484 > > Dec 1 16:23:10 acer kernel: exclusive sleep mutex bge0 (network driver) r = 0 (0xffffff8000768010) locked @ /usr/src/sys/dev/netmap/if_bge_netmap.h:60 > > Dec 1 16:23:10 acer kernel: KDB: stack backtrace: > > Dec 1 16:23:10 acer kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > Dec 1 16:23:10 acer kernel: kdb_backtrace() at kdb_backtrace+0x37 > > Dec 1 16:23:10 acer kernel: _witness_debugger() at _witness_debugger+0x2c > > Dec 1 16:23:10 acer kernel: witness_warn() at witness_warn+0x2c2 > > Dec 1 16:23:10 acer kernel: uma_zalloc_arg() at uma_zalloc_arg+0x335 > > Dec 1 16:23:10 acer kernel: malloc() at malloc+0xbe > > Dec 1 16:23:10 acer kernel: netmap_malloc() at netmap_malloc+0x86 > > Dec 1 16:23:10 acer kernel: netmap_ioctl() at netmap_ioctl+0x817 > > Dec 1 16:23:10 acer kernel: devfs_ioctl_f() at devfs_ioctl_f+0x7a > > Dec 1 16:23:10 acer kernel: kern_ioctl() at kern_ioctl+0xcd > > Dec 1 16:23:10 acer kernel: sys_ioctl() at sys_ioctl+0xfd > > Dec 1 16:23:10 acer kernel: amd64_syscall() at amd64_syscall+0x3ac > > Dec 1 16:23:10 acer kernel: Xfast_syscall() at Xfast_syscall+0xf7 > > Dec 1 16:23:10 acer kernel: --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8022aef0c, rsp = 0x7fffffffd4b8, rbp = 0x802bfb100 --- > > Dec 1 16:23:10 acer kernel: 990.041760 netmap_reset [1403] +++ NR_REINIT ok on bge0 RX[0] > > Dec 1 16:23:10 acer kernel: 990.041948 netmap_reset [1384] +++ NR_REINIT ok on bge0 TX[0] > > Dec 1 16:23:10 acer kernel: bge0: link state changed to DOWN > > Dec 1 16:23:10 acer kernel: 990.051589 netmap_dtor [352] deleting last netmap instance for bge0 > > Dec 1 16:23:12 acer kernel: bge0: link state changed to UP > > > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"