From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 13 13:11:54 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9A30A106564A for ; Tue, 13 Dec 2011 13:11:54 +0000 (UTC) (envelope-from monthadar@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6B34A8FC1D for ; Tue, 13 Dec 2011 13:11:54 +0000 (UTC) Received: by iakl21 with SMTP id l21so7098626iak.13 for ; Tue, 13 Dec 2011 05:11:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=023aYy6Liv+w3G677lQVoO0m7pIqXbW49x9RcPM8d3o=; b=SkE/w6MeGqJ4QkPH40Qdp0/YtbqOQ5Ibz8bW8xrlQfXHJxBkkkPKkDdgll8NisnO1f RNB0BJizgl8V0uEtIlCuPuwRpJgdFVH9C+yyXRMonst3bAtXy7zb+5p4Mb8eSesTfzlH 9WaxyE3pZfy6aBHwVja6vul1kKASMxCf0r9DY= MIME-Version: 1.0 Received: by 10.50.220.164 with SMTP id px4mr19442153igc.8.1323780394366; Tue, 13 Dec 2011 04:46:34 -0800 (PST) Received: by 10.50.51.233 with HTTP; Tue, 13 Dec 2011 04:46:34 -0800 (PST) Date: Tue, 13 Dec 2011 13:46:34 +0100 Message-ID: From: Monthadar Al Jaberi To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: loop inside uma_zfree critical section X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 13:11:54 -0000 Hi, I am not sure why I am having this problem, but looking at the code I dont understand uma_core.c really good. So I hope someone can shed a light on this: I am running on an arm board and and running a kernel module that behaves like a wlan interface. so I tx and rx packets. For now tx is only only sending beacon like frames. This is done through using ieee80211_beacon_alloc(). Then in a callout task to generate periodic beacons: m_dup(avp->beacon, M_DONTWAIT); mtx_lock(...); STAILQ_INSERT_TAIL(...); taskqueue_enqueue(...); mtx_unlock(...); callout_schedule(...); On the RX side, the interrupt handler will read out buffer then place it on a queue to be handled by wlan-glue code. For now wlan-glue code just frees the mbuf it instead of calling net80211 ieee80211_input() functions: m_copyback(...); /* Allocate new mbuf for next RX. */ MGETHDR(..., M_DONTWAIT, MT_DATA); bzero((mtod(sc->Rx_m, void *)), MHLEN); sc->Rx_m->m_len = 0; /* NB: m_gethdr does not set */ sc->Rx_m->m_data += 20; /* make headroom */ Then I use a lockmgr inside my kernel module that should make sure that we either are on TX or RX path. Here is an output of when it loops forever and a debug output. I enabled some of the printfs in vm/uma_core.c: KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #931: Tue Dec 13 11:35:17 UTC 2011 ... beacon_callout. becon m_dup: new 0xc0d99c00, beacon: 0xc0eb0800. queue beacon for TX. LOCKMGR: TX path locked sending... Free TX mbuf: 0xc0d99c00 LOCKMGR: TX path unlocked LOCKMGR: RX path locked current RX mbuf: 0xc0d99900 queue packet for wlan-glue allocating new Rx mbuf: 0xc0d99c00 LOCKMGR: RX path unlocked wlan-glue task running... wlan-glue code freeing mbuf: 0xc0d99900 beacon_callout. becon m_dup: new 0xc0d99b00, beacon: 0xc0eb0800. queue beacon for TX. LOCKMGR: TX path locked sending... Free TX mbuf: 0xc0d99b00 LOCKMGR: TX path unlocked LOCKMGR: RX path locked current RX mbuf: 0xc0d99c00 queue packet for wlan-glue allocating new Rx mbuf: 0xc0d99b00 [1] LOCKMGR: RX path unlocked wlan-glue task running... uma_zfree: Swapping buckets. [2] wlan-glue code freeing mbuf: 0xc0d99c00 beacon_callout. becon m_dup: new 0xc0d99b00, beacon: 0xc0eb0800. [3] queue beacon for TX. LOCKMGR: TX path locked sending... Free TX mbuf: 0xc0d99b00 [4] LOCKMGR: TX path unlocked LOCKMGR: RX path locked current RX mbuf: 0xc0d99b00 queue packet for wlan-glue allocating new Rx mbuf: 0xc0d99900 LOCKMGR: RX path unlocked wlan-glue task running... [5] uma_zfree: Swapping buckets. uma_zfree: Swapping buckets. uma_zfree: Putting old bucket on the free list. uma_zfree: Allocating new free bucket. uma_zfree: Swapping buckets. uma_zfree: Putting old bucket on the free list. uma_zfree: Allocating new free bucket. uma_zfree: Swapping buckets. uma_zfree: Putting old bucket on the free list. uma_zfree: Allocating new free bucket. uma_zfree: Swapping buckets. uma_zfree: Putting old bucket on the free list. uma_zfree: Allocating new free bucket. uma_zfree: Swapping buckets. uma_zfree: Putting old bucket on the free list. uma_zfree: Allocating new free bucket. uma_zfree: Swapping buckets. uma_zfree: Putting old bucket on the free list. uma_zfree: Allocating new free bucket. uma_zfree: Swapping buckets. uma_zfree: Putting old bucket on the free list. uma_zfree: Allocating new free bucket. uma_zfree: Swapping buckets. uma_zfree: Putting old bucket on the free list. uma_zfree: Allocating new free bucket. uma_zfree: Swapping buckets. uma_zfree: Putting old bucket on the free list. uma_zfree: Allocating new free bucket. uma_zfree: Swapping buckets. ... The problem seems to be at [2], somehow after swapping buckets in uma_zfree m_dup returns a pointer to an mbuf that is still being used by us, [1] and [3] have same address. Then we call m_freem twice on same mbuf, [4] and [5]. And a loop occurs inside uma_free. I am using mbufs in a wrong way? Shouldnt mbufs be thread safe? Problem seems to occur while swapping buckets. Br, -- Monthadar Al Jaberi