From owner-freebsd-net@FreeBSD.ORG Mon Aug 28 21:05:19 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5696516A4E1 for ; Mon, 28 Aug 2006 21:05:19 +0000 (UTC) (envelope-from rrs@cisco.com) Received: from sj-iport-5.cisco.com (sj-iport-5.cisco.com [171.68.10.87]) by mx1.FreeBSD.org (Postfix) with ESMTP id 25B4843D64 for ; Mon, 28 Aug 2006 21:05:06 +0000 (GMT) (envelope-from rrs@cisco.com) Received: from sj-dkim-5.cisco.com ([171.68.10.79]) by sj-iport-5.cisco.com with ESMTP; 28 Aug 2006 14:05:06 -0700 X-IronPort-AV: i="4.08,177,1154934000"; d="scan'208"; a="315052270:sNHT32482242" Received: from sj-core-4.cisco.com (sj-core-4.cisco.com [171.68.223.138]) by sj-dkim-5.cisco.com (8.12.11.20060308/8.12.11) with ESMTP id k7SL55vA032583 for ; Mon, 28 Aug 2006 14:05:05 -0700 Received: from xbh-sjc-211.amer.cisco.com (xbh-sjc-211.cisco.com [171.70.151.144]) by sj-core-4.cisco.com (8.12.10/8.12.6) with ESMTP id k7SL556Y016698 for ; Mon, 28 Aug 2006 14:05:05 -0700 (PDT) Received: from xfe-sjc-211.amer.cisco.com ([171.70.151.174]) by xbh-sjc-211.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.1830); Mon, 28 Aug 2006 14:05:05 -0700 Received: from [127.0.0.1] ([171.68.225.134]) by xfe-sjc-211.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.1830); Mon, 28 Aug 2006 14:05:05 -0700 Message-ID: <44F35A65.3080605@cisco.com> Date: Mon, 28 Aug 2006 17:04:37 -0400 From: Randall Stewart User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060223 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 28 Aug 2006 21:05:05.0291 (UTC) FILETIME=[A2B7BDB0:01C6CAE5] DKIM-Signature: a=rsa-sha1; q=dns; l=2755; t=1156799105; x=1157663105; c=relaxed/simple; s=sjdkim5002; h=Content-Type:From:Subject:Content-Transfer-Encoding:MIME-Version; d=cisco.com; i=rrs@cisco.com; z=From:Randall=20Stewart=20 |Subject:Problem=20with=20uipc_mbuf.c; X=v=3Dcisco.com=3B=20h=3DdfZfsNcoqS4OnyibfxSfNntZUb0=3D; b=itdBBdYvGVd3KMvghyQgRA9kJMdz+xLGEezKQUxJXykuba1EOGX8H8N0ia8CHZ+RiXWYC/vZ Cgf2MppEyY/qRKdPJVJBdY5WdHNOPPHvmkGe65Xgoz2ROoyy7eZLPjYz; Authentication-Results: sj-dkim-5.cisco.com; header.From=rrs@cisco.com; dkim=pass ( sig from cisco.com verified; ); Subject: Problem with uipc_mbuf.c X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Aug 2006 21:05:19 -0000 Hi all: In 6.1 the function mb_free_ext(struct mbuf *m) looked like this: ............................................... void mb_free_ext(struct mbuf *m) { u_int cnt; int dofree; /* Account for lazy ref count assign. */ if (m->m_ext.ref_cnt == NULL) dofree = 1; else dofree = 0; /* * This is tricky. We need to make sure to decrement the * refcount in a safe way but to also clean up if we're the * last reference. This method seems to do it without race. */ while (dofree == 0) { cnt = *(m->m_ext.ref_cnt); if (atomic_cmpset_int(m->m_ext.ref_cnt, cnt, cnt - 1)) { if (cnt == 1) dofree = 1; break; } } if (dofree) { /* * Do the free, should be safe. */ switch (m->m_ext.ext_type) { ................................. Other fine code that does the freeing... .................................. Now, in 7.0 we have: ------------------------------------------- void mb_free_ext(struct mbuf *m) { KASSERT((m->m_flags & M_EXT) == M_EXT, ("%s: M_EXT not set", __func__)); KASSERT(m->m_ext.ref_cnt != NULL, ("%s: ref_cnt not set", __func__)); /* Free attached storage if this mbuf is the only reference to it. */ if (*(m->m_ext.ref_cnt) == 1 || atomic_fetchadd_int(m->m_ext.ref_cnt, -1) == 0) { switch (m->m_ext.ext_type) { ------------------------------------- Other stuff that does the freeing ------------------------------------- This new code is broken... I am sad to say.. I have spent a LARGE amount of time hunting an "mbuf" leak.. and this is where I have traced it to... Now here is what I see happening.. I have two Xeon PIV machines.. Dell servers.. with Hyper threading.. One runs 6.1 Release.. the other 7.0... current. Now I am playing with SCTP.. and when I run netpipe the 7.0 machine leaks mbufs I have traced it down to the fact that SCTP uses finer grain locks.. so what is going on is the input processing is happening... and on the other CPU the reader (netpipe) is reading the data. This causes m_free()'s to be called on the same EXT's about the same time... and they don't get freed. Now to prove that it was NOT the SCTP code I went in and put a new mutex around the actual m_free and m_freem code... letting only one guy in at a time to free.. and all my leaks went away.. (my performance was also drug down a lot.. but thats besides the point)... So... has anyone else seen this? Or is SCTP the only one who is truely excercising this bug? I am thinking about restoring the old code.. since it appears to work... Any comments or help would be appreciated.. Thanks R -- Randall Stewart NSSTG - Cisco Systems Inc. 803-345-0369 815-342-5222 (cell)