From owner-freebsd-current@FreeBSD.ORG Thu Oct 14 13:53:55 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B75DC16A4CE for ; Thu, 14 Oct 2004 13:53:55 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5298143D2F for ; Thu, 14 Oct 2004 13:53:55 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i9EDrsJt013030 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 14 Oct 2004 09:53:54 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i9EDrmBX033659; Thu, 14 Oct 2004 09:53:48 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16750.34028.715601.329873@grasshopper.cs.duke.edu> Date: Thu, 14 Oct 2004 09:53:48 -0400 (EDT) To: freebsd-current@freebsd.org X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Subject: mbuf leak with SMP and debug.mpsafenet=1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Oct 2004 13:53:55 -0000 When bringing up our out-of-tree network driver, I noticed a pretty bad mbuf leak. Unfortunately, I think that the leak is in the system, and not in our driver because mbufs continue to leak at a rate of a few hundred per minute even when using the fxp interface. The leak is present after a reboot w/o loading our driver. You can duplicate this easily with netperf/netserver. Assume "scream" is the freebsd SMP host. Start netserver there. "scream" is a 3.06GHz P4 running RELENG_5 updated this morning. Its a HTT P4, running an SMP kernel. No witness/invariants. No zero-copy sockets. Log into another host and do: % while 1 while? ssh 'netstat -m | grep mbufs' while? netperf224 -Hscream -tUDP_STREAM -l60 -- -m 1 >& /dev/null while? end 441 mbufs in use 761 mbufs in use 1195 mbufs in use 1740 mbufs in use 2205 mbufs in use 2753 mbufs in use 3131 mbufs in use 3604 mbufs in use 4042 mbufs in use The only way I've seen to stop the leak is to set debug.mpsafenet=0 It might be interesting to note that the number of packets dropped due to full socket buffers is on the same order of magnitude as the number of mbufs leaked: % netstat -spudp udp: <...> 3933 dropped due to full socket buffers Other clues: The leak is much worse with our device (MPSAFE, 4Gb/sec, up to 128 pkts/intr) than with fxp. I've also seen the leak with TCP, but only with our 4Gb/s device, not with 100Mb/s fxp. So maybe there are 2 leaks, but I'd at least like to get the leak in the system figured out before I try to find a leak caused by my driver ;) Thanks, Drew