From owner-freebsd-current@FreeBSD.ORG Tue Mar 9 23:00:54 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF51410656C4 for ; Tue, 9 Mar 2010 23:00:54 +0000 (UTC) (envelope-from davidch@broadcom.com) Received: from MMS3.broadcom.com (mms3.broadcom.com [216.31.210.19]) by mx1.freebsd.org (Postfix) with ESMTP id B5B338FC19 for ; Tue, 9 Mar 2010 23:00:54 +0000 (UTC) Received: from [10.9.200.133] by MMS3.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Tue, 09 Mar 2010 15:00:40 -0800 X-Server-Uuid: B55A25B1-5D7D-41F8-BC53-C57E7AD3C201 Received: from IRVEXCHCCR01.corp.ad.broadcom.com ([10.252.49.30]) by IRVEXCHHUB02.corp.ad.broadcom.com ([10.9.200.133]) with mapi; Tue, 9 Mar 2010 15:02:04 -0800 From: "David Christensen" To: "Ryan Stone" Date: Tue, 9 Mar 2010 15:00:39 -0800 Thread-Topic: dev.bce.X.com_no_buffers increasing and packet loss Thread-Index: Acq/2EehDutlU8bKSFe+fo5qyfRDxAAA29Dw Message-ID: <5D267A3F22FD854F8F48B3D2B52381933AF90EEE0E@IRVEXCHCCR01.corp.ad.broadcom.com> References: <20100305210435.GF14818@michelle.cdnetworks.com> <20100305175639.GB14818@michelle.cdnetworks.com> <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> <20100309214012.GQ1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EEDA7@IRVEXCHCCR01.corp.ad.broadcom.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 X-WSS-ID: 67880C9231G54894420-01-01 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "pyunyh@gmail.com" , Ian FREISLICH , "current@freebsd.org" Subject: RE: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 23:00:55 -0000 =20 > -----Original Message----- > From: Ryan Stone [mailto:rysto32@gmail.com]=20 > Sent: Tuesday, March 09, 2010 2:31 PM > To: David Christensen > Cc: pyunyh@gmail.com; Ian FREISLICH; current@freebsd.org > Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss >=20 > > What's the traffic look like? =A0Jumbo, standard, short frames? =A0Any= =20 > > good ideas on profiling the code? =A0I haven't figured out how to use=20 > > the CPU TSC but there is a free running timer on the device=20 > that might=20 > > be usable to calculate where the driver's time is spent. > > > > Dave >=20 > In my experience hwpmc is the best and easiest way to profile=20 > anything on FreeBSD. Here's something I sent to a different=20 > thread a couple of months ago explaining how to use it: >=20 > 1) If device hwpmc is not compiled into your kernel, kldload=20 > hwpmc(you will need the HWPMC_HOOKS option in either case) > 2) Run pmcstat to begin taking samples(make sure that=20 > whatever you are profiling is busy doing work first!): >=20 > pmcstat -S unhalted-cycles -O /tmp/samples.out >=20 > The -S option specifies what event you want to use to trigger=20 > sampling. The unhalted-cycles is the best event to use if=20 > 3) After you've run pmcstat for "long enough"(a proper=20 > definition of long enough requires a statistician, which I=20 > most certainly am not, but I find that for a busy system 10=20 > seconds is enough), Control-C it to stop it*. You can use=20 > pmcstat to post-process the samples into human-readable text: >=20 > pmcstat -R /tmp/samples.out -G /tmp/graph.txt >=20 > The graph.txt file will show leaf functions on the left and=20 > their callers beneath them, indented to reflect the=20 > callchain. It's not too easy to describe and I don't have=20 > sample output available right now. Below is a quick sample I obtained running netperf. We're=20 interested in the bce(4) driver so I assume I'm interested in the time spent in bce and the functions it calls. Looks to me like memory allocation/freeing is a major source of CPU cycles in this test. Am I reading this right? @ CPU_CLK_UNHALTED_CORE [1091924 samples] 49.25% [537739] sched_idletd @ /boot/kernel/kernel 100.0% [537739] fork_exit 20.89% [228070] trash_dtor @ /boot/kernel/kernel 85.45% [194883] mb_dtor_clust 100.0% [194883] uma_zfree_arg 100.0% [194883] mb_free_ext 14.55% [33186] mb_dtor_mbuf 100.0% [33186] uma_zfree_arg 84.27% [27966] mb_free_ext 15.73% [5220] m_freem 00.00% [1] mb_dtor_pack 100.0% [1] uma_zfree_arg 100.0% [1] mb_free_ext 02.34% [25542] bce_intr @ /boot/kernel/if_bce.ko 100.0% [25542] intr_event_execute_handlers @ /boot/kernel/kernel 100.0% [25542] ithread_loop 100.0% [25542] fork_exit 02.20% [24055] trash_ctor @ /boot/kernel/kernel 96.41% [23192] mb_ctor_clust 100.0% [23192] uma_zalloc_arg 100.0% [23192] bce_fill_rx_chain @ /boot/kernel/if_bce.ko 03.39% [815] mb_ctor_mbuf @ /boot/kernel/kernel 100.0% [815] uma_zalloc_arg 99.39% [810] bce_fill_rx_chain @ /boot/kernel/if_bce.ko 00.49% [4] m_copym @ /boot/kernel/kernel 00.12% [1] tcp_output 00.20% [48] uma_zalloc_arg 100.0% [48] bce_fill_rx_chain @ /boot/kernel/if_bce.ko 100.0% [48] bce_intr Dave=