From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 16:20:41 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D30E6FE4 for ; Fri, 25 Jul 2014 16:20:41 +0000 (UTC) Received: from mail-lb0-x236.google.com (mail-lb0-x236.google.com [IPv6:2a00:1450:4010:c04::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4995622A7 for ; Fri, 25 Jul 2014 16:20:41 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id z11so3642422lbi.13 for ; Fri, 25 Jul 2014 09:20:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=QtsUebttApqMJwZhzChJnTwvYmrdrsiM5Kn7L8uOXS4=; b=k+Tr7ZJPweoabsHAkANPiwhd5iTvSjtripAO/rfYkIqFtdSjAKTYr+SjlZG/xCDeJ7 1KBe/Zm9SsJue2RbyrsAcY4Nu7er//a0ZU7KM82DokPVb3SLHPuxxU1GLXSXhUblqKBw 9sciXBxFWeUe4bEfRE3wcbzJwRr3x2lysr5v7NUfkay4+rudM6whJIjm2g2j+8RZ/7XY 7RkPVjAzlYRF/5vE3SB2Vh51ZmHZWx94krKsP6zrPz9Mv4w6A0dioBwPyM3x9RT3I0Gu 60HdnRFUPaW61/+xr6pSX1rW+IWEyS/yJPPFDIkCpmjC6dvP2ksXL0M6hiRD968LvOjF TzSg== MIME-Version: 1.0 X-Received: by 10.152.5.167 with SMTP id t7mr17848216lat.54.1406305239138; Fri, 25 Jul 2014 09:20:39 -0700 (PDT) Received: by 10.114.176.106 with HTTP; Fri, 25 Jul 2014 09:20:38 -0700 (PDT) In-Reply-To: <53CE80DD.9090109@gmail.com> References: <53CE80DD.9090109@gmail.com> Date: Fri, 25 Jul 2014 12:20:38 -0400 Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: John Jasen To: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 16:20:41 -0000 Based on advice I received, I've collected lock profile debugging output, and pmcannotate'd data from the system while it was processing about 3 million packets/second. Combined, the files are about 325k in size, so I'll submit highlights here. I can provide the raw files to interested parties privately. pmcannotate summary output: grep ^Profile pmcannotate.20140725 Profile trace for function: __rw_rlock() [17.04%] Profile trace for function: __mtx_unlock_flags() [9.10%] Profile trace for function: _rw_runlock_cookie() [7.67%] Profile trace for function: sched_idletd() [5.73%] Profile trace for function: memcpy() [5.64%] Profile trace for function: bcopy() [5.04%] Profile trace for function: bcmp() [5.01%] Profile trace for function: __mtx_lock_flags() [3.66%] Profile trace for function: t4_eth_tx() [3.25%] Profile trace for function: lock_profile_release_lock() [2.73%] Profile trace for function: ip_fastforward() [2.68%] Profile trace for function: ether_output() [2.50%] Profile trace for function: get_scatter_segment() [1.75%] Profile trace for function: rn_match() [1.74%] Profile trace for function: _mtx_lock_spin_cookie() [1.53%] Profile trace for function: lock_profile_obtain_lock_success() [1.49%] Profile trace for function: cxgbe_transmit() [1.37%] Profile trace for function: uma_zalloc_arg() [1.31%] Profile trace for function: bzero() [1.30%] Profile trace for function: service_iq() [1.26%] Profile trace for function: ether_nh_input() [1.23%] Profile trace for function: __mtx_lock_sleep() [1.19%] Profile trace for function: arpresolve() [1.07%] Profile trace for function: uma_zfree_arg() [0.95%] Profile trace for function: reclaim_tx_descs() [0.87%] Profile trace for function: _mtx_trylock_flags_() [0.80%] Profile trace for function: bounce_bus_dmamap_load_buffer() [0.72%] Profile trace for function: ether_demux() [0.64%] Profile trace for function: mb_ctor_mbuf() [0.63%] Profile trace for function: rtalloc1_fib() [0.54%] sysctl debug.lock.prof.stats summary: (some of the highest hit counts, especially in cnt_hold: 7 17 3271389 2258698 63952832 0 0 0 6761237 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry) 7 13 11543138 470545 63952832 0 0 0 815777 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry) 6 15 3943582 1545195 63952779 0 0 0 3439254 /usr/src/sys/netinet/ip_fastfwd.c:593 (sleep mutex:rtentry On Tue, Jul 22, 2014 at 11:18 AM, John Jasen wrote: > Feedback and/or tips and tricks more than welcome. >