From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 20:51:05 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 80590EF8 for ; Fri, 25 Jul 2014 20:51:05 +0000 (UTC) Received: from mail-qg0-x22d.google.com (mail-qg0-x22d.google.com [IPv6:2607:f8b0:400d:c04::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 416F32BFA for ; Fri, 25 Jul 2014 20:51:05 +0000 (UTC) Received: by mail-qg0-f45.google.com with SMTP id f51so5713598qge.18 for ; Fri, 25 Jul 2014 13:51:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=t9gyCaDCB0TNkTyHftn2kOYHJBNEhj4MA9mOdvNcGtM=; b=O326rufww3jxr0ugaxIju8uYACdj41YzNZIZmlykQgulD/Ku7Nu8M8nnJRLsjx1LUg ajGfnDMeu3ntDnsT/m0yiBqbAKljzaEpuMZK6/R4LBgDTl9PbPvs0VIJqgPaS/XKrA6x IqlFk2xvGlWqSfgog3ag9x4uBxr403GF4UGLYhTgn3I7q6/2tmPl3l27sA/Zqph+9oeG nayejvWYiFuOssB0DwcNAHF/LsewYagM+omDNDE8yl3Xg3x7GkY9XKbIxqcmb58s24Qf EXn9VlN5QTlxhXHdv3eY60ALOoFNGEgY8YaIDp5S9hqf6NSmUMG1tbilly6kYq5oBGIv pZTw== MIME-Version: 1.0 X-Received: by 10.224.112.1 with SMTP id u1mr30758615qap.7.1406321464216; Fri, 25 Jul 2014 13:51:04 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Fri, 25 Jul 2014 13:51:03 -0700 (PDT) In-Reply-To: References: <53CE80DD.9090109@gmail.com> Date: Fri, 25 Jul 2014 13:51:03 -0700 X-Google-Sender-Auth: uFJqQuQ_1t92F0DNooaDS4vH5MM Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: Adrian Chadd To: John Jasen Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 20:51:05 -0000 Ugh, the forwarding table stupidity. Try enabling FLOWTABLE as an option. I really dislike how the rtentry locking works. But that isn't a rwlock - i'll have to look at your full lock profiling output to see. -a On 25 July 2014 09:20, John Jasen wrote: > Based on advice I received, I've collected lock profile debugging output, > and pmcannotate'd data from the system while it was processing about 3 > million packets/second. > > Combined, the files are about 325k in size, so I'll submit highlights here. > I can provide the raw files to interested parties privately. > > pmcannotate summary output: > > grep ^Profile pmcannotate.20140725 > Profile trace for function: __rw_rlock() [17.04%] > Profile trace for function: __mtx_unlock_flags() [9.10%] > Profile trace for function: _rw_runlock_cookie() [7.67%] > Profile trace for function: sched_idletd() [5.73%] > Profile trace for function: memcpy() [5.64%] > Profile trace for function: bcopy() [5.04%] > Profile trace for function: bcmp() [5.01%] > Profile trace for function: __mtx_lock_flags() [3.66%] > Profile trace for function: t4_eth_tx() [3.25%] > Profile trace for function: lock_profile_release_lock() [2.73%] > Profile trace for function: ip_fastforward() [2.68%] > Profile trace for function: ether_output() [2.50%] > Profile trace for function: get_scatter_segment() [1.75%] > Profile trace for function: rn_match() [1.74%] > Profile trace for function: _mtx_lock_spin_cookie() [1.53%] > Profile trace for function: lock_profile_obtain_lock_success() [1.49%] > Profile trace for function: cxgbe_transmit() [1.37%] > Profile trace for function: uma_zalloc_arg() [1.31%] > Profile trace for function: bzero() [1.30%] > Profile trace for function: service_iq() [1.26%] > Profile trace for function: ether_nh_input() [1.23%] > Profile trace for function: __mtx_lock_sleep() [1.19%] > Profile trace for function: arpresolve() [1.07%] > Profile trace for function: uma_zfree_arg() [0.95%] > Profile trace for function: reclaim_tx_descs() [0.87%] > Profile trace for function: _mtx_trylock_flags_() [0.80%] > Profile trace for function: bounce_bus_dmamap_load_buffer() [0.72%] > Profile trace for function: ether_demux() [0.64%] > Profile trace for function: mb_ctor_mbuf() [0.63%] > Profile trace for function: rtalloc1_fib() [0.54%] > > sysctl debug.lock.prof.stats summary: (some of the highest hit counts, > especially in cnt_hold: > > 7 17 3271389 2258698 63952832 0 0 0 > 6761237 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry) > > 7 13 11543138 470545 63952832 0 0 0 > 815777 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry) > > 6 15 3943582 1545195 63952779 0 0 0 > 3439254 /usr/src/sys/netinet/ip_fastfwd.c:593 (sleep mutex:rtentry > > > > > > > > > > > > On Tue, Jul 22, 2014 at 11:18 AM, John Jasen wrote: > >> Feedback and/or tips and tricks more than welcome. >> > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"