Date: Fri, 25 Jul 2014 13:51:03 -0700 From: Adrian Chadd <adrian@freebsd.org> To: John Jasen <jjasen@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: fastforward/routing: a 3 million packet-per-second system? Message-ID: <CAJ-Vmonsc79ULDOT9trtOotq7mRh1XJkhL2JfDNxXP16OFWMFg@mail.gmail.com> In-Reply-To: <CAACLuR1r0axCYWXeLDSa-m07eAVgTMBVW5sNbt%2By_Lt2ss1r7Q@mail.gmail.com> References: <53CE80DD.9090109@gmail.com> <CAACLuR1r0axCYWXeLDSa-m07eAVgTMBVW5sNbt%2By_Lt2ss1r7Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Ugh, the forwarding table stupidity. Try enabling FLOWTABLE as an option. I really dislike how the rtentry locking works. But that isn't a rwlock - i'll have to look at your full lock profiling output to see. -a On 25 July 2014 09:20, John Jasen <jjasen@gmail.com> wrote: > Based on advice I received, I've collected lock profile debugging output, > and pmcannotate'd data from the system while it was processing about 3 > million packets/second. > > Combined, the files are about 325k in size, so I'll submit highlights here. > I can provide the raw files to interested parties privately. > > pmcannotate summary output: > > grep ^Profile pmcannotate.20140725 > Profile trace for function: __rw_rlock() [17.04%] > Profile trace for function: __mtx_unlock_flags() [9.10%] > Profile trace for function: _rw_runlock_cookie() [7.67%] > Profile trace for function: sched_idletd() [5.73%] > Profile trace for function: memcpy() [5.64%] > Profile trace for function: bcopy() [5.04%] > Profile trace for function: bcmp() [5.01%] > Profile trace for function: __mtx_lock_flags() [3.66%] > Profile trace for function: t4_eth_tx() [3.25%] > Profile trace for function: lock_profile_release_lock() [2.73%] > Profile trace for function: ip_fastforward() [2.68%] > Profile trace for function: ether_output() [2.50%] > Profile trace for function: get_scatter_segment() [1.75%] > Profile trace for function: rn_match() [1.74%] > Profile trace for function: _mtx_lock_spin_cookie() [1.53%] > Profile trace for function: lock_profile_obtain_lock_success() [1.49%] > Profile trace for function: cxgbe_transmit() [1.37%] > Profile trace for function: uma_zalloc_arg() [1.31%] > Profile trace for function: bzero() [1.30%] > Profile trace for function: service_iq() [1.26%] > Profile trace for function: ether_nh_input() [1.23%] > Profile trace for function: __mtx_lock_sleep() [1.19%] > Profile trace for function: arpresolve() [1.07%] > Profile trace for function: uma_zfree_arg() [0.95%] > Profile trace for function: reclaim_tx_descs() [0.87%] > Profile trace for function: _mtx_trylock_flags_() [0.80%] > Profile trace for function: bounce_bus_dmamap_load_buffer() [0.72%] > Profile trace for function: ether_demux() [0.64%] > Profile trace for function: mb_ctor_mbuf() [0.63%] > Profile trace for function: rtalloc1_fib() [0.54%] > > sysctl debug.lock.prof.stats summary: (some of the highest hit counts, > especially in cnt_hold: > > 7 17 3271389 2258698 63952832 0 0 0 > 6761237 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry) > > 7 13 11543138 470545 63952832 0 0 0 > 815777 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry) > > 6 15 3943582 1545195 63952779 0 0 0 > 3439254 /usr/src/sys/netinet/ip_fastfwd.c:593 (sleep mutex:rtentry > > > > > > > > > > > > On Tue, Jul 22, 2014 at 11:18 AM, John Jasen <jjasen@gmail.com> wrote: > >> Feedback and/or tips and tricks more than welcome. >> > > <snip> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmonsc79ULDOT9trtOotq7mRh1XJkhL2JfDNxXP16OFWMFg>