From owner-freebsd-net@FreeBSD.ORG Mon Jul 28 14:51:14 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 28054B9; Mon, 28 Jul 2014 14:51:14 +0000 (UTC) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 743642407; Mon, 28 Jul 2014 14:51:13 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id u10so5766223lbd.7 for ; Mon, 28 Jul 2014 07:51:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=aju0zPU5sla1Ca0BD0xKpmS4uBeMMMj7+sFJFvMsbUE=; b=T8nRJj23hwH4stzM5J+XFhRSRUDqLM37nkJ/hhyww5EqGl7yeVI/tP9V838QNn0+ss SdBbgjxMLLXXdYW7WpeR6aodb5lBqoVA8nGXTZhOOdpoqBzezRPF3NaLvezEqbBWIXq9 jjRyZfpJpQKzFn44t/Sf2SHE2cNoGB9VuV4Y21+AlxcGCNM7PU6cJtzbYrS7hpUsqsEK RkoevJ4K7ur9jCIqwLanPmJwuVWPa6g2O2pknCC9DiSBTA6dLLrr+CsjoPQlcEB74O3h T7BzYJ33xokOzCn3Yu7OsRlnV/+Xk3i3b2Nwf0Bp8pF56ad7lsVYASpcIGmJj3uuIyxL KeYw== MIME-Version: 1.0 X-Received: by 10.152.9.233 with SMTP id d9mr10897289lab.66.1406559071387; Mon, 28 Jul 2014 07:51:11 -0700 (PDT) Received: by 10.114.176.106 with HTTP; Mon, 28 Jul 2014 07:51:11 -0700 (PDT) In-Reply-To: References: <53CE80DD.9090109@gmail.com> <53D4600A.1010505@gmail.com> <53D4F77B.9020009@gmail.com> Date: Mon, 28 Jul 2014 10:51:11 -0400 Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: John Jasen To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 14:51:14 -0000 in_input crept up into the top 5, versus fastforward. Would PMC counters help? cat debug.lock.pref.stats.out-20140728-1 | sort -nk 4 | tail -5 5 4 413 115 160 2 0 0 63 /usr/src/sys/kern/kern_condvar.c:145 (sleep mutex:Giant) 1 1 148858 4095 650072 0 0 0 11184 /usr/src/sys/kern/subr_turnstile.c:552 (spin mutex:turnstile chain) 8 14 13747639 561636 72520256 0 0 0 689603 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry) 3 20 3907071 2322975 72520256 0 0 0 2529589 /usr/src/sys/netinet/ip_input.c:1315 (sleep mutex:rtentry) 3 17 3665247 3715117 72520256 0 0 0 8425384 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry) On Sun, Jul 27, 2014 at 1:50 PM, Adrian Chadd wrote: > Yeah, there's something odd going on. You shouldn't see any of that > lock contention if flowtable is enabled. Thus I think there's > > Oh wait, the fastfwd code doesn't know about flowtables. I just looked > at it (sys/netinet/ip_fastfwd.c.) > > Try disabling fastfwd for a test and see if the lock profile improves. > (Set debug.lock.prof.reset=1 to clear the profiling data before you do it.) > > > -a > > > On 27 July 2014 05:58, John Jasen wrote: > > I shouldn't even be coming close to maxflows in this test scenario. > > > > net.flowtable.enable: 1 > > net.flowtable.maxflows: 1042468 > > > > On 07/26/2014 10:20 PM, Adrian Chadd wrote: > >> Flowtable is enabled? That's odd, it shouldn't be showing up like that. > >> > >> > >> > >> -a > >> > >> > >> > > >