From owner-freebsd-net@FreeBSD.ORG Thu Feb 5 20:08:32 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B29116CD; Thu, 5 Feb 2015 20:08:32 +0000 (UTC) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com [IPv6:2607:f8b0:4001:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 790AB269; Thu, 5 Feb 2015 20:08:32 +0000 (UTC) Received: by mail-ig0-f173.google.com with SMTP id a13so604750igq.0; Thu, 05 Feb 2015 12:08:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=8pE0+tH8zHks+LnhcqwRr+3u1BprtnLNoCb6HQcjWBQ=; b=SrnNdn2rHXkKGXu76TFbHFbuHwRdufq8LB6/Mfmon6sNwvIC+FSGTYOLbSXp/P1Zb0 fRqv2UeAE37OUoczeB9lG1b7fpGZWDbyJ/WwNM/qQ4LGoLX+45epAg8JZAUs7mm6ppGa W48iWZkhxe8H50R7i7GP5cqGdOOlrIZsbOA9zzZnZ/GQmLQkZfsKPY6cjROAMvuzmMV3 sxvyx1ItvpVq/DHgwQxo0y1ap30cTXnjIfJSpL3PN9v/rw+FSq1dwdsVoH8rjHMziOHh IgpJJfhUmPRuv+1p8kNvNz2jM4tuuvRGT/x+zRQdyePU/vQZiqyybzNYwVxogxWkyh42 FqPw== MIME-Version: 1.0 X-Received: by 10.42.62.71 with SMTP id x7mr9002676ich.61.1423166911846; Thu, 05 Feb 2015 12:08:31 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.36.17.7 with HTTP; Thu, 5 Feb 2015 12:08:31 -0800 (PST) In-Reply-To: <752D84FB-0B65-47CF-973A-91C3697A28DC@yahoo.com> References: <54D3BE67.8060502@ignoranthack.me> <752D84FB-0B65-47CF-973A-91C3697A28DC@yahoo.com> Date: Thu, 5 Feb 2015 12:08:31 -0800 X-Google-Sender-Auth: VVUkTFQdwla6NyGDqHyjLBfdi7s Message-ID: Subject: Re: Silly experiments with netisr From: Adrian Chadd To: Scott Long Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 20:08:32 -0000 On 5 February 2015 at 11:31, Scott Long via freebsd-net wrote: > >> On Feb 5, 2015, at 12:03 PM, Sean Bruno wrote: >> >> >> Signed PGP part >> Some questions came up around the office and we ended up doing some >> quite silly things with lo0 and netcat. >> >> If one runs a continuous netcat on localhost to another netcat listener >> on localhost that writes the output to /dev/null, netisr gets super busy >> doing stuff/things. >> >> E.g. >> -- listener running "nc -k -l 10000 > /dev/null" >> - sender running in a while loop "nc -N localhost 10000 < >> /var/tmp/testfile" >> >> Interesting things start happening on the machine. top -SH shows netisr >> eating up about 1/2 of a cpu core. If you drop the MTU on lo0 to 1500 >> (so that it looks like something in the real world), netisr will peg out >> a cpu core. This seems logical, in that smaller MTU means busier >> netisr. Its interesting though. >> >> Looking at some pmcstat things, shows that the system is busilly >> chugging along in tcp_do_segment(). I wonder if this is meaningful in >> anyway or just "interesting". > > > Welcome to our workload. Granted, we don=E2=80=99t involve pf, but the m= ajority of our CPU processing right now is spent in TCP (with the rest bein= g spent in the VM, but that=E2=80=99s a different matter). > > FWIW, Randall has some optimizations in this area of the stack. They are= n=E2=80=99t huge, IIRC they=E2=80=99re only a few percent, but worth lookin= g at. Yeah, I see that too in all the TCP concurrency testing I'm doing. The moment TCP TSO drops in effectiveness in any way, the cost of tcp_do_segment() jumps dramatically. :( -adrian