From owner-freebsd-net@freebsd.org Sat Sep 12 17:20:46 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34CA5A03C71 for ; Sat, 12 Sep 2015 17:20:46 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CE9E419B6 for ; Sat, 12 Sep 2015 17:20:45 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id t8CHKd4q097636 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 12 Sep 2015 20:20:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua t8CHKd4q097636 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id t8CHKdkD097634; Sat, 12 Sep 2015 20:20:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 12 Sep 2015 20:20:39 +0300 From: Konstantin Belousov To: Dmitry Sivachenko Cc: hiren panchasara , FreeBSD Net Subject: Re: netisr process eats 100% cpu Message-ID: <20150912172039.GP2072@kib.kiev.ua> References: <20150911171910.GN64965@strugglingcoder.info> <5599DAD0-55CD-4B36-B667-F943737C2D85@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5599DAD0-55CD-4B36-B667-F943737C2D85@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Sep 2015 17:20:46 -0000 On Sat, Sep 12, 2015 at 11:47:45AM +0300, Dmitry Sivachenko wrote: > > > On 11 сент. 2015 г., at 20:19, hiren panchasara wrote: > > > > On 09/11/15 at 12:46P, Dmitry Sivachenko wrote: > >> > >>> hiren panchasara wrote: > >>> > >>> Unsure at the moment if loopback is causing the trouble for you or not. > >>> See: > >> > >> (please keep me CC'ed, I am not subscribed to -net) > >> > >> > >>> > >>> https://lists.freebsd.org/pipermail/freebsd-net/2015-February/041239.html > >>> > >>> > >> > >> Yes, this thread looks similar. > >> > >> > >>> You may want to try: > >>> 1) pmcstat and see if you can catch something > >> > >> What particular should I look for? Here is first lines of pmcstat -T -S instructions -w 1: > >> PMC: [INSTR_RETIRED_ANY] Samples: 157198 (100.0%) , 0 unresolved > >> > >> %SAMP IMAGE FUNCTION CALLERS > >> 13.2 kernel cpu_search_highest cpu_search_highest:12.0 sched_idletd:1.2 > >> 8.3 kernel ipfw_chk ipfw_check_packet > >> 3.1 myprogram memsetAVX _ZN12TLz4Compress7DoWriteEPKv > >> 2.3 kernel tcp_output tcp_usr_send:1.0 tcp_do_segment:0.9 > >> > >> > >>> 2) disable checksum on localhost > >> > >> > >> I tried, but nothing has changed. > >> > >> > >>> 3) look at netisr settings. sysctl net.isr o/p and how it looks under > >>> netstat -Q. I am not sure if adding more threads to netisr via > >> > >> > >> What should I look for? > >> > >> > >>> net.isr.numthreads would help. (Note its a loader.conf variable) > >> > >> > >> This netisr load looks parasitical to me (as I noted, moving haproxy to a separate machine does not burn CPU cycles on netisr, why is localhost special?) > >> Loopback is special since network stack input path must return the packets into the output path, and doing it by direct function call would recurse the stack. Netisr allows to queue the packet and then deliver it using the safe context. At least this is my limited understanding of the loopback+netisr coupling. Why it takes so much CPU for output processing, is beyond my knowledge of the network. But some resources have to be spent on localhost. > >> Even if adding more threads to netisr would boost network utilization, wouldn't those CPU cycles spent for netisr just be a waste of energy? I have other tasks for these CPU. > >> > > > > I am not sure what keep cpu busy with netisr when localhost is involved. > > > > You may want to post o/p of > > # sysctl net.isr > > net.isr.numthreads: 1 > net.isr.maxprot: 16 > net.isr.defaultqlimit: 256 > net.isr.maxqlimit: 10240 > net.isr.bindthreads: 0 > net.isr.maxthreads: 1 > net.isr.dispatch: direct > > > > # netstat -Q > > Configuration: > Setting Current Limit > Thread count 1 1 > Default queue limit 256 10240 > Dispatch policy direct n/a > Threads bound to CPUs disabled n/a > > Protocols: > Name Proto QLimit Policy Dispatch Flags > ip 1 4096 flow default --- > igmp 2 256 source default --- > rtsock 3 256 source default --- > arp 7 256 source default --- > ether 9 256 source direct --- > ip6 10 256 flow default --- > > Workstreams: > WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled > 0 0 ip 8 165 9862463 0 0 126594714 136424969 > 0 0 igmp 0 0 0 0 0 0 0 > 0 0 rtsock 0 1 0 0 0 10 10 > 0 0 arp 0 0 15640 0 0 0 15640 > 0 0 ether 0 0 9878107 0 0 0 9878107 > 0 0 ip6 0 2 4 0 0 12 16 > > > > # sysctl net.inet | grep queue > > net.inet.ip.intr_queue_maxlen: 4096 > net.inet.ip.intr_queue_drops: 0 > net.inet.ip.dummynet.queue_count: 0 > > > > > > A suggestion I see at https://calomel.org/freebsd_network_tuning.html is > > to increase localhost n/w buffers. Not sure if this'll help your case. > > net.local.stream.sendspace=164240 # (default 8192) > > net.local.stream.recvspace=164240 # (default 8192) > > > I already had 65536 for that, but increased up to 164240, nothing has changed. > > > > > > Now I'll let someone else with more ideas/clues comment. > > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"