Date: Sat, 12 Sep 2015 11:47:45 +0300 From: Dmitry Sivachenko <trtrmitya@gmail.com> To: hiren panchasara <hiren@strugglingcoder.info> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: netisr process eats 100% cpu Message-ID: <5599DAD0-55CD-4B36-B667-F943737C2D85@gmail.com> In-Reply-To: <20150911171910.GN64965@strugglingcoder.info> References: <C4039E45-3DC7-4E1A-9E46-853FE812B906@gmail.com> <EA9AD61B-E709-4387-8D5E-A33E25475D81@gmail.com> <20150911171910.GN64965@strugglingcoder.info>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 11 =D1=81=D0=B5=D0=BD=D1=82. 2015 =D0=B3., at 20:19, hiren = panchasara <hiren@strugglingcoder.info> wrote: >=20 > On 09/11/15 at 12:46P, Dmitry Sivachenko wrote: >>=20 >>> hiren panchasara <hiren at strugglingcoder.info> wrote: >>>=20 >>> Unsure at the moment if loopback is causing the trouble for you or = not. >>> See: >>=20 >> (please keep me CC'ed, I am not subscribed to -net) >>=20 >>=20 >>>=20 >>> = https://lists.freebsd.org/pipermail/freebsd-net/2015-February/041239.html >>>=20 >>>=20 >>=20 >> Yes, this thread looks similar. >>=20 >>=20 >>> You may want to try: >>> 1) pmcstat and see if you can catch something >>=20 >> What particular should I look for? Here is first lines of pmcstat -T = -S instructions -w 1: >> PMC: [INSTR_RETIRED_ANY] Samples: 157198 (100.0%) , 0 unresolved >>=20 >> %SAMP IMAGE FUNCTION CALLERS >> 13.2 kernel cpu_search_highest cpu_search_highest:12.0 = sched_idletd:1.2 >> 8.3 kernel ipfw_chk ipfw_check_packet >> 3.1 myprogram memsetAVX _ZN12TLz4Compress7DoWriteEPKv >> 2.3 kernel tcp_output tcp_usr_send:1.0 = tcp_do_segment:0.9 >>=20 >>=20 >>> 2) disable checksum on localhost >>=20 >>=20 >> I tried, but nothing has changed. >>=20 >>=20 >>> 3) look at netisr settings. sysctl net.isr o/p and how it looks = under >>> netstat -Q. I am not sure if adding more threads to netisr via >>=20 >>=20 >> What should I look for? >>=20 >>=20 >>> net.isr.numthreads would help. (Note its a loader.conf variable) >>=20 >>=20 >> This netisr load looks parasitical to me (as I noted, moving haproxy = to a separate machine does not burn CPU cycles on netisr, why is = localhost special?) >>=20 >> Even if adding more threads to netisr would boost network = utilization, wouldn't those CPU cycles spent for netisr just be a waste = of energy? I have other tasks for these CPU. >>=20 >=20 > I am not sure what keep cpu busy with netisr when localhost is = involved. >=20 > You may want to post o/p of=20 > # sysctl net.isr net.isr.numthreads: 1 net.isr.maxprot: 16 net.isr.defaultqlimit: 256 net.isr.maxqlimit: 10240 net.isr.bindthreads: 0 net.isr.maxthreads: 1 net.isr.dispatch: direct > # netstat -Q Configuration: Setting Current Limit Thread count 1 1 Default queue limit 256 10240 Dispatch policy direct n/a Threads bound to CPUs disabled n/a Protocols: Name Proto QLimit Policy Dispatch Flags ip 1 4096 flow default --- igmp 2 256 source default --- rtsock 3 256 source default --- arp 7 256 source default --- ether 9 256 source direct --- ip6 10 256 flow default --- Workstreams: WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued = Handled 0 0 ip 8 165 9862463 0 0 126594714 = 136424969 0 0 igmp 0 0 0 0 0 0 = 0 0 0 rtsock 0 1 0 0 0 10 = 10 0 0 arp 0 0 15640 0 0 0 = 15640 0 0 ether 0 0 9878107 0 0 0 = 9878107 0 0 ip6 0 2 4 0 0 12 = 16 > # sysctl net.inet | grep queue net.inet.ip.intr_queue_maxlen: 4096 net.inet.ip.intr_queue_drops: 0 net.inet.ip.dummynet.queue_count: 0 >=20 > A suggestion I see at https://calomel.org/freebsd_network_tuning.html = is > to increase localhost n/w buffers. Not sure if this'll help your case. > net.local.stream.sendspace=3D164240 # (default 8192) > net.local.stream.recvspace=3D164240 # (default 8192) I already had 65536 for that, but increased up to 164240, nothing has = changed. >=20 > Now I'll let someone else with more ideas/clues comment. >=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5599DAD0-55CD-4B36-B667-F943737C2D85>