From owner-freebsd-net@FreeBSD.ORG Fri Apr 11 02:17:56 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4042EF62 for ; Fri, 11 Apr 2014 02:17:56 +0000 (UTC) Received: from mail-ee0-x234.google.com (mail-ee0-x234.google.com [IPv6:2a00:1450:4013:c00::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C09F212B5 for ; Fri, 11 Apr 2014 02:17:55 +0000 (UTC) Received: by mail-ee0-f52.google.com with SMTP id e49so3506179eek.25 for ; Thu, 10 Apr 2014 19:17:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=7TrUrv4NSh9tE9Bhydw9PuJoG9f28Et47m8hpUcwFtM=; b=HLZW5ACAeNQK4F1R9z46dEGsd6SiD9yRAltWfIUjQxsMtClKftkm4B+Ui6xnCrwaiN AfLG1fb3gKA4BGSmk2OZXHQQ8GbRm3WduEi0DWL4cQ7kfWb7MEeA+8FVWFSTHOIzTciV upma8ZLV3cUFo+mNs3kMcnP4098T1NpMttG5A6gTPYeY+L2v4Iawu+L2yjPIx0SVFr+z 0cN1TknqjZuhJ8uMrEM1BpsNyaMvldzs0Xl92MOoEn1bjMShsKOEr6CalsZbfHla0UJ+ wnOcyrwXSE0wY+Yvmg+M7nWqFcgzuDGutA8NtSilQULWuVNqnIcmkfqJL5R2Qn1g1fKc 1Y7Q== MIME-Version: 1.0 X-Received: by 10.14.176.193 with SMTP id b41mr7598611eem.55.1397182672262; Thu, 10 Apr 2014 19:17:52 -0700 (PDT) Received: by 10.14.213.194 with HTTP; Thu, 10 Apr 2014 19:17:52 -0700 (PDT) Date: Thu, 10 Apr 2014 19:17:52 -0700 Message-ID: Subject: netisr observations From: hiren panchasara To: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2014 02:17:56 -0000 (Note: This may seem more like a rant than an actual problem report.) I am on a stable-10ish box with igb0. Workload is mainly inbound nfs traffic. About 2K connections at any point in time. device igb # Intel PRO/1000 PCIE Server Gigabit Family hw.igb.rxd: 4096 hw.igb.txd: 4096 hw.igb.enable_aim: 1 hw.igb.enable_msix: 1 hw.igb.max_interrupt_rate: 32768 hw.igb.buf_ring_size: 4096 hw.igb.header_split: 0 hw.igb.num_queues: 0 hw.igb.rx_process_limit: 100 dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 dev.igb.0.%driver: igb dev.igb.0.%location: slot=0 function=0 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x103c subdevice=0x323f class=0x020000 -bash-4.2$ netstat -I igb0 -i 1 input igb0 output packets errs idrops bytes packets errs bytes colls 18332 0 0 19096474 22946 0 18211000 0 19074 0 0 11408912 28280 0 29741195 0 15753 0 0 15499238 21234 0 16779695 0 12914 0 0 9583719 17945 0 14599603 0 13677 0 0 10818359 19050 0 15069889 0 -bash-4.2$ sysctl net.isr net.isr.dispatch: direct net.isr.maxthreads: 8 net.isr.bindthreads: 0 net.isr.maxqlimit: 10240 net.isr.defaultqlimit: 256 net.isr.maxprot: 16 net.isr.numthreads: 8 -bash-4.2$ sysctl -a | grep igb.0 | grep rx_bytes dev.igb.0.queue0.rx_bytes: 65473003127 dev.igb.0.queue1.rx_bytes: 73982776038 dev.igb.0.queue2.rx_bytes: 57669494795 dev.igb.0.queue3.rx_bytes: 57830053867 dev.igb.0.queue4.rx_bytes: 75087429774 dev.igb.0.queue5.rx_bytes: 69252615374 dev.igb.0.queue6.rx_bytes: 70565370833 dev.igb.0.queue7.rx_bytes: 90210083223 I am seeing something interesting in "top": PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root 68 -72 - 0K 1088K WAIT 0 279:36 65.77% intr I see "intr" in on of top 3 slots almost all the time. turning on -H (thread view) shows me: 12 root -72 - 0K 1088K WAIT 2 69:04 20.36% intr{swi1: netisr 3} (Does this mean netisr has swi (software interrupt) on cpu3?) also, I see this process jumping to all different CPUs (so its not sticking to 1 cpu) -bash-4.2$ vmstat -i interrupt total rate irq4: uart0 1538 0 cpu0:timer 23865486 1108 irq256: igb0:que 0 46111948 2140 irq257: igb0:que 1 49820986 2313 irq258: igb0:que 2 41914519 1945 irq259: igb0:que 3 40926921 1900 irq260: igb0:que 4 49549124 2300 irq261: igb0:que 5 47066777 2185 irq262: igb0:que 6 50945395 2365 irq263: igb0:que 7 47147662 2188 irq264: igb0:link 2 0 irq274: ahci0:ch0 196869 9 cpu1:timer 23866170 1108 cpu10:timer 23805794 1105 cpu4:timer 23870757 1108 cpu11:timer 23806733 1105 cpu13:timer 23806644 1105 cpu2:timer 23858811 1107 cpu3:timer 23862250 1107 cpu15:timer 23805634 1105 cpu7:timer 23863865 1107 cpu9:timer 23810503 1105 cpu5:timer 23864136 1107 cpu12:timer 23808397 1105 cpu8:timer 23806059 1105 cpu6:timer 23874612 1108 cpu14:timer 23807698 1105 Total 755065290 35055 So, i seems all queues are being used uniformly. -bash-4.2$ netstat -Q Configuration: Setting Current Limit Thread count 8 8 Default queue limit 256 10240 Dispatch policy direct n/a Threads bound to CPUs disabled n/a Protocols: Name Proto QLimit Policy Dispatch Flags ip 1 1024 flow default --- igmp 2 256 source default --- rtsock 3 256 source default --- arp 7 256 source default --- ether 9 256 source direct --- ip6 10 256 flow default --- But the *interesting* part from it: -bash-4.2$ netstat -Q | grep "ip " (looking at just ip in workstreams) Workstreams: WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled 0 0 ip 0 0 73815267 0 0 0 73815267 1 1 ip 0 0 68975084 0 0 0 68975084 2 2 ip 0 0 48943960 0 0 0 48943960 3 3 ip 0 67 59306618 0 0 203888563 263168729 4 4 ip 0 0 77025108 0 0 0 77025108 5 5 ip 0 0 58537310 0 0 0 58537310 6 6 ip 0 0 81896427 0 0 0 81896427 7 7 ip 0 0 69535857 0 0 0 69535857 So, looks like only cpu3 is doing all the queuing. But it doesn't look like it's getting hammered or anything: last pid: 75181; load averages: 27.81, 27.08, 26.93 up 0+06:12:37 19:04:33 508 processes: 23 running, 476 sleeping, 1 waiting, 8 lock CPU 0: 71.8% user, 0.0% nice, 13.7% system, 14.5% interrupt, 0.0% idle CPU 1: 80.9% user, 0.0% nice, 14.5% system, 4.6% interrupt, 0.0% idle CPU 2: 77.1% user, 0.0% nice, 17.6% system, 5.3% interrupt, 0.0% idle CPU 3: 88.5% user, 0.0% nice, 9.2% system, 2.3% interrupt, 0.0% idle CPU 4: 80.2% user, 0.0% nice, 14.5% system, 5.3% interrupt, 0.0% idle CPU 5: 79.4% user, 0.0% nice, 16.8% system, 3.1% interrupt, 0.8% idle CPU 6: 83.2% user, 0.0% nice, 11.5% system, 4.6% interrupt, 0.8% idle CPU 7: 68.7% user, 0.0% nice, 18.3% system, 13.0% interrupt, 0.0% idle CPU 8: 88.5% user, 0.0% nice, 11.5% system, 0.0% interrupt, 0.0% idle CPU 9: 87.8% user, 0.0% nice, 10.7% system, 0.0% interrupt, 1.5% idle CPU 10: 87.0% user, 0.0% nice, 10.7% system, 2.3% interrupt, 0.0% idle CPU 11: 80.9% user, 0.0% nice, 16.8% system, 2.3% interrupt, 0.0% idle CPU 12: 86.3% user, 0.0% nice, 11.5% system, 2.3% interrupt, 0.0% idle CPU 13: 84.7% user, 0.0% nice, 14.5% system, 0.8% interrupt, 0.0% idle CPU 14: 87.0% user, 0.0% nice, 12.2% system, 0.8% interrupt, 0.0% idle CPU 15: 87.8% user, 0.0% nice, 9.9% system, 2.3% interrupt, 0.0% idle Mem: 17G Active, 47G Inact, 3712M Wired, 674M Cache, 1655M Buf, 1300M Free Swap: 8192M Total, 638M Used, 7554M Free, 7% Inuse, 4K In My conclusion after lookinag it for a bunch of times that all CPUs are equally doing work (if we believe top -P stats) Finally, the question: why is cpu3 doing all the queuing. and what does that actually mean? Can I improve performance OR reduce cpu load any other way? Should I change anything in my netisr settings? cheers, Hiren