Date: Fri, 25 Apr 2014 15:25:04 +0100 From: "Steven Hartland" <killing@multiplay.co.uk> To: =?iso-8859-1?Q?Gerrit_K=FChn?= <gerrit.kuehn@aei.mpg.de> Cc: freebsd-net@freebsd.org Subject: Re: NFS over LAGG / lacp poor performance Message-ID: <1BD2C00D4F0D405C939C975246D41C60@multiplay.co.uk> References: <535A1354.2040309@wp.pl><20140425113711.e7c7d1c2.gerrit.kuehn@aei.mpg.de><535A482E.1030106@wp.pl><20140425140123.a76c18f9.gerrit.kuehn@aei.mpg.de><535A5268.100@wp.pl><8247FE6336414E1F97ADA561D0680097@multiplay.co.uk><535A5DD9.9060206@wp.pl><AC1233D368D94519BB59B484D6F385C0@multiplay.co.uk><535A69CE.9010800@wp.pl><4E7A280CACC84910A3DE068085976E30@multiplay.co.uk> <20140425161312.3ad95e3a.gerrit.kuehn@aei.mpg.de>
next in thread | previous in thread | raw e-mail | index | archive | help
----- Original Message -----
From: "Gerrit Kühn" <gerrit.kuehn@aei.mpg.de>
To: "Steven Hartland" <killing@multiplay.co.uk>
Cc: <freebsd-net@freebsd.org>
Sent: Friday, April 25, 2014 3:13 PM
Subject: Re: NFS over LAGG / lacp poor performance
> On Fri, 25 Apr 2014 15:02:15 +0100 "Steven Hartland"
> <killing@multiplay.co.uk> wrote about Re: NFS over LAGG / lacp poor
> performance:
>
> SH> We find that large numbers of queues causes high interrupt issues
>
> Like the thing I am seeing with igb1 on my system?
>
> ---
> root@storage:/root # vmstat -i
> interrupt total rate
> irq1: atkbd0 2743 0
> irq18: ehci0 uhci5 4445560 2
> irq21: uhci1 29 0
> cpu0:timer 355724275 227
> irq256: igb0:que 0 99437514 63
> irq257: igb0:que 1 61534816 39
> irq258: igb0:que 2 101725601 65
> irq259: igb0:que 3 100864440 64
> irq260: igb0:link 2 0
> irq261: igb1:que 0 1689527 1
> irq262: igb1:que 1 2357590958 1510
> irq263: igb1:que 2 1584474 1
> irq264: igb1:que 3 1923144 1
> irq265: igb1:link 2 0
> irq266: mps0 332232450 212
> irq267: mps1 194207894 124
> irq268: mps2 176700834 113
> irq269: ahci0 59175548 37
> cpu1:timer 419838321 268
> cpu3:timer 329696415 211
> cpu2:timer 328219053 210
> Total 4926593600 3156
> ---
>
>
> irq262 sticks out like a sore thumb...
>
> SH> however at a guess you did this to enable the machine to boot with
> SH> all nics due to lack of auto mbuf tuning in 9.x.
> SH> I'd go with ~2 queues per nic.
>
> I was wondering what to try next for my system: either manually set the
> queues back to 2 or 1 per NIC, or try upgrading to either 9.2 or 10 as it
> looked like there have been improvements in the igb driver. Do you have
> any recommendations on that?
We saw the issue with 10-RELEASE last weekend on a machine with 6 x igb's
where by the system was burning CPU in the interrupt handlers, setting
num_queues to 2 fixed the issue we where seeing.
Regards
Steve
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1BD2C00D4F0D405C939C975246D41C60>