Date: Fri, 21 Mar 2014 16:51:21 -0300 From: Christopher Forgeron <csforgeron@gmail.com> To: Markus Gebert <markus.gebert@hostpoint.ch> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Rick Macklem <rmacklem@uoguelph.ca>, Jack Vogel <jfvogel@gmail.com> Subject: Re: 9.2 ixgbe tx queue hang Message-ID: <CAB2_NwBVvvGkLQGg0ZUBN8zN-KLfVrMrxaA7mODsfK-vzDbm4g@mail.gmail.com> In-Reply-To: <CAB2_NwDJ%2BCcEkSZjnPYU_Rg-gSCkkfHtuWYiyEixAvhORLMo7g@mail.gmail.com> References: <CAB2_NwB=21H5pcx=Wzz5gV38eRN%2BtfwhY28m2FZhdEi6X3JE7g@mail.gmail.com> <1543350122.637684.1395368002237.JavaMail.root@uoguelph.ca> <CAB2_NwCGsAHdMFPoST05azb9K_O-K_khk3Bi1sF2om3puCcyCw@mail.gmail.com> <CAB2_NwC3on1xP3UAutkQa-3zu_JhK0%2B-ZjVb6_3NVemw2Or-KQ@mail.gmail.com> <CAB2_NwBSc3KWPYD-xbWYpRFTxpsKnXEr4V1ySP5g83aZM59MvQ@mail.gmail.com> <D1B4320A-DFFD-4647-8A43-238A088D7EF1@hostpoint.ch> <CAB2_NwAq-5N53hVPxOqc__kBi_QCQ0iy4fiwQJW0FTRqueN0YQ@mail.gmail.com> <34491192-F8EC-45C1-A7C8-61C3EBE5CFBD@hostpoint.ch> <CAB2_NwCYtFD1hkYwO-aC%2BuyUGjU_UopExsc0Kq8afuefiSdGtA@mail.gmail.com> <7B586D3A-BD6B-40B1-980E-7F9FD4A49F6A@hostpoint.ch> <CAB2_NwDJ%2BCcEkSZjnPYU_Rg-gSCkkfHtuWYiyEixAvhORLMo7g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Update: I've noticed a fair number of differences in the ixgbe driver between 9.2 and 10.0-RELEASE, even though they have the same 2.5.15 version. Mostly Netmap integration. I've loaded up a 9.2-STABLE ixgbe driver from Dec 25th as it was handy (I had to hack the source a bit since some #def's had changed), and I immediately notice one difference: netstat -m 21486/1464/22950 mbufs in use (current/cache/total) 4080/168/4248/6127254 mbuf clusters in use (current/cache/total/max) 4080/149 mbuf+clusters out of packet secondary zone in use (current/cache) 0/3/3/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 16384/0/16384/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 160987K/714K/161701K bytes allocated to network (current/cache/total) 17721/108/4104 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 2/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile Not a homerun, but definitely better on the jumbo clusters denied. (For reference, I'd normally see: 2/13185/0 requests for jumbo clusters denied (4k/9k/16k) ) It still gives me errors, but you can see it's really not hitting the wall for jumbo clusters on boot. Perhaps those jumbo clusters are being denied as the buffers are being setup? This is after it starts to blow up: netstat -m 21632/12838/34470 mbufs in use (current/cache/total) 4116/4808/8924/6127254 mbuf clusters in use (current/cache/total/max) 4080/4050 mbuf+clusters out of packet secondary zone in use (current/cache) 0/36/36/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 16439/121/16560/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 161591K/14058K/175649K bytes allocated to network (current/cache/total) 20581/3188/5880 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 33/84/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile ..after another 5 min of blowups netstat -m 28065/8040/36105 mbufs in use (current/cache/total) 4482/4644/9126/6127254 mbuf clusters in use (current/cache/total/max) 4112/4018 mbuf+clusters out of packet secondary zone in use (current/cache) 0/36/36/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 16384/176/16560/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 163436K/13026K/176462K bytes allocated to network (current/cache/total) 22223/3199/5880 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 33/84/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile My next attempt it ixgbe from 10.0-STABLE. I will come back to the 9.2-STABLE driver a bit later. As for what queues are locked from this: CPU0, 8, 16 - Three down, like last time.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAB2_NwBVvvGkLQGg0ZUBN8zN-KLfVrMrxaA7mODsfK-vzDbm4g>