From owner-freebsd-net@FreeBSD.ORG Fri Mar 21 19:51:22 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4B1AC740 for ; Fri, 21 Mar 2014 19:51:22 +0000 (UTC) Received: from mail-qc0-x22e.google.com (mail-qc0-x22e.google.com [IPv6:2607:f8b0:400d:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 012EDA81 for ; Fri, 21 Mar 2014 19:51:21 +0000 (UTC) Received: by mail-qc0-f174.google.com with SMTP id c9so3329357qcz.19 for ; Fri, 21 Mar 2014 12:51:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=nqXkdoKmJwVt8uNRHVtZYu6MeZJhb4lSTd9F7Ju0png=; b=i4Xxus13v1C99R2U/+kAushQigT8uAMuAgJTvT6VPIyPb33rIRx66IO8wJMgLJf5Ew L6SndY0+46u5tbPQQRmnHEZIXH6nNL7jHAqZJSEjGpuxfg1UYylrj14NX15BCs+4XjS9 dWorEvmshkXLutgg8vucrgXK/LaSOn6vhVtUdrIsbVyuYXfnMsvL/AFRbPzFiPP5SoGA Xo+i14B4R4FM0cCobQZjSxW07lQZLW7VS3UcB/jHGjceblkJRRbcJkptoWhyVg6//RSG FYMnRW2U0X44p4T2KEEQAulnCT/mXzuORTJg/yoYdfZoQMXfZ31w0wBGxFi06qI026/l nufQ== MIME-Version: 1.0 X-Received: by 10.140.29.68 with SMTP id a62mr42459222qga.57.1395431481226; Fri, 21 Mar 2014 12:51:21 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Fri, 21 Mar 2014 12:51:21 -0700 (PDT) In-Reply-To: References: <1543350122.637684.1395368002237.JavaMail.root@uoguelph.ca> <34491192-F8EC-45C1-A7C8-61C3EBE5CFBD@hostpoint.ch> <7B586D3A-BD6B-40B1-980E-7F9FD4A49F6A@hostpoint.ch> Date: Fri, 21 Mar 2014 16:51:21 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Markus Gebert Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Rick Macklem , Jack Vogel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Mar 2014 19:51:22 -0000 Update: I've noticed a fair number of differences in the ixgbe driver between 9.2 and 10.0-RELEASE, even though they have the same 2.5.15 version. Mostly Netmap integration. I've loaded up a 9.2-STABLE ixgbe driver from Dec 25th as it was handy (I had to hack the source a bit since some #def's had changed), and I immediately notice one difference: netstat -m 21486/1464/22950 mbufs in use (current/cache/total) 4080/168/4248/6127254 mbuf clusters in use (current/cache/total/max) 4080/149 mbuf+clusters out of packet secondary zone in use (current/cache) 0/3/3/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 16384/0/16384/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 160987K/714K/161701K bytes allocated to network (current/cache/total) 17721/108/4104 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 2/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile Not a homerun, but definitely better on the jumbo clusters denied. (For reference, I'd normally see: 2/13185/0 requests for jumbo clusters denied (4k/9k/16k) ) It still gives me errors, but you can see it's really not hitting the wall for jumbo clusters on boot. Perhaps those jumbo clusters are being denied as the buffers are being setup? This is after it starts to blow up: netstat -m 21632/12838/34470 mbufs in use (current/cache/total) 4116/4808/8924/6127254 mbuf clusters in use (current/cache/total/max) 4080/4050 mbuf+clusters out of packet secondary zone in use (current/cache) 0/36/36/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 16439/121/16560/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 161591K/14058K/175649K bytes allocated to network (current/cache/total) 20581/3188/5880 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 33/84/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile ..after another 5 min of blowups netstat -m 28065/8040/36105 mbufs in use (current/cache/total) 4482/4644/9126/6127254 mbuf clusters in use (current/cache/total/max) 4112/4018 mbuf+clusters out of packet secondary zone in use (current/cache) 0/36/36/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 16384/176/16560/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 163436K/13026K/176462K bytes allocated to network (current/cache/total) 22223/3199/5880 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 33/84/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile My next attempt it ixgbe from 10.0-STABLE. I will come back to the 9.2-STABLE driver a bit later. As for what queues are locked from this: CPU0, 8, 16 - Three down, like last time.