From owner-freebsd-bugs@FreeBSD.ORG Sun Oct 12 02:43:50 2014 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 409FBFF3 for ; Sun, 12 Oct 2014 02:43:50 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27ABF88D for ; Sun, 12 Oct 2014 02:43:50 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s9C2hoYQ064348 for ; Sun, 12 Oct 2014 02:43:50 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 194311] New: [ixgbe] DROP_EN needs to be set early to avoid RX queue hanging Date: Sun, 12 Oct 2014 02:43:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: adrian@freebsd.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Oct 2014 02:43:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194311 Bug ID: 194311 Summary: [ixgbe] DROP_EN needs to be set early to avoid RX queue hanging Product: Base System Version: 11.0-CURRENT Hardware: Any OS: Any Status: Needs Triage Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: adrian@freebsd.org >From Luigi Rizzo: reviving this thread: i am just running experiments on 10.1 beta3 and even setting dev.ix.*.fc=0 and flipping the interface up and down does not seem to help: if i read only from a subset of the queues, the entire rx unit stalls eventually. I need to drain all queues to keep moving. Just tested this with 8 instances of netmap-ipfw running on an 8-core machine (8 queues enabled). netmap-ipfw netmap:ix0-0 netmap:ix1-0 netmap-ipfw netmap:ix0-1 netmap:ix1-1 ... and the source on another box is blasting on multiple queues with pkt-gen -f tx -i ix0 -d 10.0.10.0-10.0.10.255 I going to look at the driver's code now to see if/how this issue can be addressed. ... don't have a way to test this on HEAD. Do you know if there is any change that could be related ? On 10.1 beta 3 I noticed is that when I open a single queue on an ixgbe (and with incoming traffic), the rx unit stalls no matter what the previous state of dev.ix.*.fc (i.e. the DROP_EN bits) or QDE are. In this state, toggling DROP_EN has no effect, whereas something that seems effective is setting the QDE bit(s) in the PFQDE register for all queues, _after_ i open the device. >From the above my take is the following: - on NIC reset, the SRRCTL register starts at 0 including DROP_EN; same goes for PFQDE.QDE - setting SRRCTL.DROP_EN must happen before the receive unit is started; - conversely, toggling PFQDE.QDE has effect even when the receive unit has started - sysctl dev.ix.*.fc sets/clear DROP_EN but at the wrong time (the right time seems to be the window between reset and start) I am going to run more tests to figure out. cheers ... Index: sys/dev/ixgbe/ixgbe.c =================================================================== --- sys/dev/ixgbe/ixgbe.c (revision 272974) +++ sys/dev/ixgbe/ixgbe.c (working copy) @@ -4377,6 +4377,20 @@ srrctl &= ~IXGBE_SRRCTL_BSIZEPKT_MASK; srrctl |= bufsz; srrctl |= IXGBE_SRRCTL_DESCTYPE_ADV_ONEBUF; + + /* + * Set DROP_EN iff we have no flow control and >1 queue. + * Note that srrctl was cleared shortly before during reset, + * so we do not need to clear the bit, but do it just in case + * this code is moved elsewhere. + */ + if (adapter->num_queues > 1 && + adapter->hw.fc.requested_mode == ixgbe_fc_none) { + srrctl |= IXGBE_SRRCTL_DROP_EN; + } else { + srrctl &= ~IXGBE_SRRCTL_DROP_EN; + } + IXGBE_WRITE_REG(hw, IXGBE_SRRCTL(i), srrctl); /* Setup the HW Rx Head and Tail Descriptor Pointers */ -- You are receiving this mail because: You are the assignee for the bug.